Pipelines fail frequently

[patrickakk]

@dpancic
Recently a lot of pipelines fail, because of the error specified below. It doesn't work to choose "re-run failed". Usually it only works when changing a file (adding a space), Doing a commit and push. Then it works.
So it doesn't has anything to do with the code or the tests.

In the past it sometimes worked to re-run the failed jobs. Now that usually doesn't work. And it seems that the error happens more frequently.

Is there anything we can do about this?

Thanks in advance for your help.

[patrickakk]

Update:
It seems like everything works fine, as long as the pipelines are triggered often. Maybe at least once a week?

As soon as the were no commits and therefore no pipelines did run for approx 3 weeks or longer, this problems seems to happen. Maybe because in that case something else did change in the meantime?

Unfortunately, at this moment, I can't describe it more detailed. Does this information help?

[patrickakk]

Update: The situation above does not apply. Right now it failed again, although there were other successful pipelines at the same day.

[acdh-ch]

@patrickakk the problem is caused by missing image tag. It seems that sometimes the image tag is not forwarded from the build stage to deployment stage. I tried to fix it by adding 30 sec delays before deployment stage.
Please test it and, if works, add it to the test and prod branches. If problem appears again, please check first generated auto-deploy-yamls before you try with the next commit. I assume that you will not see image tag in the yaml file. Adding delay is an attempt to give more time to the runner for generating auto-deploy.yaml file with all necessary values.

[patrickakk]

@acdh-ch
Thanks for the explanation and fix.

How should I test it? Since the problem does not always occur, but only "sometimes".

Adding this to test and prod branche, will be done "automatically" when to code from dev is merged, when working to the next release?

For now I'll move the issue to the January milestone and keep it open, as a reminder to check it.

[patrickakk]

@acdh-ch
Today a few pipelines worked fine.

Than they failed again:
https://github.com/acdh-oeaw/dhcr-main/actions/runs/12226073948
https://github.com/acdh-oeaw/dhcr-main/actions/runs/12226098669

For the first one 1 checked the auto-deploy-app-values.yaml and it does contain an image, tag?

Does this provide you sufficient info?

[acdh-ch]

It seems that although auto-deploy-app-values.yaml contains an image tag, it is not always forwarded to the auto deploy application.

The next try is setting image tag to latest in autodeploy app template helper. With this change the auto deploy app should deploy image even if tag is not forwarded. The only difference will be in the image name. In cases tag is not forwarded deployed image will have tag latest.

[patrickakk]

@acdh-ch

Thanks.

Both pipelines (dev and test) runned fine now. But, unfortunately that doesn't say anything, since this error only occurs "sometimes".

I'll mark the issue as "blocked" and wait until the end of January to see how it goes.

[patrickakk]

@acdh-ch
Today, same problem, see here:
https://github.com/acdh-oeaw/dhcr-main/actions/runs/12280448811

Since I needed the pipelines to work, I used the usual trick:
Add one space in a file, commit and push.

Does this provide you with sufficient info?

[acdh-ch]

@patrickakk There is a new error. The image is tagged with test:%!s(float64=9.544892e+06)
It seems that when image tag (commit) starts with a number, the tag is misinterpreted as a floating-point number and not treated as a string by auto-deploy-app. That is causing an error and pipeline fails.
Tried to fix this by using ''. The tag is now changed to '$tag'.

[patrickakk]

@acdh-ch

Everything worked fine today.

Thank you.

[patrickakk]

Status update:
Keep issue open until end of January and see if further problems occur.