Fixes #37035 - add unit starting limits in place #9980
Conversation
| StartLimitIntervalSec=120 | ||
| StartLimitBurst=5 |
There was a problem hiding this comment.
that means 5 restarts in 120 seconds, which also applies to "normal" restarts (= user requested ones), which I find rather conservative.
would 5 in 30 also elevate our concerns?
There was a problem hiding this comment.
I wanted to be on the safe side, let me do some measuring on the error I currently see, the booting takes few seconds before crashing (also I don't think user would commonly restart Foreman 5 times in 2 minutes)
There was a problem hiding this comment.
hm, it takes 20-60 seconds until it crashes (based on the system load), 5*40 can be even 200 seconds, how about we increase both numbers then? 10 restarts in 30 minutes? it crashes after the whole Rails env loads...
There was a problem hiding this comment.
Commonly? No. When debugging something? for sure.
E.g. pretty sure I did that when poking https://community.theforeman.org/t/unable-to-login-to-foreman-gui-using-smart-proxy-link-after-upgrading-to-3-8-0-version/35950
There was a problem hiding this comment.
So what would be you recommendation?
There was a problem hiding this comment.
Couple of thoughts, not strictly well sorted, but anyhow.
- We do not set
Restart=alwaysin the service, so when it crashes it actually remains down - What we do have is socket activation, so if your problematic setup has something actually accessing the webui while it's broken, you get restarts in the background via the socket (and once that started to try to activate, it runs into a loop).
- Recently, I've read https://michael.stapelberg.ch/posts/2024-01-17-systemd-indefinite-service-restarts/ and that suggests to set
RestartSec=5sto avoid this prolonged CPU burn you're trying to avoid. It seems however, that this doesn't apply to socket-triggered restarts :/ (I've pinged Michael if he has an idea, but let's assume the answer is no for now)
That said, I think if we want to limit this "problem", we'll have to take your approach.
(But I also don't strictly see this as a problem, but it could be me)
There was a problem hiding this comment.
Well it happened to me twice already, if I didn't hear the cpu fan I'm sure the machine would burn at some point. I think the issue was always related to some issue in nightlies, which I guess people don't run in the production. Still, very annoying.
There was a problem hiding this comment.
Yeah. I still think it's your setup, not Foreman.
But if you rebase and squash this, we can get it in for 3.14 and see where that goes.
Co-authored-by: Evgeni Golov <[email protected]>
|
Thank you for your contribution! This PR has been inactive for 3 months, closing for now. Feel free to reopen when you return to it. This is an automated process. |
Systemd unit is configured in a way to start the application over and over again. In case the app fails e.g. on missing seeded data or some bug, it's always restarted. We should put the limit in place.