Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #37035 - add unit starting limits in place #9980

Open
wants to merge 2 commits into
base: develop
Choose a base branch
from
Open

Fixes #37035 - add unit starting limits in place #9980

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9dda960
Fixes #37035 - add unit starting limits in place
ares Jan 8, 2024
520f5b8
Update extras/systemd/foreman.service
ares Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions extras/systemd/foreman.service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ Description=Foreman
Documentation=https://theforeman.org
After=network.target remote-fs.target nss-lookup.target
Requires=foreman.socket
StartLimitIntervalSec=120
StartLimitBurst=5
Comment on lines +6 to +7
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that means 5 restarts in 120 seconds, which also applies to "normal" restarts (= user requested ones), which I find rather conservative.
would 5 in 30 also elevate our concerns?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to be on the safe side, let me do some measuring on the error I currently see, the booting takes few seconds before crashing (also I don't think user would commonly restart Foreman 5 times in 2 minutes)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm, it takes 20-60 seconds until it crashes (based on the system load), 5*40 can be even 200 seconds, how about we increase both numbers then? 10 restarts in 30 minutes? it crashes after the whole Rails env loads...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commonly? No. When debugging something? for sure.

E.g. pretty sure I did that when poking https://community.theforeman.org/t/unable-to-login-to-foreman-gui-using-smart-proxy-link-after-upgrading-to-3-8-0-version/35950

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So what would be you recommendation?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of thoughts, not strictly well sorted, but anyhow.

  • We do not set Restart=always in the service, so when it crashes it actually remains down
  • What we do have is socket activation, so if your problematic setup has something actually accessing the webui while it's broken, you get restarts in the background via the socket (and once that started to try to activate, it runs into a loop).
  • Recently, I've read https://michael.stapelberg.ch/posts/2024-01-17-systemd-indefinite-service-restarts/ and that suggests to set RestartSec=5s to avoid this prolonged CPU burn you're trying to avoid. It seems however, that this doesn't apply to socket-triggered restarts :/ (I've pinged Michael if he has an idea, but let's assume the answer is no for now)

That said, I think if we want to limit this "problem", we'll have to take your approach.

(But I also don't strictly see this as a problem, but it could be me)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well it happened to me twice already, if I didn't hear the cpu fan I'm sure the machine would burn at some point. I think the issue was always related to some issue in nightlies, which I guess people don't run in the production. Still, very annoying.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I still think it's your setup, not Foreman.

But if you rebase and squash this, we can get it in for 3.14 and see where that goes.


ares marked this conversation as resolved.
Show resolved Hide resolved
[Service]
Type=notify
Expand Down
Loading