Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reload on file change handling for tini-watched processes #82

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

mb-m
Copy link

@mb-m mb-m commented Apr 6, 2017

Some container-based programs can do an inline reload when a
config file (particularly those mounted on a Volume) changes, but
need to be signalled. For a local docker, we can do docker kill
and let the signal pass through tini, but on a scheduler, like
Kubernetes, we either have to run commands in each container or
delete and recreate the containers (as per the replication-group
strategy).

With this patch, we can run something like prometheus or fluentd
in containers, and be able to use kubernetes to push updates,
with the relevant program auto-reloading, without the need for a
side-bar watcher, making tini a little bit more like a "supervisor".

@mb-m
Copy link
Author

mb-m commented Apr 6, 2017

Hi @krallin, I apologise in advance for doing a bit of a drive-by PR here - I'm actually about to head off on holiday where I'm not likely to have internet access, but some of my colleagues are also watching this PR (I did it as part of my work), and may be able to help with any questions you have.

I've basically been wanting this for (in our environment), fluentd and prometheus in Kubernetes, although I'm sure we could do better with our kubectl automation to rotate the pods. Ideally, IMO, these kinds of services would automatically pick up their config changes, but that's not the world we live in, so allowing tini to do it is also helpful.

Some container-based programs can do an inline reload when a
config file (particularly those mounted on a Volume) changes, but
need to be signalled. For a local docker, we can do docker kill
and let the signal pass through tini, but on a scheduler, like
Kubernetes, we either have to run commands in each container or
delete and recreate the containers (as per the replication-group
strategy).

With this patch, we can run something like prometheus or fluentd
in containers, and be able to use kubernetes to push updates,
with the relevant program auto-reloading, without the need for a
side-bar watcher, making tini a little bit more like a "supervisor".
@krallin
Copy link
Owner

krallin commented Apr 11, 2017

(Just a quick note that I've seen your PR and I'm spending a bit of time thinking through it, as this would be a new direction for Tini)

@mb-m
Copy link
Author

mb-m commented Apr 18, 2017

Hi @krallin, I'm back now to help. I appreciate that it's a new direction - but I think it's a valid one in a kubernetes world, and makes tini useful.

If we want to make it a bit nicer, it's plausible to do it by inotify (if available) rather than the polled stat() calls.

@krallin
Copy link
Owner

krallin commented Apr 21, 2017

Hey @mb-m, thanks for your reply. I'm sorry I'm taking a while to respond to this — I have been pretty busy at work (and I will be out next week for vacation, so my responsiveness is going to get worse before it gets better 😄 ).

Can you tell me a little bit about how you're injecting these files into your containers using Kubernetes? I haven't worked much with Kubernetes so I'm not 100% sure how this is generally useful there.

Re: inotify: if this goes in, I'm fine with polling stat unless there's a fundamental reason not to: I'd rather keep Tini as simple as possible.

Cheers,

@mb-m
Copy link
Author

mb-m commented Apr 24, 2017

Hi @krallin, first off, thanks for getting back, and I hope you have a lovely holiday!
In Kubernetes, but also in docker, you can end up with volume mounts which are in some way outside the container. In the case of kubernetes, these end up being a "configmap" which is a separate structure to the "deployment" structure of the pod of containers. These end up creating what is effectively a docker volume for a single file, meaning that you can trivially replace the file without stopping and restarting the container.

While the container itself can be stopped and started, this can be more disruptive (eg. in the case where the container is running your monitoring - the stop and the start will miss some data collection and may miss generation of an alert), so where possible, it's worth having the reload-in-place semantics.

This obviously isn't the most useful in every situation, and for many situations the correct behaviour is to do the stop and restart as you'd expect, but in terms of improving reliability there are some where keeping the process running if possible is worthwhile, and doing this automatically rather than manually even more so (especially if you're making the same update across multiple environments).

@mb-m
Copy link
Author

mb-m commented May 4, 2017

Hi @krallin, any further thoughts?

@krallin
Copy link
Owner

krallin commented May 20, 2017

Ok; I've been giving this some thought and I'd like to pull this in. I think there's quite a bit to be done to make Tini a slightly more feature-complete init system (outside of "minimal mode"), with this, and a few other possible features like #75 or auto-restart.

I'll take a clover look at the code you're proposing here! I think that if we want to support additional features like this, a bit of a re-architecting might be worth it — the current architecture is very much tied to its "reap zombies and that's it" functionality.

@savar
Copy link

savar commented Aug 1, 2017

any updates on this?

@danlsgiga
Copy link

Just as a matter of feedback on how useful this feature would be.

Currently I'm evaluating the use of the NOMAD scheduler and I'm facing a challenge regarding the use of Spring Boot Java applications and dynamic generated secrets.

Since we'll be using dynamic secrets in our environment, we can't really stand for restarting our Java applications everytime a token changes. Since Spring Boot applications only accept refreshes to the Beans scope through an API call to the /refresh endpoint having a init process being able to kick off a custom command once a file is updated would be pretty awesome.

At this point, our only solution is to have a sidekick daemon (watcher in this case) to monitor the file and execute a curl http://localhost/refresh.

@savar
Copy link

savar commented Dec 20, 2017

any updates on this topic?

karteek pushed a commit to karteek/tini that referenced this pull request Feb 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants