FR: Rollback of podman auto-update shall include volume contents

[Zocker1999NET]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

It would be great if podman auto-update could also backup the volume's contents of the container and rollback those as well if the container fails to start. This could help to mitigate problems arising if e.g. the new container version tried to migrate the databases to a new version and failed unexpectedly.

This should only affect read-write mounted volumes as other volumes could not be damaged by the updated version. Volumes with a certain label might be ignored as well (e.g. because it is not expected to be affected negatively by the update).

The feature may require volumes to only be mounted into the one container which should be updated as otherwise other containers might fail because of the unexpected rollback. This requirement might be lifted by adding a special label to the volume (e.g. if other containers are expected to handle such changes correctly).

This feature may be optional and disabled by default to not break the current expectation of auto-update just ignoring the volumes.

[mheon]

Could potentially be handled by checkpointing the container and then restoring that checkpoint if the update fails

[Luap99]

I think this is an interesting use case but it it looks hard to get right.

The feature may require volumes to only be mounted into the one container which should be updated as otherwise

Well this does still not mean the the volume is not used by others. It can still be used by other users or applications on the host.

Another problem would be where to store the backup? We need to duplicate all data which is expensive and it could run out of disk space.

Also should this work for all volumes (including normal bind mounts -v /data:/data) or only named volumes?

I think the only way that would reasonable work is if it is opt in and configurable on a per volume basis (using a volume labels can work but will exclude bind mounts)

Also the rollback functionality is based on sdnotify, see https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html#rollback
So your container has to use sdnotify and only sends the READY message after it changes volume content in order for this feature to make sense.

@vrothberg WDYT?

[Zocker1999NET]

Also should this work for all volumes (including normal bind mounts -v /data:/data) or only named volumes?

I thought about excluding bind mounts for the beginning as there it isn't easy to check whether other apps are also accessing the data.

It can still be used by other users or applications on the host.

In theory, yes, but I would assume that volumes shall only be accessed by podman containers.

[vrothberg]

The use case sounds convincing and I am interested in finding a solution. I am not an expert in database, so I do not know how common this scenario is. I would expect that such updates are idempotent but there may be other scenarios.

It's definitely a feature where users "know what they are doing".

[rhatdan]

This would seem to scream out for some kind of checkpointing/snap shooting file system like BTRFS. Could this be done in a Volume Driver which could take a snapshot on update?

[Luap99]

I am not an expert in database, so I do not know how common this scenario is.

I would say that it is definitely common that applications change the database schema on updates, making a rollback impossible. We even did this in Podman v4.
However the problem is that this is often done by another application.

Lets consider my nextcloud setup:
I have a nextcloud instance on my server. This contains a nextcloud db container and another nextcloud application container (webserver with php)

When I update nextcloud it will often update the db. Now if the nextcould update fails for some reason and we get a rollback it might fail because the db schema was already updated. (AFAIK nextcoud does not support sdnotify so this is just an imaginary example)

The reason I bring this use case up is because often the dependencies go across containers. In this case restoring volumes on the nexcould container would not help, we would have to stop the db container make a backup resume the db now update nexcloud if it fails stop the db, restore db backup, start db, downgrade nextcloud.

Of course if we limit it to volumes for the updated container it will be simpler but also will not work in scenarios where the db and application are separated.

---

This would seem to scream out for some kind of checkpointing/snap shooting file system like BTRFS. Could this be done in a Volume Driver which could take a snapshot on update?

Yes that would make things a lot more performant and we would avoid duplication.

[vrothberg]

This would seem to scream out for some kind of checkpointing/snap shooting file system like BTRFS. Could this be done in a Volume Driver which could take a snapshot on update?

Deferring that to volume drivers would be very nice and easy to plug into auto updates.

[Zocker1999NET]

However the problem is that this is often done by another application.

I would exclude supporting this scenario from this feature, as that depends on the composing of the containers, which Podman is not aware of. podman-compose or Kubernetes are aware of that and so IMO would need to handle that (or additional tools which are aware of the composing as well).

It sounds nice to be able to checkpoint volumes, maybe this could also exposed as own features (volume commit & volume restore), which auto-update then uses.

For the local driver on a FS without native copy-on-write support, maybe overlayfs could be used?

[rhatdan]

I actually think that would be a nice advance to podman volume support. Good idea.
Podman does support kubernetes.YAML file so the question is does Kubernetes support something in the YAML file to tell us to snap shot and restore volumes, or is this something handled by a controller?

[rhatdan]

Moving this to a discussion, since I think this would be a nice feature for Edge devices to be able to commit, update, healthcheck and on failure, rollback the volume.

[nolange]

Naive idea, but can't you mount an tmpfs overlay on top of everything, see if the container runs correctly, then either restart the new one without said overlay or the old one?

[Luap99]

That causes an extra container run in case of an successful update, I don't think this is desirable. Also there is no guarantee that it will work the second time again.

[nolange]

That causes an extra container run in case of an successful update, I don't think this is desirable. Also there is no guarantee that it will work the second time again.

In which case you have a 100% guarantee that the update does work?

I would argue that you should provide a set of options. The tmpfs overlay method should ensure a rather safe rollback, potentially even if you have some dependent containers (eg. database in its own container).
Any state stores in the trial run will be lost, but that might be no issue.

[Luap99]

In which case you have a 100% guarantee that the update does work?

In the case when we started it and it works, as soon as we have to restart it to remove the overlay it could fail.

Also overlay has other problerms as well. AFAIK lower and upper dir must be on the same underlying fs and it could cause performance problems when it has to copy up big files.