Will isolation feature work for you? It supports https://github.com/cirruslabs/vetu too.

How do you orchestrate those VMs/containers?

Thanks! I've spent the past few days having a look at vetu and set up an example infrastructure with it. While the CI runs and works, I'm missing a few features from vetu. I understand that it's probably still in an early stage of development.

• With tart and containers you are able to mount volumes and shared directories for e.g. caches. With vetu you can't do this yet.
• In general, you can't set limits to the vetu VM disk size, memory size, or CPU limit on the host side. These are all set in the .cirrus.yml in the repository. If someone (maliciously) increase these in a PR, the cirrus runner would spawn a VM with all available CPU cores/uses up a lot of memory to a point where the cirrus-worker crashes and other tasks fail too/or fill up the disk making the system unusable.
• vetu doesn't seem to have many users yet, the documentation is a bit sparse, and it's hard to control VM details

From the vetu README:

We say effortlessly, because the existing virtualization solutions like the traditional QEMU and the new-wave Firecracker and Cloud Hypervisor provide lots of options and require users to essentially build a tooling on top of them to be able to simply run a basic VM.

I think, in our case, we'd want to have this control over the VMs and provide these options to the VMs our self.

How do you orchestrate those VMs/containers?

After having tried a few options, our current plan would be to have a few, fixed size (CPU, memory, disk space) QEMU/cloud-hypervisor/firecracker VMs on a single host. These can then share cache directories. Inside each VM, we'd run a persistent worker. With something like cirrus worker --one-task, the cirrus process would end after completing a single task and we'd trigger a VM shutdown. Before bringing it up again, the VMs volumes would be re-created. Once back up, it will be waiting in standby for a new task or pick up an existing task.

I'm happy to work on an implementation of --one-task, give it a try, and then PR it here.

Thank you for the details. We use veru for Linux part of Cirrus Runners and it's been working great! But yeah, folder mounts not yet supported.

Seems this functionality can be under --ephemeral flag similar to GitHub Actions. PRs are always appreciated!