Dokku3 high disk usage

[lucyb]

On Tuesday 12th Nov, Dokku4 ran out of disk space and it was then noticed that Dokku3 had used almost all of its disk space too.

Running dokku cleanup didn't free up any additional space. However, running docker system prune did delete a large number of build cache objects and freed up around 70G of space. Taking disk usage down from (around) 93% to 56%.

It appears that there's a dokku/dokku#7216 with Dokku failing to clean up appropriately when deploying via git:from-image. This may or may not have caused the problem.

Until Dokku release a fix, or unless we find a workaround, we should regularly run docker system prune.

Note: it was necessary to redeploy Metrics after running a system prune. Might be because the Docker image isn't tagged, but might just be a quirk of how we're using it. Consider documenting this?

[lucyb]

Update: I've added a Digital Ocean alert for both dokku3 and dokku4, so tech support will be notified if we start running low on disk space. So this work is much less urgent now.