Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build fails with 'no space left on device' if "too many" packages are installed in builder #54

Closed
fwilhe opened this issue Jan 8, 2024 · 6 comments · Fixed by #55
Closed
Assignees
Labels

Comments

@fwilhe
Copy link
Member

fwilhe commented Jan 8, 2024

What happened:
I'm building an image using the builder and the build fails with 'no space left on device' while installing packages.

What you expected to happen:
My build should work.

How to reproduce it (as minimally and precisely as possible):
Github actions workflow: https://github.com/gardenlinux/dev-image/actions/runs/7448117408/job/20261765952?pr=3

Try building this commit: https://github.com/gardenlinux/dev-image/tree/ba57ad53a8d8a4af1f01daf1a0f6862674ef76d7

@fwilhe fwilhe added the kind/bug Bug label Jan 8, 2024
@fwilhe
Copy link
Member Author

fwilhe commented Jan 8, 2024

Analysis: This happens because the builder creates temp fs in multiple places and their size is hardcoded to 2G

https://github.com/search?q=repo%3Agardenlinux%2Fbuilder%202G&type=code

It works when using 4G instead like in this commit

@fwilhe fwilhe self-assigned this Jan 8, 2024
@fwilhe
Copy link
Member Author

fwilhe commented Jan 8, 2024

Reasoning for having a hard-coded limit: Make the build reproducible. It should not randomly fail based on how much memory the build machine has. 2G seems large enough for building the Garden Linux image, but it does not seem to be enough for the Dev Image.

@fwilhe
Copy link
Member Author

fwilhe commented Jan 8, 2024

Potential Solution 1

Increase hard-coded size #53

Pro:

  • Simple to implement

Con:

  • Still hardcoded
  • Not needed for main gardenlinux repo
  • Contradicts the design goal to work with minimal required resources

@fwilhe
Copy link
Member Author

fwilhe commented Jan 8, 2024

Potential Solution 2

Add a build script like get_repo, get_commit, get_timestamp to set a custom tempfs size

Pro:

  • Simple to implement

Con:

  • Adds more files to the repo root which might seem odd to users
  • If only a single feature needs a lot of space, it forces the whole repo to increase this setting

@fwilhe
Copy link
Member Author

fwilhe commented Jan 8, 2024

Potential Solution 3

Allow features to declare how much space they need

Assuming I have 3 features: base, large and huge.

base needs 2Gig, large needs 3Gig and huge needs 4Gig, how much space does the overall build need? Is it max(2,3,4)? Is it sum(2,3,4)?

Pro:

  • Only large features need to request more space

Con:

  • Needs to parse files in all features (implementation and test effort)
  • Unclear calculation requirements

@fwilhe
Copy link
Member Author

fwilhe commented Jan 8, 2024

Potential Solution 4

Introduce a new build.config file that optionally can customise build parameters.

The file is placed in the root of any directory/git repo containing an image definition.

If the file is absent, or does not define all values, the values should still have a sensible default value.

The builder_exmple repo will get a build.config file containing sample values which increases visibility of those settings to the user.

The file is in a format so it can be sourced by a shell

For example

TEMPFS_SIZE=4G

Pro:

  • One place for customizations
  • The configuration is part of the git repo (important for reproducible builds)

Con:

  • If only a single feature needs a lot of space, it forces the whole repo to increase this setting

Potential issues:

  • It might be possible that the user accidentally overrides some builder-internal variables?

Potentially this could also replace get_repo and get_version (don't need to be a script for providing a single string value), which could result in a file like this:

APT_REPO=http://deb.debian.org/debian
VERSION=trixie
TEMPFS_SIZE=4G

And not needing the get_repo and get_version script.

fwilhe added a commit that referenced this issue Jan 10, 2024
In some cases the current hard-coded size limit for tempfs makes it impossible to build images with the builder. This change introduces a new config interface via a 'build.config' file which builder users might use to override selected variables.

Fixes #54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant