Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-work of mucoll-spack to enable build of releases fully on github #17

Open
wants to merge 82 commits into
base: master
Choose a base branch
from

Conversation

madbaron
Copy link
Contributor

@madbaron madbaron commented Nov 25, 2024

In preparation for the next software release, this PR updates mucoll-spack to enable the complete build of releases within the github workflows.

This relies on an intermediate spack environment (key4hep-dev-base) by @tmadlener that contains the external dependencies of the key4hep stack. This environment is used to build an intermediate docker image (called mucoll-minimal) that is subsequently picked up to build the muon collider stack.

The CI is configured so that whenever the dockerfiles, or the spack configuration is changed, only the affected images (and those downstream) will be re-built.
The resulting docker images are automatically published and linked to the github page.

The complete workflow takes about 8 hours to complete on the github runners.

Please note that deploying the workflows to re-build the releases requires (without using external triggers) the dockerfiles to be bundled together with mucoll-spack, which is different from what we did with 2.9.
Personally I think this is ok (perhaps even a little better for discoverability) but please let me know if you disagree.

Adding a few people as reviewers!


Some notes for the future:

  • the preparation of the mucoll-minimal image (which is really a key4hep-minimal) would be best moved to key4hep-spack as it is completely generic
  • we should set up a cron-triggered workflow to run as nightly build with the latest versions of all packages
  • the previous concretize workflows have been removed, as they are part of the docker image building process. They could be split off to a separate step of the workflow if useful
  • we can review the list of packages to be compiled (i.e. all event generators) to make the resulting image a little slimmer

@madbaron madbaron marked this pull request as draft November 25, 2024 15:52
@madbaron madbaron changed the title Draft: re-work of mucoll-spack to enable build of releases fully on github Re-work of mucoll-spack to enable build of releases fully on github Nov 25, 2024
Copy link
Contributor

@tmadlener tmadlener left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't have time to look in-depth, I just have one comment already about the new grouping of packages into mucoll-stack variants.

packages/mucoll-stack/package.py Outdated Show resolved Hide resolved
@madbaron
Copy link
Contributor Author

A couple of notes:

  • The image building pipeline is not expected to complete successfully on a PR (the github token is not given write access in PRs to public repositories, per github policy). If people would like to take a look at the outputs, the corresponding workflow was executed in my fork: https://github.com/madbaron/mucoll-spack/actions/runs/12018805366
  • Since the build workflow won't succeed on PRs, I have re-introduced the concretisation checks (and plan to make them a bit more granular once a version of the various docker image is present in the muoncollidersoft registry

@madbaron madbaron requested a review from tmadlener November 26, 2024 08:27
@madbaron madbaron marked this pull request as ready for review November 26, 2024 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants