Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test ACTSTracking Processors Using GitHub Actions #2

Draft
wants to merge 61 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
b00fd04
Start work on image building GitHub action.
kkrizka Sep 28, 2021
c327f69
Add runs-on.
kkrizka Sep 28, 2021
e70d4f7
Specify correct environment file.
kkrizka Sep 28, 2021
227b011
Build context should be CentOS8.
kkrizka Sep 28, 2021
8f9dea9
Try fixing the context.
kkrizka Sep 28, 2021
5bab58e
More context debug.
kkrizka Sep 28, 2021
4e40872
Do git checkout to get context.
kkrizka Sep 28, 2021
6c2a6c1
More fixes to paths.
kkrizka Sep 28, 2021
202ccd8
Add kkrizka to image tag.
kkrizka Sep 28, 2021
68ebd10
Build all images using Git SHA as tag version.
kkrizka Sep 28, 2021
1f99e11
Fix build-args.
kkrizka Sep 28, 2021
d764208
Fix build-args.
kkrizka Sep 28, 2021
26acaa9
Fix multiple docker_build ID's.
kkrizka Sep 28, 2021
db9712c
Use a secret for the DockerHub repository.
kkrizka Sep 28, 2021
e93e692
Add release action to tag the final image.
kkrizka Sep 28, 2021
972d702
Start using docker/metadata-action for tags.
kkrizka Sep 28, 2021
dc973e3
Fix YAML typo.
kkrizka Sep 28, 2021
27e86f2
Consisten repository and version among all images.
kkrizka Sep 29, 2021
209def9
Add dummy image argument to Docker Meta.
kkrizka Sep 29, 2021
23223fe
image -> images
kkrizka Sep 29, 2021
0644214
Merge pull request #1 from kkrizka/kk_action
kkrizka Sep 29, 2021
485af8c
Change image version in README to devel to correspond to default valu…
kkrizka Sep 29, 2021
e66db91
Start work on testing code by running simulation.
kkrizka Sep 29, 2021
217b3fe
Try fixing runs-on via temporary fixed value.
kkrizka Sep 29, 2021
7f9dae0
Fix container specification.
kkrizka Sep 29, 2021
24d2035
Try fixing runs-on via temporary fixed value.
kkrizka Sep 29, 2021
1c04d90
Try using secrets.DOCKERHUB_REPOSITORY again.
kkrizka Sep 29, 2021
7b59756
Reuse sim events between runs.
kkrizka Sep 29, 2021
036c888
Debug
kkrizka Sep 29, 2021
a999df0
Run Marlin
kkrizka Sep 29, 2021
67e86aa
Readable workflow names
kkrizka Sep 29, 2021
4fbd70e
Debug ddsim load.
kkrizka Sep 29, 2021
ea3816f
Fix init script path for mcc software.
kkrizka Sep 29, 2021
837e308
Debug event sim caching.
kkrizka Sep 29, 2021
0a61c23
Fix cache key.
kkrizka Sep 30, 2021
dd4e07d
Fix cache key.
kkrizka Sep 30, 2021
37218ce
Update MuonCutil with ACTS 13.0.0
kkrizka Sep 30, 2021
2a03dc2
Update MuonCutil to working branch with updated ACTSTracking tag.
kkrizka Sep 30, 2021
19a580c
Update ACTSTracking CI tests.
kkrizka Oct 1, 2021
f0d5a21
Test cache running outside of container.
kkrizka Oct 2, 2021
a6379f5
Fix path to Marlin job cache.
kkrizka Oct 2, 2021
c496787
Single workflow for build and test.
kkrizka Oct 2, 2021
69cf1e3
Try a different variable
kkrizka Oct 2, 2021
50cc251
Test image passing via cache.
kkrizka Oct 2, 2021
27ec288
Fix docker image cache.
kkrizka Oct 3, 2021
51cddb2
Test fetching image from DockerHub.
kkrizka Oct 5, 2021
e7b40d7
Test with bypassing cache for events.
kkrizka Oct 6, 2021
395f08b
Test jobs now depend on build job.
kkrizka Oct 6, 2021
8123b2f
Fix syntax when specifying multiple tags.
kkrizka Oct 6, 2021
40723ce
Tagged ACTSTracking v1.0.0.
kkrizka Oct 7, 2021
510df35
Update MuonCutil to test latest code.
kkrizka Dec 1, 2021
16f016e
Version independent test CI stage.
kkrizka Dec 1, 2021
6b0b535
Add Apache Arrow.
kkrizka Dec 16, 2021
a2099d1
Escape + files.
kkrizka Dec 16, 2021
4199dc4
Undo changes.
kkrizka Dec 16, 2021
755a8de
Readd Apache Arrow with fixed path.
kkrizka Dec 16, 2021
6307267
test
kkrizka Dec 16, 2021
848ef7f
Escape plus size.
kkrizka Dec 17, 2021
119741d
test
kkrizka Dec 17, 2021
ccb3cf1
test
kkrizka Dec 17, 2021
d4ff5d2
test
kkrizka Dec 17, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 131 additions & 0 deletions .github/workflows/build-and-test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
name: Build Docker Images
on:
push:
branches:
- 'master'
tags:
- 'v*'
pull_request:
branches:
- 'master'
jobs:
#
# Build and deploy the docker images
build:
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
-
name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
-
name: Docker Meta
id: meta
uses: docker/metadata-action@v3
with:
images: ${{ secrets.DOCKERHUB_REPOSITORY }}/mucoll-dummy
-
name: Run checks
run: |
pwd
ls
docker --version
-
name: Build and Push (mucoll-environment)
id: docker_build_mucoll-environment
uses: docker/build-push-action@v2
with:
push: true
context: ./CentOS8
file: ./CentOS8/Dockerfile-environment
tags: ${{ secrets.DOCKERHUB_REPOSITORY }}/mucoll-environment:${{steps.meta.outputs.version}}-centos8
-
name: Build and Push (mucoll-ilc-base)
id: docker_build_mucoll-ilc-base
uses: docker/build-push-action@v2
with:
push: true
context: ./CentOS8
file: ./CentOS8/Dockerfile-ilc-base
build-args: |
VERSION=${{ steps.meta.outputs.version }}
REPOSITORY=${{ secrets.DOCKERHUB_REPOSITORY }}
tags: ${{ secrets.DOCKERHUB_REPOSITORY }}/mucoll-ilc-base:${{steps.meta.outputs.version}}-centos8
-
name: Build and Push (mucoll-ilc-framework)
id: docker_build_mucoll-ilc-framework
uses: docker/build-push-action@v2
with:
push: true
context: ./CentOS8
file: ./CentOS8/Dockerfile-ilc-framework
build-args: |
VERSION=${{ steps.meta.outputs.version }}
REPOSITORY=${{ secrets.DOCKERHUB_REPOSITORY }}
tags: ${{ secrets.DOCKERHUB_REPOSITORY }}/mucoll-ilc-framework:${{steps.meta.outputs.version}}-centos8,${{ secrets.DOCKERHUB_REPOSITORY }}/mucoll-ilc-framework:${{github.sha}}-centos8
#
# Generate test events
sim:
needs: build
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
-
id: cache-sim
name: Cache Event Simulation
uses: actions/cache@v2
with:
path: ./muonGun_sim_MuColl_v1.slcio
key: events-${{ hashFiles('test/sim_steer_muonGun_MuColl_v1.py') }}
- name: Simulate Events
if: steps.cache-sim.outputs.cache-hit != 'true'
run: |
docker run -v ${PWD}:/var/myrun -w /var/myrun ${{ secrets.DOCKERHUB_REPOSITORY }}/mucoll-ilc-framework:${{github.sha}}-centos8 /bin/bash -c "
source /opt/ilcsoft/muonc/init_ilcsoft.sh &&
ddsim --steeringFile test/sim_steer_muonGun_MuColl_v1.py
"
#
# Test ACTS
proc:
needs: sim
runs-on: ubuntu-latest
strategy:
matrix:
steer:
- /opt/ilcsoft/muonc/ACTSTracking/*/share/ACTSTracking/example/actstruth_steer.xml
- /opt/ilcsoft/muonc/ACTSTracking/*/share/ACTSTracking/example/actstruthckf_steer.xml
- /opt/ilcsoft/muonc/ACTSTracking/*/share/ACTSTracking/example/actsseed_steer.xml
- /opt/ilcsoft/muonc/ACTSTracking/*/share/ACTSTracking/example/actsseedckf_steer.xml
steps:
-
name: Checkout
uses: actions/checkout@v2
with:
fetch-depth: 0
-
name: Cache Event Simulation
uses: actions/cache@v2
with:
path: ./muonGun_sim_MuColl_v1.slcio
key: events-${{ hashFiles('test/sim_steer_muonGun_MuColl_v1.py') }}
-
name: Run ACTSTracking example
run: |
docker run -v ${PWD}:/var/myrun -w /var/myrun ${{ secrets.DOCKERHUB_REPOSITORY }}/mucoll-ilc-framework:${{github.sha}}-centos8 /bin/bash -c "
source /opt/ilcsoft/muonc/init_ilcsoft.sh &&
Marlin ${{ matrix.steer }} --global.LCIOInputFiles=muonGun_sim_MuColl_v1.slcio
"
10 changes: 7 additions & 3 deletions CentOS8/Dockerfile-environment
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
###############################################################################
# Repository: mucoll-environment
# Tag: 1.5-centos8
# Image: mucoll-environment
###############################################################################

FROM centos:centos8
Expand Down Expand Up @@ -31,6 +30,12 @@ RUN yum -y install vim-enhanced xorg-x11-xauth screen git subversion rsync make
root-geom root-graf3d-eve7 root-gui-browserv7 root-tpython \
root-unuran root-minuit2 root-smatrix gsl-devel geant4-devel clhep-devel HepMC3-devel

## Apache Arrow (https://arrow.apache.org/install/)
RUN yum install -y https://apache.jfrog.io/artifactory/arrow/centos/$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1)/apache-arrow-release-latest.rpm && \
yum install -y arrow-devel arrow-glib-devel \
arrow-dataset-devel \
parquet-devel parquet-glib-devel

## all datasets in the container !!!!
RUN geant4-dataset-download

Expand All @@ -42,4 +47,3 @@ RUN alternatives --set python /usr/bin/python3 && \
COPY c++config.h /usr/include/c++/8/x86_64-redhat-linux/bits/c++config.h
COPY c++config_32.h /usr/include/c++/8/x86_64-redhat-linux/32/bits/c++config.h


11 changes: 6 additions & 5 deletions CentOS8/Dockerfile-ilc-base
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
###############################################################################
# Repository: mucoll-ilc-base
# Tag: 1.5-centos8
# Image: mucoll-ilc-base
###############################################################################

FROM infnpd/mucoll-environment:1.5-centos8
ARG VERSION=devel
ARG REPOSITORY=infnpd
FROM ${REPOSITORY}/mucoll-environment:${VERSION}-centos8

RUN cd /opt && git clone https://github.com/MuonColliderSoft/MuonCutil.git && \
RUN cd /opt && git clone https://github.com/kkrizka/MuonCutil.git && \
git clone https://github.com/MuonColliderSoft/iLCInstall.git && \
cd /opt/MuonCutil && git checkout 0769c7c790d47a71f7a9a012dcbfb1d2a23d34dd && \
cd /opt/MuonCutil && git checkout officialacts && \
cd /opt/iLCInstall && git checkout aee417fceb83b6f1892bdd492dca8e358a520c5c && \
sed -i 's|/usr/bin/env python|/usr/bin/env python2|g' ilcsoft-install && \
mkdir releases/development && \
Expand Down
9 changes: 4 additions & 5 deletions CentOS8/Dockerfile-ilc-framework
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
###############################################################################
# Repository: mucoll-ilc-framework
# Tag: 1.5-centos8
# Image: mucoll-ilc-framework
###############################################################################

FROM infnpd/mucoll-ilc-base:1.5-centos8
ARG VERSION=devel
ARG REPOSITORY=infnpd
FROM ${REPOSITORY}/mucoll-ilc-base:${VERSION}-centos8

RUN cd /opt/iLCInstall && \
./ilcsoft-install releases/development/release-ilcsoft.cfg --install-prefix=/opt/ilcsoft -v -i
Expand All @@ -15,5 +16,3 @@ RUN cd /opt/ilcsoft/muonc && \
RUN sed -i -e 's|/bin/env python|/usr/bin/env python2|g' /opt/ilcsoft/muonc/CEDViewer/*/bin/ced2go

COPY profile.d_ilcsoft.sh /etc/profile.d/ilcsoft.sh


8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ Docker files for Muon Collider software
The incremental images for the framework can be built with the following command:
```
cd CentOS8
docker build -f Dockerfile-environment -t infnpd/mucoll-environment:1.5-centos8 .
docker build -f Dockerfile-ilc-base -t infnpd/mucoll-ilc-base:1.5-centos8 .
docker build -f Dockerfile-ilc-framework -t infnpd/mucoll-ilc-framework:1.5-centos8 .
docker build -f Dockerfile-environment -t infnpd/mucoll-environment:devel-centos8 .
docker build -f Dockerfile-ilc-base -t infnpd/mucoll-ilc-base:devel-centos8 .
docker build -f Dockerfile-ilc-framework -t infnpd/mucoll-ilc-framework:devel-centos8 .
```

A useful way to test the image is running a shell on the container:
```
docker run -ti --rm --entrypoint /bin/bash infnpd/mucoll-ilc-framework:1.5-centos8
docker run -ti --rm --entrypoint /bin/bash infnpd/mucoll-ilc-framework:devel-centos8
```
and inspect its content.
Loading