Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update aspera-client-docker image to use the new download workflow #5

Merged
merged 7 commits into from
Jan 29, 2024
Merged

Update aspera-client-docker image to use the new download workflow #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7a0b8aa
Update aspera-client-docker image to use the new download workflow
sbesson Dec 19, 2023
ab405c3
DockerfileM1
pwalczysko Dec 20, 2023
9cc1c02
Modify Dockerfile cf. Seb's comment
pwalczysko Dec 20, 2023
39b8bf4
Remove DockerfileM1 as now Dockerfile works for both M1 and intel
pwalczysko Dec 20, 2023
4b40389
Merge pull request #1 from pwalczysko/m1
sbesson Dec 20, 2023
bdb8f26
Use Rocky Linux 9 as the base image
sbesson Jan 3, 2024
455b9cc
Bump GitHub Actions version
sbesson Jan 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 11 additions & 13 deletions .github/workflows/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,31 +11,28 @@ jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v4
- name: Build
run: docker build -t aspera-client .
- name: Test
run: docker run --rm aspera-client idr0001 md5sum.txt /tmp/idr0001/
run: docker run --rm aspera-client idr0001-graml-sysgro/md5sum.txt /tmp/idr0001/
upload:
needs: build
runs-on: ubuntu-latest
env:
name: imagedata/download
steps:
- name: Get prefix
id: getprefix
run: |
if [ ! -z ${{ env.name }} ]; then
echo "::set-output name=prefix::${{ env.name }}:"
else
echo "::set-output name=prefix::${{ github.repository }}:"
fi
- name: Get other tags
id: gettags
uses: jupyterhub/action-major-minor-tag-calculator@v1.1.0
uses: jupyterhub/action-major-minor-tag-calculator@v3
with:
githubToken: ${{ secrets.GITHUB_TOKEN }}
prefix: "${{ steps.getprefix.outputs.prefix }}"
prefix: "${{ env.name }}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like I missed there was a breaking change and a : need to be appended as in ome/omero-server-docker@6f31737.

Unfortunately, this has created quite a few unwanted images at https://hub.docker.com/u/imagedata. @jburel @joshmoore do you have the credentials to delete those? I can open a PR for a follow-up 0.4.1 release (unless you'd prefer me to delete the tag).

- name: Docker metadata
id: meta
uses: docker/metadata-action@v4
with:
images: ${{ env.name }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Login to DockerHub
Expand All @@ -45,7 +42,8 @@ jobs:
password: ${{ secrets.DOCKER_PASSWORD }}
if: startsWith(github.ref, 'refs/tags')
- name: Push to Docker Hub
uses: docker/build-push-action@v2
uses: docker/build-push-action@v4
with:
tags: ${{ join(fromJson(steps.gettags.outputs.tags)) }}
labels: ${{ steps.meta.outputs.labels }}
push: ${{ startsWith(github.ref, 'refs/tags') }}
3 changes: 2 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM centos:centos7
FROM --platform=linux/amd64 rockylinux:9
MAINTAINER [email protected]

RUN useradd data
Expand All @@ -19,6 +19,7 @@ ENV ASCP_KEY /home/data/.aspera/connect/etc/asperaweb_id_dsa.openssh
ENV ASCP_LIMIT 40m
ENV ASCP_RETRANSFER_FILE 0
ENV ASCP_PORT 33001
ENV ASCP_USER fasp-public
ENV ASCP_SERVER fasp.ebi.ac.uk
ADD ascp.sh /usr/local/bin/ascp.sh
ENTRYPOINT ["/usr/local/bin/ascp.sh"]
35 changes: 7 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,11 @@ IDR Download
The https://hub.docker.com/r/imagedata/download image can be used to acquire
the original files which were imported into IDR:

docker run -ti --rm -v /tmp/data:/data imagedata/download idr0001 20151116-verified/JL_120731_S6A /data/JL_120731_S6A

Some exemplary source directories are listed below:

| study | source | plate | size |
|---------|-------------------------------------------------------------------------------------------------------------|---------------------------------------------|--------|
| idr0001 | 20151116-verified/JL_120731_S6A | JL_120731_S6A | 99G |
| idr0002 | 20150401-original/chr_cond_screen/plate1_1_013/experiment_descriptor.xml | plate1_1_013 | 164G |
| idr0003 | 201301120/Images/DTT/p1 | DTT p1 | 2.4G |
| idr0005 | 20150520-original-jcb/v2/001/2011-04-19 | Primary_001 | 811M |
| idr0005 | 20150520-original-jcb/v2/Secondary 001a/2011-08-29-plate-001.HTD | Secondary_001a | 811M |
| idr0006 | 20150915-glencoe-jcb/11001-11089/plate 11001_Plate_136/TimePoint_1 | 11001 | 8.3G |
| idr0007 | 20150522-original-jcb/wt/Plate1 | WT_plate1 | 1.5G |
| idr0008 | original-actinome2/Plate10_Actinome2 | Plate10_Actinome2 | 2.7G |
| idr0008 | original-actinome1/Plate10_Actinome1 | Plate10_Actinome1 | 2.4G |
| idr0008 | original-drosophila/Drosophila screen//001B30_S2R | 001B30_S2R | 2.0G |
| idr0009 | 20150507-VSVG/VSVG/0305-01--2007-04-03 | 0305-01--2007-04-03 | 3.0G |
| idr0009 | 20150507-VSVG/VSVG/0001-03--2005-08-01 | 0001-03--2005-08-01 | 3.0G |
| idr0017 | 20151124/14_X-Man_10x/source/2011-11-17_X-Man_LOPAC_X01_LP_S01_1 | AKT1-/-_&_AKT2-/-_LOPAC_Plate_1_Replicate_1 | 25G |
| idr0019 | 20160414-original/22 Cell Lines Sero MSB 2015/Raw images - OME tif files by plate/22_lines_HC_EGF_200145913 | 22_lines_HC_EGF_200145913 | 3.8G |
| idr0020 | 20160414-original/chTOG genetic interaction screen/chTOGsh MAP screen/Alexis 040413 hMAP screen[2056] | 200972429 TOG[2702] | 5.8G |
| idr0028 | 20160818-original/Images/MDA-MB-231_GEFGAP_siGENOME/MDA-MB-231_GEFGAP_siGENOME_1A | MDA-MB-231_siGENOME_1A | 12G |
| idr0028 | 20160818-original/Images/LM2_GEFGAP_ONTARGETPlus/LM2_GEFGAP_ONTARGETPlus_1A | LM2_ONTARGETPlus_1A | 12G |
| idr0028 | 20160818-original/Images/LM2_GEFGAP_siGENOME /LM2_GEFGAP_siGENOME_1A | LM2_siGENOME_1A | 7.3G |
| idr0028 | 20160818-original/Images/MDA-MB-231_GEFGAP_ONTARGETPlus/MDA-MB-231_GEFGAP_ONTARGETPlus_1A | MDA-MB-231_ONTARGETPlus_1A | 11G |
| idr0030 | 20170110-original/200972726[5176]_kinome_YAP_TAZ/MeasurementIndex.ColumbusIDX.xml | 200972726[5176]_kinome_YAP_TAZ | 13G |
| idr0034 | 20170421-original/Nathalie/HipSci Experiment 10 Plate 1__2014-08-21T17_57_35-Measurement1 | experiment_10 | 5.1G |
docker run -ti --rm -v /tmp/data:/data imagedata/download SOURCE TARGET

where SOURCE can be any subfolder under https://ftp.ebi.ac.uk/pub/databases/IDR/. For instance
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️


docker run -ti --rm -v /tmp/data:/data imagedata/download idr0001-graml-sysgro/20151116-verified/JL_120731_S6A /data/JL_120731_S6A

## Docker environment variables

Expand All @@ -40,7 +18,7 @@ Note that you can modify the environment variables in the dockerfile directly in
For example, to increase the download rate to `100 Mbps`, add an environment variable to the `docker run` command line:

```bash
docker run -ti --rm -v /tmp/data:/data -e ASCP_LIMIT=100m imagedata/download idr0001 20151116-verified/JL_120731_S6A /data/JL_120731_S6A
docker run -ti --rm -v /tmp/data:/data -e ASCP_LIMIT=100m imagedata/download idr0001-graml-sysgro/20151116-verified/JL_120731_S6A /data/JL_120731_S6A
```

You can modify additional variables as well
Expand All @@ -51,6 +29,7 @@ You can modify additional variables as well
| ASCP_LIMIT | `-l` | 40m | max_rate |
| ASCP_RETRANSFER_FILE | `-k` | 0 | Transfer file overwrite |
| ASCP_PORT | `-P` | 33001 | ssh-port|
| ASCP_USER | NA | fasp-public | user |
| ASCP_SERVER | NA | fasp.ebi.ac.uk | host |

For a more detailed description of these commands, see the [ascp documentation](https://www.ibm.com/docs/en/aci/3.9.2?topic=ascp-command-reference).
5 changes: 2 additions & 3 deletions ascp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
set -e
set -u

STUDY=$1; shift
SOURCE=$1; shift
TARGET=$1; shift

Expand All @@ -13,12 +12,12 @@ echo /home/data/.aspera/connect/bin/ascp \
-i "$ASCP_KEY" \
-k "$ASCP_RETRANSFER_FILE" \
"$@" \
"$STUDY@$ASCP_SERVER:$SOURCE" "$TARGET"
"$ASCP_USER@$ASCP_SERVER:/pub/databases/IDR/$SOURCE" "$TARGET"

/home/data/.aspera/connect/bin/ascp \
-P$ASCP_PORT \
-l "$ASCP_LIMIT" \
-i "$ASCP_KEY" \
-k "$ASCP_RETRANSFER_FILE" \
"$@" \
"$STUDY@$ASCP_SERVER:$SOURCE" "$TARGET"
"$ASCP_USER@$ASCP_SERVER:/pub/databases/IDR/$SOURCE" "$TARGET"
Loading