Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avoid repeating to build SIF image for Singularity: use local SIF image if an env variable LOCAL_SIF_PATH is set #30

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

szcf-weiya
Copy link

@szcf-weiya szcf-weiya commented Dec 27, 2023

When I run the program in the cluster with Singularity, I found that it takes a long time to build a SIF image. And for another run, it again takes a long time to build image, it seems that the cached image in .singularity folder does not help. Here is the running time (330s)

> library(dyno)     
> x = matrix(rpois(9900, 1), nrow = 99)                                                                                                                                               
>     rownames(x) = 1:nrow(x)                                                                                                                                                         
>     colnames(x) = 1:ncol(x)                                                                                                                                                         
>     dataset = wrap_expression(counts = x, expression = log(x+1))     
> system.time({infer_trajectory(dataset, "comp1", debug=T)})                                                                                                                          
Running singularity exec 'docker://dynverse/ti_comp1:v0.9.9.01' echo hi
Running /usr/bin/singularity exec --containall -B \ 
  '/tmp/RtmpzcjE6q/file32c7f55a74b8e9/:/copy_mount,/tmp/RtmpzcjE6q/file32c7f541ac7236/tmp:/tmp2' \ 
  'docker://dynverse/ti_comp1:v0.9.9.01' cp /code/definition.yml /copy_mount/
Executing 'comp1' on '20231227_181629__data_wrapper__jSj89c7coa'        
With parameters: list(dimred = "pca", ndim = 2L, component = 1L) 
...
...
                                                                                                              
Running /usr/bin/singularity exec --containall --pwd /ti/workspace -B \
  '/tmp/RtmpzcjE6q/file32c7f542b877db/ti:/ti,/tmp/RtmpzcjE6q/file32c7f54c14b061/tmp:/tmp2' \
  'docker://dynverse/ti_comp1:v0.9.9.01' bash
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
Getting image source signatures
Copying blob sha256:bd5da474b8baade003eb744e862574f758b269c20cd2bc0d3c6860e1b3b4e08a
Copying blob sha256:844c33c7e6ea19e3f6847e0667befdfb3ef02d6fc735b22c2d070261b6263b97
...
...

INFO:    Using cached SIF image      
   user  system elapsed 
309.030  12.012 330.661 

To avoid repeating to build the SIF image, I introduce an environment variable, LOCAL_SIF_PATH. If the variable is set, then pull the image and save as .sif locally via

singularity pull xxxx.sif docker://

To align with the existing code, if the env variable is unset, then just run as before.

After saving to a local SIF image, we can speed it up significantly --- only around 2 seconds for the same dataset.

devtools::install_github("szcf-weiya/babelwhale")
Sys.setenv(LOCAL_SIF_PATH=getwd())

image

user  system elapsed
  0.865   0.663   1.952

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant