It sounds double, only concern is trying to avoid the proliferation of magic env variables. What about adding a option in the config file eg. singularity.cacheReadOnly. ?

Understandable 👍🏻 I'd prefer an env variable if possible though.. In our use case I'm thinking of having @nf-core pipelines set up as environment modules on the cluster, so that people can do module load nf-core-rnaseq on our secure offline and have everything that they need. An env variable is easy to do with this, a nextflow config option is maybe doable but would require workarounds and hacks. Different users will be doing this so can't have it in the ~/.nextflow/config file and would rather avoid having to edit the pipeline source code if possible.

Then we can have a compromise and have both 😄

It would be enough adding a similar method getLibraryDir

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Still planning to do this, one day (and if not new then would still be great if someone did it!)

Was reading this again, I think the real point is having a flag (read env var) that allows enabling/disabling the pull of singularity containers. Agree on that?

I'd prefer it as an additional location: check if the container I need is in the read-only location, fetch it to the write-location if not.. So then needs to be a little more than just a flag.

Therefore I may not fully understand your use. You are planning to have two locations for singularity images:

1. NXF_SINGULARITY_LIBRARY: NF checks this path , if the image exists take, otherwise point 2:
2. NXF_SINGULARITY_CACHEDIR: NF check if the image exists, yes => run it, no => pull, store in cachedir, run it.

Make sense?

Yes, exactly 👍🏻

So NXF_SINGULARITY_CACHEDIR is unchanged from its current behaviour. NXF_SINGULARITY_LIBRARY is a new additional read-only second location that is checked first, as you say.

ps. Whilst we're here, could even do something similar for $NXF_ASSETS and have a $NXF_ASSETS_LIBRARY. Though in our use case we would like to include the pipeline tag in the directory structure so that multiple versions can coexist, which doesn't currently work with the default assets file path structure.

This was implemented by #2381. Regarding the assets directory, not sure it should follow the same pattern. The logic behind is much more complex, we should check the impact of that before.

Sounds good 👍🏻 The assets thing was more of a spur of the moment thought. I'm happy for that to be forgotten for now.

hi @ewels @pditommaso I see this is marked as completed and also in the associated ticket #1706

However it sounds as if this is only implemented currently via an environment variable NXF_SINGULARITY_LIBRARYDIR ; is that the case?

This is really problematic for us because its far more difficult (and far less ideal) to police users' environment variables inside the HPC to try to make sure their Nextflow runs are configured correctly to run on our systems. It would be much preferred if this same configuration was possible to be set via a nextflow.config setting. That would allow us to simply provide our users with a supplemental nextflow.config file which they could use with nextflow -c our-settings.config run main.nf ... in order to easily use our systems. Currently I do not see any mention of NXF_SINGULARITY_LIBRARYDIR in the configs for Signularity here;

https://nextflow.io/docs/latest/reference/config.html#singularity

is this the case or would we need a separate Issue to request graduating this to a config option instead of just an env var?

This seems to be a documentation bug (which is odd, I believe I've seen it there earlier). singularity.libraryDir can be used instead of the environment variable. E.g. https://nextflow.io/docs/stable/container.html also mentions it.

Yup, I'm also pretty sure that the config option exists. @christopher-hakkaart would you mind taking a look at this if you get a second please?

Having some kind of automated checks (or auto-generated docs?) to make sure that every config option is documented would be a great thing to look into once we have a new config parser @bentsherman

Thanks all; I'll add it to my sprint starting next week. If it was available earlier it might have been accidentally dropped when we did the docs restructure. Either way, it should be quick to add it back in. I'll update here with the PR once it's ready.