python3 not found; using peppro with bulker

[stolarczyk]

when running examples/meta/peppro_test.yaml with an activated databio/peppro bulker crate pipeline fails:

databio/peppro|~/Uczelnia/UVA/code/peppro$ looper run examples/meta/peppro_test.yaml --compute local
Command: run (Looper version: 0.12.6-dev)
Activating compute package 'local'
Finding pipelines for protocol(s): pro
Known protocols: PRO, pro, GRO, gro, PRO-SEQ, PRO-seq, PROSEQ, proseq, groseq, GROSEQ, GRO-SEQ, GRO-seq
## [1 of 1] test (pro)
Writing script to /Users/mstolarczyk/Desktop/peppro/peppro_test/submission/peppro_test.sub
Job script (n=1; 0.00Gb): /Users/mstolarczyk/Desktop/peppro/peppro_test/submission/peppro_test.sub
Compute node: docker-desktop
Start time: 2020-03-31 20:10:34
env: ‘python3’: No such file or directory

Looper finished
Samples valid for job generation: 1 of 1
Successful samples: 1 of 1
Commands submitted: 1 of 1
Jobs submitted: 1

• installed bulker@dev and re-initialized config
• pulled and activated databio/peppro
• using peppro@master
• updated PEP and pipeline interface to match new PEP/looper standards (this should be unrelated?)

Is there something else I should do to get it running?

[stolarczyk]

interestingly the command works in both activated environment and the host:

[mstolarczyk@MichalsMBP ~]: python3
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 05:52:31) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
[mstolarczyk@MichalsMBP ~]: bulker-activate databio/peppro
Bulker config: /Users/mstolarczyk/bulker_config.yaml
Activating bulker crate: databio/peppro
databio/peppro|~$ python3
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 05:52:31) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()

[nsheff]

which python3?

what's your type python3 ?

[stolarczyk]

databio/peppro|~$ type python3 
python3 is hashed (/Users/mstolarczyk/bulker_crates/databio/peppro/default/python3)

[mstolarczyk@MichalsMBP code]: type python3
python3 is /Library/Frameworks/Python.framework/Versions/3.6/bin/python3

[nsheff]

it's probably related to this:

https://github.com/databio/hub.bulker.io/blob/fa5b0b3d5cabb61b995f29d087135c8c8203186f/databio/peppro.yaml#L7-L8

[nsheff]

This is how I do it...don't activate the crate, then use:

looper run examples/meta/peppro_test.yaml --compute bulker_local

instead.

[nsheff]

ok, but why... ? Could it be that when you do it the above way, then looper is running inside the crate, so env is used to find python3 in this line:

peppro/pipelines/peppro.py

Line 1 in 5e5ada4

#!/usr/bin/env python3

is actually running in a container, which has no python3 because of this:

https://github.com/databio/hub.bulker.io/blob/fa5b0b3d5cabb61b995f29d087135c8c8203186f/bulker/coreutils.yaml#L69-L70

Solutions:

• could move env from coreutils into alpine
• could move the python from a shebang to the inferface command template

[nsheff]

databio/peppro|~$ env python3
env: ‘python3’: No such file or directory
databio/peppro|~$ exit
nsheff@zither:~$ env python3
Python 3.7.5 (default, Nov 20 2019, 09:21:52) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.

but:

databio/peppro|~$ /usr/bin/env python3
Python 3.7.5 (default, Nov 20 2019, 09:21:52) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.

[nsheff]

@jpsmith5 do you have an opinion on these solutions? I think moving env is the way to go...

[jpsmith5]

Okay, sorry. Catching up here. But yes, agree that moving env is the solution.

[j-lawson]

Hi all,

Was there a solution to this problem?

Nathan's solution gives me an error (below) so I'm guessing the format for specifying compute via command line has changed.

Error: ValueError: Could not correctly parse itemized compute specification. Correct format: k1=v1 k2=v2

Ignoring that though, the bulker crate is specified under "compute" in the pipeline interface file but this does not seem to work either. (https://github.com/databio/dnameth_pipelines/blob/dev/piface_rrbs.yaml)

[j-lawson]

Ok so I figured out the new --compute format but even with (I think) the right format, it doesn't work: --compute bulker_crate=databio/methpl

[jpsmith5]

Hey John are you running local or on slurm? Try running it with bulker_slurm as the package if so.

[j-lawson]

Thanks Jason. I'm running it locally.
looper rerun $CODE/aml_e3999/metadata/project_config.yaml -a glass_samples -p local --command-extra="-R" --compute bulker_crate=databio/methpl
So you're saying that if I was running it on Slurm I should use
--compute bulker_slurm=databio/methpl ?

[stolarczyk]

just tested the --compute argument an it worked as expected.

@j-lawson, as you can see, in line 17 of slurm_bulker_template.sub there's a template variable to be filled: {BULKER_CRATE}. So your task is to fill this in. One way to do this is the --compute argument which adds key-value pairs to the compute looper namespace.

Now, if you look in the DIVCFG we use on Rivanna you'll notice that the BULKER_CRATE key is mapped to compute.bulker_crate under adapters. So something like this is required to get the behavior you need:

looper run <cfg_path> -p bulker_slurm --compute bulker_crate=databio/methpl

[nsheff]

@j-lawson the confusion is that looper changed the usage of compute to package after the above posting. So, what I wrote about using --compute should now be passed to --package, and compute does something different. You can look into the new usage here: https://looper.databio.org/en/latest/usage/. So, -p or --package for bulker_slurm like in @stolarczyk's command. I think you shouldn't need to pass the crate with compute at all since it's defined in the piface -- but you can.

[j-lawson]

Thanks. I had a couple other points of confusion. I was using the -p local instead of -p bulker_local. Also I didn't realize there was a divvy config file for the lab that I needed to load/have the path to in DIVCFG. This works for me now.