Add user-configurable environment variable passthrough

[asmacdo]

Summary

When a job is passed to SLURM via sbatch, the job is not executed in the same environment. Any environment variables needed by the job need to be explicitly included in the --export option to sbatch.

Additional details

In my case, the issue is the use of an environment variable for the location of miniconda.

I was able to hardcode the location of miniconda into the script preamble, but the MINICONDA_PATH variable is not available unless included in the sbatch command.

call_test_job.py

#!/bin/bash                                                                                                                                
#SBATCH --mem=2G                                                                                                                           
                                                                                                                                           
# Script preambles:                                                                                                                        
echo "miniconda path is: ${MINICONDA_PATH}" && \                                                                                           
. "$MINICONDA_PATH/etc/profile.d/conda.sh" && \                                                                                            
conda activate babs                          

Currently this is what is scaffolded by babs:

sbatch --export=DSLOCKFILE=/home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/.SLURM_datalad_lock --job-name toy_test_job -e /home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/logs/toy_test_job.e%A -o /home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/logs/toy_test_job.o%A /home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/code/check_setup/call_test_job.sh

This is what I'd like it to be:

sbatch --export=DSLOCKFILE=/home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/.SLURM_datalad_lock,MINICONDA_PATH=/home/austin/miniconda3 --job-name toy_test_job -e /home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/logs/toy_test_job.e%A -o /home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/logs/toy_test_job.o%A /home/austin/devel/babs/.testdata/babs_test_project/test_project/analysis/code/check_setup/call_test_job.sh

Next steps

I think it would be straightforward to add to the config.yaml environment variables to pass through to the job. I'm not sure if this issue is present on SGE, so they could be slurm_passthrough_variables or similar.

config_toybidsapp.yaml

slurm_passthrough_variables:
    - MINICONDA_PATH  # read the value from the current env and pass into the sbatch command

Which would be consumed by generate_job_submit_template function in babs/babs.py

# Pseudocode
for env_var in system.env_vars:  # env vars could live in the `System` class?
    env_flags += f",{env_var}={os.getenv('{env_var}')