PSA - How to Use Custom Batch Configuration (for 'pine' folder work-around)

[wasciutto]

clpipe can be adapted to various batch compute systems via its batch configuration file. clpipe comes pre-loaded with such configuration files for institutions like UNC, Pitt, Duke, etc.

UNC is the default configuration profile, so users of clpipe on Longleaf may not be as familiar with the configuration as those who have to deal with it by necessity. However, since Longleaf has made a change to disable pine as an available folder, clpipe users on that platform will need to interact with this configuration in order to use certain commands (namely preprocessing/fmriprep_process and bids_validate) with versions of clpipe older than 1.9. This is because the older versions all list pine as a folder that should exist, which causes an error when used with Singularity containers.

Tucked away at the end of your configuration file is a field called "batch_config," pointing to the UNC batch configuration:

    },
    "RunLog": [],
    "StatusCache": "",
    "BatchConfig": "slurmUNCConfig.json"
}

The path "slurmUNCConfig.json" gets resolved as a file internal to the clpipe package, meaning it isn't something you would normally interact with yourself. However, you can repoint this path to your own config file by using a full path like this:

    },
    "RunLog": [],
    "StatusCache": "",
    "BatchConfig": "/proj/path/to/my/clpipe/project/custom_config.json"
}

Currently, we don't advertise how to make this file yourself very well (an update is in the works there), so here is the default UNC template that you can start with:

{
  "SubmissionHead": "sbatch --no-requeue",
  "SubmissionOptions": [{
    "command": "-n",
    "args": "1"
  }],
  "SubOptionsEqual": [],
  "NThreadsCommand": "--cpus-per-task={nthreads}",
  "NThreads": "1",
  "MemoryCommand": "--mem={mem}",
  "MemoryDefault": "1000",
  "TimeCommand": "--time={time}",
  "TimeDefault": "1:0:0",
  "JobIDCommand": "--job-name=\"{jobid}\"",
  "OutputCommand": "--output={output}",
  "CommandWrapper": "--wrap=\"{cmdwrap}\"",
  "EmailAddress": "",
  "EmailCommand": "--mail-user {email} --mail-type=FAIL",
  "FMRIPrepBatchCommands": "-e --no-home",
  "NoQuotes": false,
  "TimeCommandActive": true,
  "ThreadCommandActive": true,
  "JobIDCommandActive": true,
  "OutputCommandActive": true,


  "SingularityBindPaths": "/proj,/work,/nas02,/nas"
}

Note the last key: SingularityBindPaths. This field allows you to specify what bind paths Singularity uses (essentially what folders in Longleaf containerized programs like fmriprep are allowed to use). If you are on an older version of clpipe and on Longleaf, you should redirect your batch config to a new file like above, which has /pine removed from the Singularity bind path.

You can also use this file to customize other things on Longleaf - for example, if you would like to submit your batch jobs under a group other than your default (such as if you need to process for another lab than your own), you could modify the "SubmissionHead": "sbatch --no-requeue", section to include an account, like this: `"SubmissionHead": "sbatch --no-requeue --account my_other_group_psx"