Skip to content

clpipe v1.6.0
Compare
Choose a tag to compare
@wasciutto wasciutto released this 23 Aug 14:41
· 1047 commits to master since this release
v1.6.0.3
7ef8175
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

CLI Update

A subset of commands have been gathered under one umbrella command, clpipe. Running the clpipe command with no arguments will show the user the various processing steps provided by clpipe:

Usage: clpipe [OPTIONS] COMMAND [ARGS]...

  Welcome to clpipe. Please choose a processing command.

Options:
  -v, --version  Display clpipe's version.
  --help         Show this message and exit.

Commands:
  bids_validate      Check that the given directory conforms to the BIDS...
  convert2bids       Convert DICOM files to BIDS format
  fmri_postprocess   Additional preprocessing for connectivity analysis
  fmri_postprocess2  Perform additional processing on fMRIPrepped data
  fmriprep_process   Submit BIDS-formatted images to fMRIPrep
  glm                GLM Commands
  project_setup      Set up a clpipe project

The command bids_validate, for example, can be accessed like this:

clpipe bids_validate --help

Usage: clpipe bids_validate [OPTIONS] [BIDS_DIR]

  Check that the given directory conforms to the BIDS standard

Options:
  -config_file FILE  Uses a given configuration file
  -log_dir FILE      Where to put HPC output files (such as SLURM output
                     files)
  -verbose           Creates verbose validator output. Use if you want to see
                     ALL files with errors/warnings.
  -submit            Flag to submit commands to the HPC
  -interactive       Run in an interactive session. Only use in an interactive
                     compute session.
  -debug             Flag to enable detailed error messages and traceback
  --help             Show this message and exit.

The GLM commands have been grouped into the sub-command glm, and can be accessed like this:
clpipe glm

Usage: clpipe glm [OPTIONS] COMMAND [ARGS]...

  GLM Commands

Options:
  --help  Show this message and exit.

Commands:
  apply_mumford_workaround  Apply the Mumford registration workaround to...
  fsl_onset_extract         Convert onset files to FSL's 3 column format
  l1_prepare_fsf            Propagate an .fsf file template for L1 GLM...
  l2_prepare_fsf            Propagate an .fsf file template for L2 GLM...
  launch                    Launch all prepared .fsf files for L1 or L2...
  report_outliers           Generate a confound outliers report.
  setup                     Prepare task images and confound files for...

Further clpipe commands will be wrapped into this interface with future updates.
The interface currently does not support auto-complete for sub commands - this is possible, and can be included in a future release.
The stand-alone commands such as bids_validate remain available for use outside of the clpipe command like in previous versions of clpipe.

GLM Updates

GLM Launchers

Launchers have been added clpipe's GLM line, allowing .fsf files generated with l1_prepare_fsfs and l2_prepare_fsfs to be launched automatically using your GLM configuration.

Usage: glm_l1_launch [OPTIONS]

  Launch all prepared .fsf files for L1 GLM analysis

Options:
  -glm_config_file FILE  Use a given GLM configuration file.  [required]
  -l1_name TEXT          Name of your L1 model  [required]
  -test_one              Only submit one job for testing purposes.
  -submit                Flag to submit commands to the HPC
  -debug                 Flag to enable detailed error messages and traceback
  --help                 Show this message and exit.

Usage: glm_l2_launch [OPTIONS]

  Launch all prepared .fsf files for L2 GLM analysis

Options:
  -glm_config_file FILE  Use a given GLM configuration file.  [required]
  -l2_name TEXT          Name of your L2 model  [required]
  -test_one              Only submit one job for testing purposes.
  -submit                Flag to submit commands to the HPC
  -debug                 Flag to enable detailed error messages and traceback
  --help                 Show this message and exit.

Stand-alone Mumford Workaround Command

The logic of the Mumford Workaround (see here), normally applied by clpipe at L2, can now be applied to an L1 folder manually. This helps with L3 issues in the case where some subjects have L2, and some don't, causing the subjects with only L1s to not have the Mumford Workaround applied.

clpipe glm apply_mumford_workaround --help

Usage: clpipe glm apply_mumford_workaround [OPTIONS]

  Apply the Mumford registration workaround to L1 FEAT folders.  Applied by
  default in glm-l2-preparefsf.

Options:
  -glm_config_file FILE           Location of your GLM config file.
  -l1_feat_folders_path DIRECTORY
                                  Location of your L1 FEAT folders.
  --help                          Show this message and exit.

Outlier Reporting

A new command has also been provided to report on the number of outliers each subject has after running through glm_setup or fmri_postprocess2. The logic of this command will likely be integrated into fmri_postprocess2 in a future update, but it is available as a stand-alone command for now as:

clpipe glm report_outliers --help

Usage: clpipe glm report_outliers [OPTIONS]

  Generate a confound outliers report.

Options:
  --confounds_dir DIRECTORY  Path to a directory containing subjects and
                             confounds files.
  --confounds_file FILE      Path to confounds file
  --output_file FILE         Path to save outlier count results.
  --confound_suffix TEXT     Confound file to search for, like 'confounds.tsv'
  --help                     Show this message and exit.

Logging & Display Updates

New Combined-view Log File

A new log file, logs/clpipe.log, will now be generated that shows the outputs of main line (setup, convert2bids, validate, fmriprep, postproc) commands all in one place. This makes it easier to help track what commands have been previously run for your project.

image

General Verbosity Improvement

The amount of information displayed by all of the main line commands has been expanded, both in and out of debug mode, making it easier to tell where your outputs are going and what particular settings have been used.

Here is an example from the L2 launcher:

Setting up L2 .fsf launch using model: emot
Targeting .fsfs in dir: ...projects/MAP/data_GLM/data_GLM_emot/level2_emot/design_files
Output dir: .../projects/MAP/data_GLM/data_GLM_emot/level2_emot/gfeat
Using log dir: .../MAP/data_GLM/data_GLM_emot/level2_emot/gfeat
Batch job output path: .../MAP/data_GLM/data_GLM_emot/level2_emot/gfeat

These log lines are also printed to the logs/clpipe.log file

Improved Submission Preview

The submission preview display, shown when running most commands without submit, has been improved to be more readable. Tabs and new lines space out the individual commands, and additional commands beyond the first 5 are folded up as a simple tally. You can still see all commands by including -debug.

Jobs to run:

        sbatch --no-requeue -n 1 --mem=5G --time=5:00:00 --cpus-per-task=4 --job-name="sub-0081" --output=.../data_GLM/data_GLM_emot/level2_emot/gfeat/Output-sub-0081-jobid-%j.out --wrap="feat .../data_GLM/data_GLM_emot/level2_emot/design_files/sub-0081.fsf"

        sbatch --no-requeue -n 1 --mem=5G --time=5:00:00 --cpus-per-task=4 --job-name="sub-0132" --output=.../data_GLM/data_GLM_emot/level2_emot/gfeat/Output-sub-0132-jobid-%j.out --wrap="feat .../data_GLM/data_GLM_emot/level2_emot/design_files/sub-0132.fsf"

        sbatch --no-requeue -n 1 --mem=5G --time=5:00:00 --cpus-per-task=4 --job-name="sub-1011" --output=.../data_GLM/data_GLM_emot/level2_emot/gfeat/Output-sub-1011-jobid-%j.out --wrap="feat .../data_GLM/data_GLM_emot/level2_emot/design_files/sub-1011.fsf"

        sbatch --no-requeue -n 1 --mem=5G --time=5:00:00 --cpus-per-task=4 --job-name="sub-0231" --output=.../data_GLM/data_GLM_emot/level2_emot/gfeat/Output-sub-0231-jobid-%j.out --wrap="feat .../data_GLM/data_GLM_emot/level2_emot/design_files/sub-0231.fsf"

        sbatch --no-requeue -n 1 --mem=5G --time=5:00:00 --cpus-per-task=4 --job-name="sub-0201" --output=.../data_GLM/data_GLM_emot/level2_emot/gfeat/Output-sub-0201-jobid-%j.out --wrap="feat .../data_GLM/data_GLM_emot/level2_emot/design_files/sub-0201.fsf"

        ...and 3 more job(s).
Re-run with the '-submit' flag to launch these jobs.

Under the Hood Changes

  • Main line commands no longer eat uncaught exceptions when not in debug mode
  • Logging is now initialized at the top init.py file
  • Each main line command is now a sub-logger of the main clpipe logger, including batch-manager
  • Each of the main line commands have been refactored for readability, including shortening the line lengths and moving constants to the top of the module.
  • Created centralized configuration module
Assets 2
Loading