Replace restricted confound sets with config files (PennLINC#1255)

.circleci/config.yml

@@ -40,21 +40,38 @@ jobs:
     steps:
       - checkout
       - restore_cache:
-          key: ds001419-fmriprep-08
+          key: ds001419-10
       - run: *runinstall
       - run:
-          name: Download ds001419-fmriprep test data
+          name: Download ds001419 fMRIPrep test data
           command: |
             cd /src/xcp_d/.circleci
             python get_data.py $PWD/data ds001419
       - save_cache:
-          key: ds001419-fmriprep-08
+          key: ds001419-10
           paths:
-            - /src/xcp_d/.circleci/data/ds001419-fmriprep
+            - /src/xcp_d/.circleci/data/ds001419
     # The resource_class feature allows configuring CPU and RAM resources for each job. Different resource classes are available for different executors. https://circleci.com/docs/2.0/configuration-reference/#resourceclass
     # Why do we need a big executor for this job?
     resource_class: large
 
+  download_data_ds001419_aroma:
+    <<: *dockersetup
+    steps:
+      - checkout
+      - restore_cache:
+          key: ds001419-aroma-02
+      - run: *runinstall
+      - run:
+          name: Download ds001419 fMRIPost-AROMA test data
+          command: |
+            cd /src/xcp_d/.circleci
+            python get_data.py $PWD/data ds001419-aroma
+      - save_cache:
+          key: ds001419-aroma-02
+          paths:
+            - /src/xcp_d/.circleci/data/ds001419-aroma
+
   download_data_pnc:
     <<: *dockersetup
     steps:
@@ -227,7 +244,9 @@ jobs:
               circleci step halt
             fi
       - restore_cache:
-          key: ds001419-fmriprep-08
+          key: ds001419-10
+      - restore_cache:
+          key: ds001419-aroma-02
       - run: *runinstall
       - run:
           name: Run full xcp_d on nifti with freesurfer
@@ -269,7 +288,7 @@ jobs:
               circleci step halt
             fi
       - restore_cache:
-          key: ds001419-fmriprep-08
+          key: ds001419-10
       - run: *runinstall
       - run:
           name: Run full xcp_d on cifti with freesurfer
@@ -481,7 +500,7 @@ jobs:
       - restore_cache:
           key: pnc-02
       - restore_cache:
-          key: ds001419-fmriprep-08
+          key: ds001419-10
       - restore_cache:
           key: fmriprepwithoutfreesurfer-03
       - restore_cache:
@@ -594,6 +613,13 @@ workflows:
             tags:
               only: /.*/
 
+      - download_data_ds001419_aroma:
+          requires:
+            - build
+          filters:
+            tags:
+              only: /.*/
+
       - download_data_pnc:
           requires:
             - build
@@ -658,6 +684,7 @@ workflows:
       - ds001419_nifti:
           requires:
             - download_data_ds001419
+            - download_data_ds001419_aroma
           filters:
             branches:
               ignore:

docs/outputs.rst

@@ -315,14 +315,15 @@ Other outputs include quality control, framewise displacement, and confounds fil
 
       sub-<label>/[ses-<label>/]
          func/
-            <source_entities>[_desc-filtered]_motion.tsv
+            <source_entities>_motion.tsv
             <source_entities>_outliers.tsv
             <source_entities>_design.tsv
             <source_entities>_space-<label>_desc-linc_qc.tsv
 
-``[desc-filtered]_motion.tsv`` is a tab-delimited file with seven columns:
+``_motion.tsv`` is a tab-delimited file with seven columns:
 one for each of the six filtered motion parameters, as well as "framewise_displacement".
-If no motion filtering was applied, this file will not have the ``desc-filtered`` entity.
+If motion filtering was applied, this file will seven extra columns: the seven described above,
+with ``_filtered`` appended to each column.
 This file includes the high-motion volumes that are removed in most other derivatives.
 
 ``outliers.tsv`` is a tab-delimited file with one column: "framewise_displacement".

docs/usage.rst

@@ -370,38 +370,39 @@ Therefore, once a user specifies the container options and the image to be run,
 the command line options are the same as the *bare-metal* installation.
 
 
-.. _usage_custom_confounds:
-
 ****************
 Custom Confounds
 ****************
 
-XCP-D can include custom confounds in its denoising.
-Here, you can supply your confounds, and optionally add these to a confound strategy already
-supported in XCP-D.
-
-To add custom confounds to your workflow, use the ``--custom-confounds`` parameter,
-and provide a folder containing the custom confounds files for all of the subjects, sessions, and
-tasks you plan to post-process.
+If you would like to denoise your data with confounds that are not included in the built-in
+confound strategies, you can create your own.
+Please see :ref:`confound_config` for more info on the configuration file format.
 
-The individual confounds files should be tab-delimited, with one column for each regressor,
-and one row for each volume in the data being denoised.
+If you have a set of custom confounds that you would like to use, you need to organize them into a
+BIDS dataset.
+The dataset should only require the actual confound files, preferably with associated sidecar
+JSONs, and a ``dataset_description.json`` file.
+Using the same filenames as the fMRIPrep confound files is an easy way to organize this dataset.
 
 
 Including Signal Regressors
 ===========================
 
+.. warning::
+   Signal regressors are not currently supported in combination with voxel-wise regressors.
+
+.. warning::
+   Please be careful when including signal regressors.
+   Most of the time this is probably a bad idea.
+   For example, if you run tedana, only noise components from tedana should be orthogonalized
+   with respect to signal components.
+   You wouldn't want to orthogonalize other noise signals (e.g., motion parameters) with respect
+   to the signal components from tedana.
+
 Let's say you have some nuisance regressors that are not necessarily orthogonal to some associated
 regressors that are ostensibly signal.
-For example, if you ran `tedana <https://tedana.readthedocs.io/en/stable/>`_ on multi-echo data,
-you would have a series of "rejected" (noise) and "accepted" (signal) ICA components.
-Because tedana uses a spatial ICA, these components' time series are not necessarily independent,
-and there can be shared variance between them.
-If you want to properly denoise your data using the noise components,
-you need to account for the variance they share with the signal components.
-
-XCP-D allows users to include the signal regressors in their custom confounds file,
-so that the noise regressors can be orthogonalized with respect to the signal regressors.
+For example, if you want to denoise your data while still retaining task signal,
+you could include the predicted task signals in your confounds.
 
 For more information about different types of denoising,
 see `tedana's documentation <https://tedana.readthedocs.io/en/latest/denoising.html>`_,
@@ -415,23 +416,18 @@ them without modification,
 you should include those regressors in your custom confounds file,
 with column names starting with ``signal__`` (lower-case "signal", followed by two underscores).
 
-.. important::
-
-   XCP-D will automatically orthogonalize noise regressors with respect to signal regressors
-   with any nuisance-regressor option that uses AROMA regressors (e.g., ``aroma`` or ``aroma_gsr``).
-
 
 Task Regression
 ===============
 
-If you want to regress task-related signals out of your data, you can use the custom confounds
-option to do it.
+If you want to regress task-related signals out of your data, you can use a custom confound
+configuration.
 
 Here we document how to include task effects as confounds.
 
 .. tip::
    The basic approach to task regression is to convolve your task regressors with an HRF,
-   then save those regressors to a custom confounds file.
+   then save those regressors to a confounds file.
 
 .. warning::
    This method is still under development.
@@ -471,31 +467,61 @@ plot_design_matrix.html#create-design-matrices>`_.
       index=False,
    )
 
-Then, when you run XCP-D, you can use the flag
-``--custom_confounds /my/project/directory/custom_confounds``.
+Then, create a confounds config file to include derivatives from ``custom_confounds``.
+Something like this should work:
+
+```yaml
+name: my_custom_confounds
+description: |
+   Nuisance regressors were task regressors convolved with an HRF and motion parameters.
+confounds:
+   motion:
+      dataset: preprocessed
+      query:
+         space: null
+         cohort: null
+         res: null
+         den: null
+         desc: confounds
+         extension: .tsv
+         suffix: timeseries
+      columns:
+      - trans_x
+      - trans_y
+      - trans_z
+      - rot_x
+      - rot_y
+      - rot_z
+   task:
+      dataset: custom
+      query:
+         space: null
+         cohort: null
+         res: null
+         den: null
+         desc: confounds
+         extension: .tsv
+         suffix: timeseries
+      columns:
+      - condition1
+      - condition2
+```
 
 
 Command Line XCP-D with Custom Confounds
 ========================================
 
-Last, supply the file to xcp_d with the ``--custom_confounds`` option.
-``--custom_confounds`` should point to the directory where this file exists, rather than to the
-file itself;
-XCP-D will identify the correct file based on the filename,
-which should match the name of the preprocessed BOLD data's associated confounds file.
-You can simultaneously perform additional confound regression by including,
-for example, ``--nuisance-regressors 36P`` in the call.
+Last, run XCP-D with your custom configuration file and the path to the custom derivatives dataset.
 
 .. code-block:: bash
 
-   apptainer run --cleanenv -B /my/project/directory:/mnt xcpabcd_latest.simg \
+   apptainer run --cleanenv -B /my/project/directory:/mnt xcpd_latest.sif \
       /mnt/input/fmriprep \
       /mnt/output/directory \
       participant \
       --participant_label X \
-      --task-id Z \
-      --nuisance-regressors 36P \
-      --custom_confounds /mnt/custom_confounds
+      --derivatives custom=/mnt/custom_confounds \
+      --nuisance-regressors /mnt/custom_config.yaml
 
 
 ********************