Merge branch 'main' into core

.gitignore

@@ -139,3 +139,6 @@ dmypy.json
 # Pyre type checker
 .pyre/
 *pyscript*
+
+# For mac users
+*.DS_Store

CONTENT.md

@@ -0,0 +1,7 @@
+### Contents overview
+
+- :snake: :package: `narps_open/` contains the Python package with all the pipelines logic.
+- :brain: `data/` contains data that is used by the pipelines, as well as the (intermediate or final) results data. Instructions to download data are available in [INSTALL.md](/INSTALL.md#data-download-instructions).
+- :blue_book: `docs/` contains the documentation for the project. Start browsing it with the entry point [docs/README.md](/docs/README.md)
+- :orange_book: `examples/` contains notebooks examples to launch of the reproduced pipelines.
+- :microscope: `tests/` contains the tests of the narps_open package.

CONTRIBUTING.md

@@ -1,15 +1,19 @@
 # How to contribute to NARPS Open Pipelines ? 
 
-General guidelines can be found [here](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) in the GitHub documentation. 
+For the reproductions, we are especially looking for contributors with the following profiles:
+ - 👩‍🎤 SPM, FSL, AFNI or nistats has no secrets for you? You know this fMRI analysis software by heart 💓. Please help us by reproducing the corresponding NARPS pipelines. 👣 after step 1, follow the fMRI expert trail.
+ - 🧑‍🎤 You are a nipype guru? 👣 after step 1, follow the nipype expert trail.
 
-## Reproduce a pipeline :keyboard:
-:thinking: Not sure which one to start with ? You can have a look on [this table](https://github.com/Inria-Empenn/narps_open_pipelines/wiki/pipeline_status) giving the work progress status for each pipeline. This will help choosing the one that best suits you!
+# Step 1: Choose a pipeline to reproduce :keyboard:
+:thinking: Not sure which pipeline to start with ? 🚦The [pipeline dashboard](https://github.com/Inria-Empenn/narps_open_pipelines/wiki/pipeline_status) provides the progress status for each pipeline. You can pick any pipeline that is in red (not started).
 
-Need more information ? You can have a look to the pipeline description [here](https://docs.google.com/spreadsheets/d/1FU_F6kdxOD4PRQDIHXGHS4zTi_jEVaUqY_Zwg0z6S64/edit?usp=sharing). Also feel free to use the `narps_open.utils.description` module of the project, as described [in the documentation](/docs/description.md).
+Need more information to make a decision? The `narps_open.utils.description` module of the project, as described [in the documentation](/docs/description.md) provides easy access to all the info we have on each pipeline.
 
 When you are ready, [start an issue](https://github.com/Inria-Empenn/narps_open_pipelines/issues/new/choose) and choose **Pipeline reproduction**!
 
-### If you have experience with NiPype
+# Step 2: Reproduction 
+
+## 🧑‍🎤 NiPype trail
 
 We created templates with modifications to make and holes to fill to create a pipeline. You can find them in [`narps_open/pipelines/templates`](/narps_open/pipelines/templates).
 
@@ -21,9 +25,9 @@ Feel free to have a look to the following pipelines, these are examples :
 | 2T6S | SPM | Yes | [/narps_open/pipelines/team_2T6S.py](/narps_open/pipelines/team_2T6S.py) |
 | X19V | FSL | Yes | [/narps_open/pipelines/team_X19V.py](/narps_open/pipelines/team_2T6S.py) |
 
-### If you have experience with the original software package but not with NiPype
+## 👩‍🎤 fMRI software trail
 
-A fantastic tool named [Giraffe](https://giraffe.tools/porcupine/TimVanMourik/GiraffePlayground/master) is available. It allows you to create a graph of your pipeline using NiPype functions but without coding! Just save your NiPype script in a .py file and send it as a new issue, we will convert this script to a script which works with our specific parameters. 
+...
 
 ## Find or propose an issue :clipboard:
 Issues are very important for this project. If you want to contribute, you can either **comment an existing issue** or **proposing a new issue**. 
@@ -64,3 +68,7 @@ Once your PR is ready, you may add a reviewer to your PR, as described [here](ht
 Please turn your Draft Pull Request into a "regular" Pull Request, by clicking **Ready for review** in the Pull Request page.
 
 **:wave: Thank you in advance for contributing to the project!**
+
+## Additional resources
+
+ - git and Gitub: general guidelines can be found [here](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) in the GitHub documentation. 

README.md

@@ -15,45 +15,26 @@
         <img src="https://img.shields.io/github/commit-activity/m/Inria-Empenn/narps_open_pipelines" /></a>
 </p>
 
-## Table of contents
-
-- [Project presentation](#project-presentation)
-- [Getting Started](#getting-started)
-	- [Contents overview](#contents-overview)
-	- [Installation](#installation)
-	- [Contributing](#contributing)
-- [References](#references)
-- [Funding](#funding)
-
 ## Project presentation
 
-Neuroimaging workflows are highly flexible, leaving researchers with multiple possible options to analyze a dataset [(Carp, 2012)](https://www.frontiersin.org/articles/10.3389/fnins.2012.00149/full).
-However, different analytical choices can cause variation in the results [(Botvinik-Nezer et al., 2020)](https://www.nature.com/articles/s41586-020-2314-9), leading to what was called a "vibration of effects" [(Ioannidis, 2008)](https://pubmed.ncbi.nlm.nih.gov/18633328/) also known as analytical variability. 
+**The goal of the NARPS Open Pipelines project is to create a codebase reproducing the 70 pipelines of the NARPS study (Botvinik-Nezer et al., 2020) and share this as an open resource for the community**. 
 
-**The goal of the NARPS Open Pipelines project is to create a codebase reproducing the 70 pipelines of the NARPS project (Botvinik-Nezer et al., 2020) and share this as an open resource for the community**. 
+We base our reproductions on the [original descriptions provided by the teams](https://github.com/poldrack/narps/blob/1.0.1/ImageAnalyses/metadata_files/analysis_pipelines_for_analysis.xlsx) and test the quality of the reproductions by comparing our results with the original results published on NeuroVault.
 
-To perform the reproduction, we are lucky to be able to use the [descriptions provided by the teams](https://github.com/poldrack/narps/blob/1.0.1/ImageAnalyses/metadata_files/analysis_pipelines_for_analysis.xlsx).
-We also created a [shared spreadsheet](https://docs.google.com/spreadsheets/d/1FU_F6kdxOD4PRQDIHXGHS4zTi_jEVaUqY_Zwg0z6S64/edit?usp=sharing) that can be used to add comments on pipelines (e.g.: identify the ones that are not reproducible with NiPype).
+:vertical_traffic_light: See [the pipeline dashboard](https://github.com/Inria-Empenn/narps_open_pipelines/wiki/pipeline_status) to view our current progress at a glance.
 
-:vertical_traffic_light: Lastly, please find [here in the project's wiki](https://github.com/Inria-Empenn/narps_open_pipelines/wiki/pipeline_status) a dashboard to see pipelines work progresses at first glance.
+## Contributing
 
-## Getting Started
+NARPS open pipelines uses [nipype](https://nipype.readthedocs.io/en/latest/index.html) as a workflow manager and provides a series of templates and examples to help reproduce the different teams’ analysis. 
 
-### Contents overview
-
-- :snake: :package: `narps_open/` contains the Python package with all the pipelines logic.
-- :brain: `data/` contains data that is used by the pipelines, as well as the (intermediate or final) results data. Instructions to download data are available in [INSTALL.md](/INSTALL.md#data-download-instructions).
-- :blue_book: `docs/` contains the documentation for the project. Start browsing it with the entry point [docs/README.md](/docs/README.md)
-- :orange_book: `examples/` contains notebooks examples to launch of the reproduced pipelines.
-- :microscope: `tests/` contains the tests of the narps_open package.
+There are many ways you can contribute 🤗 :wave: Any help is welcome ! Follow the guidelines in [CONTRIBUTING.md](/CONTRIBUTING.md) if you wish to get involved !
 
 ### Installation
 
 To get the pipelines running, please follow the installation steps in [INSTALL.md](/INSTALL.md)
 
-### Contributing 
-
-:wave: Any help is welcome ! Follow the guidelines in [CONTRIBUTING.md](/CONTRIBUTING.md) if you wish to get involved !
+## Getting started
+If you are interested in using the codebase to run the pipelines, see the [user documentation (work-in-progress)].
 
 ## References
 
@@ -64,7 +45,7 @@ To get the pipelines running, please follow the installation steps in [INSTALL.m
 
 ## Funding
 
-This project is supported by Région Bretagne (Boost MIND). 
+This project is supported by Région Bretagne (Boost MIND) and by Inria (Exploratory action GRASP). 
 
 ## Credits
 

docs/status.md

@@ -1,10 +1,18 @@
 # Access the work progress status pipelines
 
 The class `PipelineStatusReport` of module `narps_open.utils.status` allows to create a report containing the following information for each pipeline:
+* a work progress status : `idle`, `progress`, or `done`;
 * the software it uses (collected from the `categorized_for_analysis.analysis_SW` of the [team description](/docs/description.md)) ;
 * whether it uses data from fMRIprep or not ;
 * a list of issues related to it (the opened issues of the project that have the team ID inside their title or description) ;
-* a work progress status : `idle`, `progress`, or `done`.
+* a list of pull requests related to it (the opened pull requests of the project that have the team ID inside their title or description) ;
+* whether it was excluded from the original NARPS analysis ;
+* a reproducibility rating :
+  * default score is 4;
+  * -1 if the team did not use fmriprep data;
+  * -1 if the team used several pieces of software (e.g.: FSL and AFNI);
+  * -1 if the team used custom or marginal software (i.e.: something else than SPM, FSL, AFNI or nistats);
+  * -1 if the team did not provided his source code.
 
 This allows contributors to best select the pipeline they want/need to contribute to. For this purpose, the GitHub Actions workflow [`.github/workflows/pipeline_status.yml`](/.github/workflows/pipeline_status.yml) allows to dynamically generate the report and to store it in the [project's wiki](https://github.com/Inria-Empenn/narps_open_pipelines/wiki).
 
@@ -55,22 +63,31 @@ python narps_open/utils/status --json
 #         "softwares": "FSL",
 #         "fmriprep": "No",
 #         "issues": {},
-#         "status": "idle"
+#         "excluded": "No",
+#         "reproducibility": 3,
+#         "reproducibility_comment": "",
+#         "issues": {},
+#         "pulls": {},
+#         "status": "2-idle"
 #     },
 #     "0C7Q": {
 #         "softwares": "FSL, AFNI",
 #         "fmriprep": "Yes",
 #         "issues": {},
+#         "excluded": "No",
+#         "reproducibility": 3,
+#         "reproducibility_comment": "",
+#         "issues": {},
+#         "pulls": {},
 #         "status": "idle"
 #     },
 # ...
 
 python narps_open/utils/status --md
-# | team_id | status | softwares used | fmriprep used ? | related issues |
-# | --- |:---:| --- | --- | --- |
-# | 08MQ | :red_circle: | FSL | No |  |
-# | 0C7Q | :red_circle: | FSL, AFNI | Yes |  |
-# | 0ED6 | :red_circle: | SPM | No |  |
-# | 0H5E | :red_circle: | SPM | No |  |
+# ...
+# | team_id | status | main software | fmriprep used ? | related issues | related pull requests | excluded from NARPS analysis | reproducibility |
+# | --- |:---:| --- | --- | --- | --- | --- | --- |
+# | Q6O0 | :green_circle: | SPM | Yes |  |  | No | :star::star::star::black_small_square:<br /> |
+# | UK24 | :orange_circle: | SPM | No | [2](url_issue_2),  |  | No | :star::star::black_small_square::black_small_square:<br /> |
 # ...
 ```

narps_open/data/description/analysis_pipelines_comments.tsv

@@ -1,71 +1,71 @@
 teamID	excluded_from_narps_analysis	exclusion_comment	reproducibility	reproducibility_comment
-50GV	no	N/A	?	Uses custom software (Denoiser)
-9Q6R	no	N/A		
-O21U	no	N/A		
-U26C	no	N/A		
-43FJ	no	N/A		
-C88N	no	N/A		
-4TQ6	yes	Resampled image offset and too large compared to template.		
-T54A	no	N/A		
-2T6S	no	N/A		
-L7J7	no	N/A		
-0JO0	no	N/A		
-X1Y5	no	N/A		
-51PW	no	N/A		
-6VV2	no	N/A		
-O6R6	no	N/A		
-C22U	no	N/A	?	Custom Matlab script for white matter PCA confounds
-3PQ2	no	N/A		
-UK24	no	N/A		
-4SZ2	yes	Resampled image offset from template brain.		
-9T8E	no	N/A		
-94GU	no	N/A	?	Multiple software dependencies : SPM + ART + TAPAS + Matlab.
-I52Y	no	N/A		
-5G9K	no	N/A	?	? 
-2T7P	yes	Missing thresholded images.	?	?
-UI76	no	N/A		
-B5I6	no	N/A		
-V55J	yes	Bad histogram : very small values.		
-X19V	no	N/A		
-0C7Q	yes	Appears to be a p-value distribution, with slight excursions below and above zero.		
-R5K7	no	N/A		
-0I4U	no	N/A		
-3C6G	no	N/A		
-R9K3	no	N/A		
-O03M	no	N/A		
-08MQ	no	N/A		
-80GC	no	N/A		
-J7F9	no	N/A		
-R7D1	no	N/A		
-Q58J	yes	Bad histogram : bimodal, zero-inflated with a second distribution centered around 5.		
-L3V8	yes	Rejected due to large amount of missing brain in center.		
-SM54	no	N/A		
-1KB2	no	N/A		
-0H5E	yes	Rejected due to large amount of missing brain in center.		
-P5F3	yes	Rejected due to large amounts of missing data across brain.		
-Q6O0	no	N/A		
-R42Q	no	N/A	?	Uses fMRIflows, a custom software based on NiPype.
-L9G5	no	N/A		
-DC61	no	N/A		
-E3B6	yes	Bad histogram : very long tail, with substantial inflation at a value just below zero.		
-16IN	no	N/A	?	Multiple software dependencies : matlab + SPM + FSL + R + TExPosition + neuroim
-46CD	no	N/A		
-6FH5	yes	Missing much of the central brain.		
-K9P0	no	N/A		
-9U7M	no	N/A		
-VG39	no	N/A		
-1K0E	yes	Used surface-based analysis, only provided data for cortical ribbon.	?	?
-X1Z4	yes	Used surface-based analysis, only provided data for cortical ribbon.	?	Multiple software dependencies : FSL + fmriprep + ciftify + HCP workbench + Freesurfer + ANTs
-I9D6	no	N/A		
-E6R3	no	N/A		
-27SS	no	N/A		
-B23O	no	N/A		
-AO86	no	N/A		
-L1A8	yes	Resampled image much smaller than template brain.	?	?
-IZ20	no	N/A		
-3TR7	no	N/A		
-98BT	yes	Rejected due to very bad normalization.		
-XU70	no	N/A	?	Uses custom software : FSL + 4drealign
-0ED6	no	N/A	?	? 
-I07H	yes	Bad histogram : bimodal, with second distribution centered around 2.5.		
-1P0Y	no	N/A		
+50GV	No	N/A	3	Uses custom software (Denoiser)
+9Q6R	No	N/A	2	
+O21U	No	N/A	3	
+U26C	No	N/A	4	Link to shared analysis code : https://github.com/gladomat/narps
+43FJ	No	N/A	2	
+C88N	No	N/A	3	
+4TQ6	Yes	Resampled image offset and too large compared to template.	3	
+T54A	No	N/A	3	
+2T6S	No	N/A	3	
+L7J7	No	N/A	3	
+0JO0	No	N/A	3	
+X1Y5	No	N/A	2	
+51PW	No	N/A	3	
+6VV2	No	N/A	2	
+O6R6	No	N/A	3	
+C22U	No	N/A	1	Custom Matlab script for white matter PCA confounds
+3PQ2	No	N/A	2	
+UK24	No	N/A	2	
+4SZ2	Yes	Resampled image offset from template brain.	3	
+9T8E	No	N/A	3	
+94GU	No	N/A	1	Multiple software dependencies : SPM + ART + TAPAS + Matlab.
+I52Y	No	N/A	2	
+5G9K	Yes	Values in the unthresholded images are not z / t stats	3	
+2T7P	Yes	Missing thresholded images.	2	Link to shared analysis code : https://osf.io/3b57r
+UI76	No	N/A	3	
+B5I6	No	N/A	3	
+V55J	Yes	Bad histogram : very small values.	2	
+X19V	No	N/A	3	
+0C7Q	Yes	Appears to be a p-value distribution, with slight excursions below and above zero.	2	
+R5K7	No	N/A	2	
+0I4U	No	N/A	2	
+3C6G	No	N/A	2	
+R9K3	No	N/A	3	
+O03M	No	N/A	3	
+08MQ	No	N/A	2	
+80GC	No	N/A	3	
+J7F9	No	N/A	3	
+R7D1	No	N/A	3	Link to shared analysis code : https://github.com/IMTAltiStudiLucca/NARPS_R7D1
+Q58J	Yes	Bad histogram : bimodal, zero-inflated with a second distribution centered around 5.	3	Link to shared analysis code : https://github.com/amrka/NARPS_Q58J
+L3V8	Yes	Rejected due to large amount of missing brain in center.	2	
+SM54	No	N/A	3	
+1KB2	No	N/A	2	
+0H5E	Yes	Rejected due to large amount of missing brain in center.	2	
+P5F3	Yes	Rejected due to large amounts of missing data across brain.	2	
+Q6O0	No	N/A	3	
+R42Q	No	N/A	2	Uses fMRIflows, a custom software based on NiPype. Code available here : https://github.com/ilkayisik/narps_R42Q
+L9G5	No	N/A	2	
+DC61	No	N/A	3	
+E3B6	Yes	Bad histogram : very long tail, with substantial inflation at a value just below zero.	4	Link to shared analysis code : doi.org/10.5281/zenodo.3518407
+16IN	Yes	Values in the unthresholded images are not z / t stats	2	Multiple software dependencies : matlab + SPM + FSL + R + TExPosition + neuroim. Link to shared analysis code : https://github.com/jennyrieck/NARPS
+46CD	No	N/A	1	
+6FH5	Yes	Missing much of the central brain.	2	
+K9P0	No	N/A	3	
+9U7M	No	N/A	2	
+VG39	Yes	Performed small volume corrected instead of whole-brain analysis	3	
+1K0E	Yes	Used surface-based analysis, only provided data for cortical ribbon.	1	
+X1Z4	Yes	Used surface-based analysis, only provided data for cortical ribbon.	1	Multiple software dependencies : FSL + fmriprep + ciftify + HCP workbench + Freesurfer + ANTs
+I9D6	No	N/A	2	
+E6R3	No	N/A	2	
+27SS	No	N/A	2	
+B23O	No	N/A	3	
+AO86	No	N/A	2	
+L1A8	Yes	Not in MNI standard space.	2	
+IZ20	No	N/A	1	
+3TR7	No	N/A	3	
+98BT	Yes	Rejected due to very bad normalization.	2	
+XU70	No	N/A	1	Uses custom software : FSL + 4drealign
+0ED6	No	N/A	2	
+I07H	Yes	Bad histogram : bimodal, with second distribution centered around 2.5.	2	
+1P0Y	No	N/A	2