ODF reports

[araikes]

I don't know if this has been "solved" or not, but I'm thinking it hasn't. As of 0.19.0, I still get crashing on the peaks plotter. See crash log below just for proof of its existence.

If this hasn't been solved somewhere else by someone else (and this is particular to singularity), the following does appear to work:

singularity run --containall --writable-tmpfs ....

Ran the same code with and without the --writable-tmpfs flag. Got the error below without the flag and got HTML files with peak plots when using the flag.

Node: qsirecon_wf.sub-01_dsistudio_pipeline.sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf.dsistudio_gqi.plot_peaks
Working directory: /tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/dsistudio_gqi/plot_peaks

Node inputs:

args = <undefined>
background_image = <undefined>
directions_file = <undefined>
environ = {}
fib_file = <undefined>
mask_file = <undefined>
mif_file = <undefined>
odf_file = <undefined>
odf_report = odfs_mosaic.png
odf_rois = <undefined>
peak_report = peaks_mosaic.png
peaks_only = False
subtract_iso = True

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node plot_peaks.

Cmdline:
	recon_plot --background_image /output/qsiprep/sub-01/dwi/sub-01_dir-AP_run-001_space-T1w_dwiref.nii.gz --fib /tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/dsistudio_gqi/gqi_recon/sub-01_dir-AP_run-001_space-T1w_desc-preproc_dwi.src.gz.odf.gqi.1.25.fib.gz --mask_file /tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/resample_mask/sub-01_desc-brain_mask_resample.nii.gz --odfs_image odfs_mosaic.png --odf_rois /tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/odf_rois/crossing_rois_trans.nii.gz --peaks_image peaks_mosaic.png --subtract-iso
Stdout:
	230817-10:56:15,850 nipype.interface INFO:
		 converting /tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/dsistudio_gqi/gqi_recon/sub-01_dir-AP_run-001_space-T1w_desc-preproc_dwi.src.gz.odf.gqi.1.25.fib.gz to plot ODF/peaks
	230817-10:56:24,650 nipype.interface INFO:
		 saving peaks image to /tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/dsistudio_gqi/plot_peaks/peaks_mosaic.png
	230817-10:56:24,801 nipype.interface INFO:
		 Plotting slice indices {'x': [73, 80, 87], 'y': [90, 99, 107], 'z': [78, 85, 92]}
Stderr:
	Fatal Python error: Aborted

	Current thread 0x00007f16d4518240 (most recent call first):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/fury/window.py", line 1018 in record
	  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/recon_plot.py", line 181 in plot_peak_slice
	  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/recon_plot.py", line 201 in peak_slice_series
	  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/recon_plot.py", line 115 in recon_plot
	  File "/usr/local/miniconda/bin/recon_plot", line 8 in <module>
	Aborted
Traceback:
	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'peak_report' trait of a _ReconPeaksReportOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/dsistudio_gqi/plot_peaks/peaks_mosaic.png' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/tmp/qsirecon_wf/sub-01_dsistudio_pipeline/sub_01_dir_AP_run_001_space_T1w_desc_preproc_recon_wf/dsistudio_gqi/plot_peaks/peaks_mosaic.png' for output 'peak_report' of a CLIReconPeaksReport interface

[mattcieslak]

you may have actually solved this issue @araikes! I will investigate from here to see if I can make this work without the writable tmpfs. If not this should definitely go in the docs, this issue is the worst

[araikes]

After a bit of checking, it appears that the plotting functions cannot make necessary writes to an X11 folder.

The following works (though there are a couple of warnings about locked folders at the beginning and non-empty folders at the end). Here, the "work" folder was just an empty folder at the beginning of each run created at the same folder level as nifti and derivatives:

singularity run --containall --writable-tmpfs \
-B $PWD/nifti:/nifti:ro \
-B $PWD/derivatives:/output \
-B /groups/adamraikes/license.txt:/license.txt \
-B $PWD/work:/tmp \
-B /tmp/.X11-unix/:/tmp/.X11-unix \
-B /groups/adamraikes/templateflow:/opt/templateflow \
--env TEMPLATEFLOW_HOME=/opt/templateflow \
/groups/adamraikes/singularity_images/qsiprep_0.19.0.sif \
/nifti /output/dsi_test3 participant \
--participant-label 01 \
--output-resolution 1 \
--recon-only \
--recon-input /output/qsiprep \
--recon-spec /output/dsi_test3/recon_spec.json \
-w /tmp \
--fs-license-file /license.txt \
--verbose

The following fails the peak report workflow, with the only difference being the --writable-tmpfs flag:

singularity run --containall \
-B $PWD/nifti:/nifti:ro \
-B $PWD/derivatives:/output \
-B /groups/adamraikes/license.txt:/license.txt \
-B $PWD/work:/tmp \
-B /tmp/.X11-unix/:/tmp/.X11-unix \
-B /groups/adamraikes/templateflow:/opt/templateflow \
--env TEMPLATEFLOW_HOME=/opt/templateflow \
/groups/adamraikes/singularity_images/qsiprep_0.19.0.sif \
/nifti /output/dsi_test3 participant \
--participant-label 01 \
--output-resolution 1 \
--recon-only \
--recon-input /output/qsiprep \
--recon-spec /output/dsi_test3/recon_spec.json \
-w /tmp \
--fs-license-file /license.txt \
--verbose

[dkp]

Another data point:

I tried adding --writable-tmpfs to qsirecon mrtrix_multishell_msmt_ACT-hsvs reconstruction on our HPC and it generated a nice report!

export RECON_SPEC=mrtrix_multishell_msmt_ACT-hsvs

apptainer run --cleanenv --writable-tmpfs
--bind ${DATA}:/data:ro
--bind ${APP_PREP_DERIV_DIR}:/qsiprep-output:ro
--bind ${APP_DERIV_DIR}:/out
--bind ${WORK_DIR}:/work
${APP} /data /out participant
--participant_label ${Subject}
--recon-input /qsiprep-output
--recon-spec ${RECON_SPEC}
--stop-on-first-crash
--output-resolution 1.3 -w /work -v -v
--n-cpus 16 --omp-nthreads 15
--recon-only
--fs-license-file ${HOME}/license.txt
--freesurfer_input ${FREESURFER_DIR}

[dkp]

Unfortunately, I did not have the same luck with dipy_mapmri.
This created some files and then crashed after 48 minutes
slurm-1615412_mapmri_crash.txt using --writable-tmpfs and allowing the odf reports.

However, this ran fine:
apptainer run --cleanenv
--bind ${DATA}:/data:ro
--bind ${APP_PREP_DERIV_DIR}:/qsiprep-output:ro
--bind ${APP_DERIV_DIR}:/out
--bind ${WORK_DIR}:/work
${APP} /data /out participant
--participant_label ${Subject}
--recon-input /qsiprep-output
--recon-spec ${RECON_SPEC}
--stop-on-first-crash
--output-resolution 1.3 -w /work -v -v
--n-cpus 16 --omp-nthreads 15
--skip-odf-reports --recon-only
--fs-license-file ${HOME}/license.txt
--freesurfer_input ${FREESURFER_DIR}

[mattcieslak]

There is a new opengl and fury in v0.21 that I think might fix this issue, along with --writable-tmpfs.

I'm also thinking about making the documentation suggest --writable-tmpfs and --containall for singularity.

[tsalo]

I think @mattcieslak fixed this in #174 and #182. Can I close this?