Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CREST fmriprep: another recon-all error #2

Open
jdkent opened this issue Mar 18, 2020 · 2 comments
Open

CREST fmriprep: another recon-all error #2

jdkent opened this issue Mar 18, 2020 · 2 comments
Labels
argon fmriprep issue when running fmriprep singularity

Comments

@jdkent
Copy link
Member

jdkent commented Mar 18, 2020 •
edited
Loading

Script/Job File

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M [email protected]
#$ -o /Shared/vosslabhpc/Projects/CREST/code/fmriprep/out/
#$ -e /Shared/vosslabhpc/Projects/CREST/code/fmriprep/err/
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg \
/mnt/Projects/CREST/ /mnt/Projects/CREST/derivatives \
participant --participant_label BETTER120044 \
-w /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep \
--write-graph --mem_mb 35000 --omp-nthreads 10 --nthreads 16 --use-aroma \
--output-space template \
--template MNI152NLin2009cAsym \
--fs-license-file /mnt/UniversalSoftware/freesurfer_license.txt

Error Message

200308-19:04:37,644 nipype.workflow WARNING:
	 [Node] Error on "fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aseg.tonative" (/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative)
200308-19:04:38,854 nipype.workflow ERROR:
	 Node tonative failed to run on host argon-itf-cf49-34.hpc.
200308-19:04:38,859 nipype.workflow ERROR:
	 Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120044/log/20200308-100511_d66a5daa-772a-4968-b8d4-99daf24ea0a9/crash-20200308-190438-ariveradompenciel-tonative-78122109-9cd6-4992-a632-81babfdc0001.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/freesurfer/base.py", line 144, in run
    return super(FSCommand, self).run(**inputs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 522, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 1038, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 975, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
mri_label2vol --regheader /mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/aseg.mgz --seg /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg.mgz --temp /mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/rawavg.mgz --o /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg_vol.nii.gz
Standard output:
Number of labels: 0
Annot File:      (null)
Template Volume: /mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/rawavg.mgz
Outut Volume: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg_vol.nii.gz
Registration File: (null)
Fill Threshold: 0
Label Vox Vol:  1
ProjType:       (null)
ProjTypeId:     0
ProjStart:      0
ProjStop:       0
ProjDelta:      0.1
Subject:  (null)
Hemi:     (null)
UseNewASeg2Vol:  0
DoLabelStatVol  0
LabelCodeOffset  0
setenv SUBJECTS_DIR /opt/freesurfer/subjects
$Id: mri_label2vol.c,v 1.46 2014/12/08 21:11:54 greve Exp $
Template RAS-to-Vox: --------
-1.00000  -0.00000  -0.00000   98.00000;
-0.00000  -0.00000  -1.00000   128.00000;
-0.00000   1.00000  -0.00000   128.00000;
-0.00000  -0.00000  -0.00000   1.00000;
Template Voxel Volume: 1
nHits Thresh: 0
Computing registration based on header
RegMat: --------
-1.00000   0.00000   0.00000   0.00000;
 0.00000   0.00000   1.00000   0.00000;
 0.00000  -1.00000   0.00000   0.00000;
 0.00000   0.00000   0.00000   1.00000;
Label RAS-to-Vox: --------
 1.00000   0.00000   0.00000   98.00000;
 0.00000   1.00000   0.00000   128.00000;
 0.00000   0.00000   1.00000   128.00000;
 0.00000   0.00000   0.00000   1.00000;
ERROR: loading aseg /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg.mgz
Standard error:
mghRead(/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg.mgz): could not read 262144 bytes at slice 151
Return code: 1

Potential Solution

delete the working directory and run again: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf
@jdkent jdkent added fmriprep issue when running fmriprep singularity argon labels Mar 18, 2020
@jdkent
Copy link
Member Author

jdkent commented Mar 18, 2020

Error when ran again:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 303, in main
    fmriprep_wf.write_graph(graph2use="colored", format='svg', simple_form=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 430, in write_graph
    outfname = format_dot(dotfilename, format=format)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 1390, in format_dot
    formatted_dot, _ = _run_dot(dotfilename, format_ext=format)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 1407, in _run_dot
    resource_monitor=False).run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 522, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 1038, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 975, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
dot -Tsvg -o"/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.svg" "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot"
Standard output:

Standard error:
Warning: syntax ambiguity - badly delimited number '120065_' in line 2179 of /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot splits into two tokens
Error: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot: syntax error in line 2179 near '120065'
Return code: 1

Potential Solution

delete the graph* files in the working directory for fmriprep: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph*

(follow up, this works to fix this particular error)

@jdkent
Copy link
Member Author

jdkent commented Mar 18, 2020

Next step was deleting the freesurfer directory entirely for the participant, but this resulted in another error:

200315-11:58:00,540 nipype.workflow WARNING:
	 Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

	Node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm
	Interface: fmriprep.interfaces.freesurfer.PatchedRobustRegister
	Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))

traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:00,593 nipype.workflow WARNING:
	 Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

	Node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern
	Interface: fmriprep.interfaces.freesurfer.FSInjectBrainExtracted
	Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))

traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:02,686 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern" in "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/skull_strip_extern".
200315-11:58:02,793 nipype.workflow INFO:
	 [Node] Setting-up "fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm" in "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/fsnative_2_t1_xfm".
200315-11:58:02,796 nipype.workflow ERROR:
	 Node skull_strip_extern failed to run on host argon-itf-cf49-34.hpc.
200315-11:58:02,829 nipype.workflow ERROR:
	 Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120044/log/20200315-115448_fb93b1e9-95ad-4522-8453-1fcd9f8aaff3/crash-20200315-115802-ariveradompenciel-skull_strip_extern-c2044b36-c4c2-46c3-9c9f-d7c3f3e924c9.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:05,572 nipype.workflow ERROR:
	 Node fsnative_2_t1_xfm failed to run on host argon-itf-cf49-34.hpc.
200315-11:58:05,573 nipype.workflow ERROR:
	 Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120044/log/20200315-115448_fb93b1e9-95ad-4522-8453-1fcd9f8aaff3/crash-20200315-115805-ariveradompenciel-fsnative_2_t1_xfm-6adebb56-12c4-4213-a796-3543734cac32.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:30,708 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern
200315-11:58:30,712 nipype.workflow ERROR:
	 could not run node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm

Potential Solution

Need to delete the relevant working directory as well: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
argon fmriprep issue when running fmriprep singularity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jdkent