Missing data type when creating nwb

[rpswenson]

Going through this code:

from trodes_to_nwb.convert import create_nwbs
path = data_path # from step 1
output_dir = "/cumulus/rio/"

# for dual site recording:
probe_metadata_paths=['/cumulus/rio/BS28_raw_old/tetrode_12.5.yml',
                          '/cumulus/rio/BS28_raw_old/128c-4s8mm6cm-20um-40um-sl.yml']

create_nwbs(
    path,
    output_dir=output_dir,
    header_reconfig_path='/cumulus/rio/BS28_raw_old/20231107/20231107_BS28.trodesconf',
    #probe_metadata_paths=['/home/mcoulter/tetrode_12.5.yml'],
    #probe_metadata_paths=None,
    probe_metadata_paths=probe_metadata_paths,
    convert_video=False,
    n_workers=1,
    query_expression=None,
)

gets this error:

Cell In[6], line 11
      7 # for dual site recording:
      8 probe_metadata_paths=['[/cumulus/rio/BS28_raw_old/tetrode_12.5.yml](http://localhost:6502/cumulus/rio/BS28_raw_old/tetrode_12.5.yml)',
      9                           '[/cumulus/rio/BS28_raw_old/128c-4s8mm6cm-20um-40um-sl.yml](http://localhost:6502/cumulus/rio/BS28_raw_old/128c-4s8mm6cm-20um-40um-sl.yml)']
---> 11 create_nwbs(
     12     path,
     13     output_dir=output_dir,
     14     header_reconfig_path='[/cumulus/rio/BS28_raw_old/20231107/20231107_BS28.trodesconf](http://localhost:6502/cumulus/rio/BS28_raw_old/20231107/20231107_BS28.trodesconf)',
     15     #probe_metadata_paths=['[/home/mcoulter/tetrode_12.5.yml](http://localhost:6502/home/mcoulter/tetrode_12.5.yml)'],
     16     #probe_metadata_paths=None,
     17     probe_metadata_paths=probe_metadata_paths,
     18     convert_video=False,
     19     n_workers=1,
     20     query_expression=None,
     21 )

File [~/trodes_to_nwb/src/trodes_to_nwb/convert.py:185](http://localhost:6502/lab/tree/trodes_to_nwb/notebooks/trodes_to_nwb/src/trodes_to_nwb/convert.py#line=184), in create_nwbs(path, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, n_workers, query_expression, disable_ptp)
    183 else:
    184     for session, session_df in file_info.groupby(["date", "animal"]):
--> 185         _create_nwb(
    186             session,
    187             session_df,
    188             header_reconfig_path,
    189             probe_metadata_paths,
    190             output_dir,
    191             video_directory,
    192             convert_video,
    193             disable_ptp,
    194         )

File [~/trodes_to_nwb/src/trodes_to_nwb/convert.py:326](http://localhost:6502/lab/tree/trodes_to_nwb/notebooks/trodes_to_nwb/src/trodes_to_nwb/convert.py#line=325), in _create_nwb(session, session_df, header_reconfig_path, probe_metadata_paths, output_dir, video_directory, convert_video, disable_ptp)
    324 # run NWB Inspector to validate file for best practices before upload to DANDI
    325 logger.info("RUNNING NWB INSPECTOR")
--> 326 _inspect_nwb(output_path, logger)
    328 logger.info("DONE")
    330 return output_path

File [~/trodes_to_nwb/src/trodes_to_nwb/convert.py:337](http://localhost:6502/lab/tree/trodes_to_nwb/notebooks/trodes_to_nwb/src/trodes_to_nwb/convert.py#line=336), in _inspect_nwb(nwbfile_path, logger)
    334 """Run the resulting NWB file through the NWB Inspector to ensure it passes validation checks
    335 required for upload to the DANDI archive."""
    336 # this may take some time
--> 337 messages = list(
    338     nwbinspector.inspect_nwbfile(
    339         nwbfile_path=nwbfile_path, config=nwbinspector.load_config("dandi")
    340     )
    341 )
    342 logger.info("NWB Inspector output:")
    343 logger.info(messages)

File [~/miniforge3/envs/trodes_to_nwb/lib/python3.12/site-packages/nwbinspector/nwbinspector.py:557](http://localhost:6502/lab/tree/trodes_to_nwb/notebooks/miniforge3/envs/trodes_to_nwb/lib/python3.12/site-packages/nwbinspector/nwbinspector.py#line=556), in inspect_nwbfile(nwbfile_path, driver, skip_validate, max_retries, checks, config, ignore, select, importance_threshold)
    555     validation_error_list, _ = pynwb.validate(paths=[nwbfile_path], driver=driver)
    556     for validation_namespace_errors in validation_error_list:
--> 557         for validation_error in validation_namespace_errors:
    558             yield InspectorMessage(
    559                 message=validation_error.reason,
    560                 importance=Importance.PYNWB_VALIDATION,
   (...)
    563                 file_path=nwbfile_path,
    564             )
    566 with pynwb.NWBHDF5IO(path=nwbfile_path, mode="r", load_namespaces=True, driver=driver) as io:

TypeError: 'MissingDataType' object is not iterable

I would guess that there's something missing or incorrect in the metadata yaml file, but I'm not sure where to look for that in the metadata.

[samuelbray32]

My guess is this may just be an odd case where there's exactly one error of a type so it can't iterate through. in %debug can you see what validation_error_list and validation_namespace_errors are?

[samuelbray32]

If you could also supply what animal/session this was (what's in the data path) that would be helpful

[rpswenson]

Had to rerun the cell so i'll let you know once i can look at %debug. data_path is "/cumulus/rio/BS28_raw_old/20231107/".
I suspect something crashes around here in the output?

[/home/rswenson/miniforge3/envs/trodes_to_nwb/lib/python3.12/site-packages/hdmf/build/objectmapper.py:671](http://localhost:6502/lab/tree/trodes_to_nwb/notebooks/miniforge3/envs/trodes_to_nwb/lib/python3.12/site-packages/hdmf/build/objectmapper.py#line=670): MissingRequiredBuildWarning: Position 'position' is missing required value for attribute 'spatial_series'.
  warnings.warn(msg, MissingRequiredBuildWarning)

but we should be able to make nwbs without trodes position info

[samuelbray32]

• Yes, you can vreate nwb's without position tracking. That in itself shouldn't be the error
• You've actually generated the nwb file. The error is happening during inspection. I assume you've already started rerunning because the file's locked, but in future you don't need to rerun the whole generation process to debug this

/home/rswenson/miniforge3/envs/trodes_to_nwb/lib/python3.12/site-packages/hdmf/build/objectmapper.py:671: MissingRequiredBuildWarning: Position 'position' is missing required value for attribute 'spatial_series'.
warnings.warn(msg, MissingRequiredBuildWarning)

• This is just a Dandi format warning, not an error and is ok

• Knowing The other output from conversion would be helpful. trodes_to_nwb makes a conversion log in the directory you run the code in. Can you point me to where that is

[rpswenson]

The conversion log is in /cumulus/rio/BS28_raw_old/
Cell finished rerunning (oops) so from %debug validation_error_list = [Position (processing/behavior/position): missing data type SpatialSeries] and validation_namespace_errors = Position (processing/behavior/position): missing data type SpatialSeries

[edeno]

Ah it seems like it doesn't like Position being created without a SpatialSeries object. That should be something in trodes_to_nwb to be fixed. @samuelbray32 are you already on this?

[samuelbray32]

I think the actual breaking error was just the fact that based on the way the validator returns things validation_namespace_errors was a single item rather than an iterable list. This first level of problem is probably fixed by just adding a typecheck before iterating to catch this case of a single error item

[samuelbray32]

@rpswenson I was able to run the inspector for this file. Could you check your version of nwbinspector? This can be done by running:

import nwbinspector
nwbinspector.__version__

[rpswenson]

@rpswenson I was able to run the inspector for this file. Could you check your version of nwbinspector? This can be done by running:

import nwbinspector
nwbinspector.__version__

Mine is 0.4.34

[samuelbray32]

Hm, I'm still unable to recreate the error. Has this happened on other files you've converted?

[rpswenson]

No, these are the first files I've tried to convert

[samuelbray32]

@rpswenson Was this still an issue?

[emreybroyles]

@samuelbray32 actually now I am getting this error too. I talked to Rio, and they still get it too. This is happening for all days/NWBs for my two rats SC59 and SC65

[rly]

Could you share the paths to the files on /stelmo or somewhere else where we can take a look? The first level error seems like it might be an issue with pynwb or nwbinspector as @samuelbray32 mentioned above re: validation_namespace_errors being a single item rather than an iterable list

[emreybroyles]

@rly sure! For rat SC65, I am using the script /home/ebroyles/trodes_to_nwb/src/trodes_to_nwb_convert_SC65.py to convert to NWB for each date with the video files found in /stelmo/sc4712/rats/SC65/raw/<date> and the yamls found in /home/sc4712/yaml/SC65/generated

[samuelbray32]

@emreybroyles Are the nwb's still successfully generated when you hit this error? e.g. if you go look for them in /stelmo/nwb/raw (or wherever you're saving them) do they exist and are ~ the right size?

[emreybroyles]

@samuelbray32 yes the NWBs generate and are of the correct size. This issue isn't too large or debilitating, but mainly just annoying because I'd like to batch-create NWBs for an animal in a loop, and the loop breaks after each NWB creation, so I end up having to create each one individually anyways