[ENH] microelectrode electrophysiology specification (BEP032) #1705

yarikoptic · 2024-02-14T15:24:13Z

Replaces #1352 submitted from a fork outside of bids-specification.

Add specification for microelectrode electrohpysiology datasets based on the BEP032 proposal

Note

We meet regularly and everyone is welcome

Next meeting: insert date on URL to join

Communication channel: https://framalistes.org/sympa/info/neuroscience-data-structure

Tip

HTML preview of this BEP

To use this WiP schema on sample datasets, see https://deno.land/x/[email protected]#modifying-and-building-a-new-schema on how to use "stock" bids-validator with a custom schema. (attn @TheChymera)

TODOs

Issues this PR would likely to address

Fixes run node tests in github actions legacy-validator#1375

Issues to see being addressed while working on this BEP (likely to move above) or not (moved below):

Other issues which relate but not in scope here and provided for reference/backreference

… _acq-

src/schema/rules/files/raw/channels.yaml

TheChymera · 2024-03-06T16:39:46Z

src/schema/objects/columns.yaml

@@ -42,6 +42,38 @@ age:
    for privacy purposes.
  type: number
  unit: year
+alpha_rotation:


I'd really recommend renaming the rotation axes to yaw, roll, and pitch (that would be the analogous angle order). There was no consensus either way on the google docs discussion. Someone said both are confusing, which I guess might be expected, but alpha, beta, gamma, are just more confusing...

I agree -- very adhoc. Let's discuss in google doc

https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

Could you chime in? I think the other guy commenting might be amenable to accepting this as well.

I agree that both are confusing, but at least alpha, beta and gamma are SO confusing that everyone realizes that additional specification is needed to define them properly. With roll, yaw and pitch it seems at first that all is clear, until you have a number of different people go through different use cases. See the more challenging examples that I posted on the google doc under https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAAA4fkI4eY

@robertoostenveld sorry, I only now saw the update, reply there. I don't think confusion on purpose is good in this case, because the documentation is very eclectic so we might be sending people down rabbit holes. Wiki, where people will invariably go first, does a particularly poor job explaining both euler and TB angles for the casual non-mathematics-versed user. The only thing that wiki has going for it here are the aircraft animations on the TB page. Yaw, Pitch, Roll, will be intuited correctly as long as we specify the starting postion. That we can do (1) as (I think, it's pretty vague) is currently proposed, aligning the implant with the world coordinate system (meaning most implants will be at yaw 0 pitch -90 roll 0) or (2) relative to the implantation stereotax normal (meaning most implants will be at 0 0 0).

For comparison, Euler commonly has the normal pointing up so most implants will be.... 0 -180 0 🤔

in the last BEP032 meeting we discussed further and wanted to follow this approach now (based on the Allen Inst. standard and IBL standard):

Assumption: x,y,z is posterior, ventral, right (unit needs to be specified).

Translational origin (0,0,0) needs to be defined (typically Bregma for rodents).

Rotational origin (0,0,0) is the probe facing up with the tip facing forward. Rotations are all clockwise in degrees and around the tip. For multi-shank probes, the “tip” of the probe is defined as the end of the left shank if you are looking at the electrodes.

yaw: clockwise when looking down

pitch: In the direction of the electrode face

roll: clockwise when looking down at the probe

The depth (unit needs to be specified) is a translation in the direction that the tip is pointing.

We need to add a “probe_model”, which references probeinterface_library

versioning of probe library files (raised here)

annotation of “tip” position (raised here)

NOTE: We need to change the electrode x,y,z.
X,y,z in BIDS refers to location in brain, not on probe.

TheChymera · 2024-03-06T16:46:27Z

src/schema/objects/columns.yaml

@@ -391,6 +462,12 @@ reference__ieeg:
    - type: string
      enum:
        - n/a
+reference_atlas:


The “reference atlas” if you visually verify the area is basically the atlas you looked at before you nod and said “eh, good enough”. This seems at once an overly detailed (does this really matter? the coordinates are commonly given with sub-mm precision) and underdetermined (what coordinates did you look at in the atlas?). I think it's best just to drop this.

TheChymera · 2024-03-12T01:33:27Z

Also relevant if you'd like to comment. → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIzHGpUU

TheChymera · 2024-03-12T01:57:39Z

Also I forgot to link to this when I wrote it → https://docs.google.com/document/d/1oG-C8T-dWPqfVzL2W8HO3elWK8NIh2cOCPssRGv23n0/edit?disco=AAABIGPAMOw

src/schema/rules/files/raw/ephys.yaml

…t was missing

src/schema/objects/datatypes.yaml

src/schema/objects/modalities.yaml

src/schema/objects/suffixes.yaml

src/modality-specific-files/microelectrode-electrophysiology.md

src/schema/objects/columns.yaml

bendichter · 2024-12-18T15:41:12Z

src/schema/objects/columns.yaml

+height__probes:
+  name: height
+  display_name: Height
+  description: |
+    Physical height of the probe, for example, '5'.
+    This dimension corresponds to the y’axis of the Euler transformation defined by
+    alpha, beta and gamma rotations values below.
+  type: number


Suggested change

height__probes:

name: height

display_name: Height

description: |

Physical height of the probe, for example, '5'.

This dimension corresponds to the y’axis of the Euler transformation defined by

alpha, beta and gamma rotations values below.

type: number

I don't understand what this is supposed to be

So these are about geometry of the probes. Thus relating to

Formalize specification of shape(s) (AKA contour(s)) #2013
?

bendichter · 2024-12-18T15:41:37Z

src/schema/objects/columns.yaml

+width__probes:
+  name: width
+  display_name: Width
+  description: |
+    Physical width of the probe, for example, '5'.
+    This dimension corresponds to the x’axis of the Euler transformation defined by
+    alpha, beta and gamma rotations values.
+  type: number


Suggested change

width__probes:

name: width

display_name: Width

description: |

Physical width of the probe, for example, '5'.

This dimension corresponds to the x’axis of the Euler transformation defined by

alpha, beta and gamma rotations values.

type: number

I don't understand what this is supposed to be.

Co-authored-by: Ben Dichter <[email protected]>

…nnels

… references

* origin/master: Update blood.tsv example Convert participants.tsv and samples.tsv fix: Use default YAML loader Update all TSV blocks MNT: Only check yaml syntax ENH: Add code to generate tables from TSV fence blocks Do explicitly "allow" for having dotfiles and explicitly ignore them Move description of n/a above "String values" to avoid any association

…maceuticals

bendichter · 2024-12-19T06:38:57Z

src/modality-specific-files/microelectrode-electrophysiology.md

+It follows the [general BIDS specifications to describe participants](../modality-agnostic-files.md#participants-file).
+
+On top of the existing columns that can be present in this file and that are described in the BIDS specifications (`participant_id`, `species`, `strain`, `strain_rrid`, `sex`, `handedness`, and `age`), we propose to allow adding the following ones:
+


It looks like this section describing the additional fields of the participants table is missing

bendichter · 2024-12-19T06:40:52Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+## Coordinate System JSON (`*_coordsystem.json`) & Photos of electrode positions (`_photo.jpg`)
+
+This file provides metadata on the global coordinate system in which the electrodes are placed.


Suggested change

This file provides metadata on the global coordinate system in which the electrodes are placed.

This file provides metadata on the coordinate system in which the electrodes are placed.

I think the word "global" is confusing here. Is it serving a purpose I am missing?

bendichter · 2024-12-19T06:54:57Z

src/modality-specific-files/microelectrode-electrophysiology.md

+        "MicroephysCoordinateUnits":"REQUIRED",
+        "MicroephysCoordinateSystemDescription":"RECOMMENDED",
+        "MicroephysCoordinateSystemPhoto":"OPTIONAL",
+        "MEGCoordinateSystem":"REQUIRED",


Suggested change

"MEGCoordinateSystem":"REQUIRED",

was this included by mistake?

bendichter · 2024-12-19T06:56:53Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+
+## Coordinate System JSON (`*_coordsystem.json`) & Photos of electrode positions (`_photo.jpg`)
+


We recently settled on the idea that the coordinate system of the standard electrodes.tsv file (without a "space" designation") should be in probe space. This means that x, and y, (and maybe z), should be relative positions of contacts on the probe. It would be idea if the (0,0) point is the tip of the probe, but honestly this is not really important for spike sorting.

I think it's a bit strange to discuss coordinate systems before we discuss probes and electrodes. Is there a motivation for having these sections in the current order or can I move things around a bit?

bendichter · 2024-12-19T07:02:00Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+This file contains the following information:
+- The electrode name
+- The electrode coordinates in 3 columns (`xyz`)


Suggested change

- The electrode coordinates in 3 columns (`xyz`)

- The electrode coordinates in 2 or 3 columns (`xy[z]`)

bendichter · 2024-12-19T07:08:25Z

src/modality-specific-files/microelectrode-electrophysiology.md

+- The electrode coordinates in 3 columns (`xyz`)
+- The ID of the probe the electrode is located on
+
+The coordinates are relative to the position on the probe.


Suggested change

The coordinates are relative to the position on the probe.

When the 'space' label is not defined, the coordinates are relative position on the probe. To specify electrode positions in subject-space, an atlas space, or some other space, use another electrodes.tsv file with the [space-<label>](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#space) entity: (`*[_space-<label>]_electrodes.tsv`). Each of these additional electrodes.tsv files should be accompanied by a matching *_space-<label>_coordsystem.json file.

For example:

└─ sub-01/

├─ sub-01_space-CCFv3_electrodes.tsv

├─ sub-01_space-CCFv3_coordsystem.json

└─ ...

bendichter · 2024-12-19T07:16:59Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+We propose to store all metadata that is not directly related to one of the other metadata files (probe/electrode/channel information) into a single JSON file: `_ephys.json`.
+
+There should be one such JSON file for each data file.


I'm not really sure what this means

bendichter · 2024-12-19T07:29:40Z

src/modality-specific-files/microelectrode-electrophysiology.md

+### Probes
+
+It is RECOMMENDED to use the following structure to provide more metadata for each probe:
+
+```JSON
+"ProbeContours": {
+   "probe_infoid": {
+      "<my_probe_id>": {
+         "Contour": <list of contour points>,
+         "Unit": "<my spatial unit>"
+      }
+   }
+}
+```
+
+
+Whereas `<my_probe_id>` has to exist also in the `probes.tsv` file and `<list of contour points>` is
+a list of x, y(, z) tuples providing the contour of the probe in the same reference frame as the
+electrode coordinates (see `electrodes.tsv`).
+In case of different units used than in the `electrodes.tsv`, the key `Unit` can be used here to
+provide the spatial unit of the coordinate points.


This is an accurate depiction of what was discussed in the November 2024, meeting, but in the December 2024 meeting we decided to punt on the countour specification, since it's a bit tricky to include, we might want to refer to ProbeInterface, and it's not critical for spike sorting.

bendichter · 2024-12-19T07:31:22Z

src/modality-specific-files/microelectrode-electrophysiology.md

+Task Events documentation.
+Note that this file can also be used to describe stimulation performed during the recording. For this,
+please follow the iEEG stimulation documentation.


To do: add links here.

bendichter · 2024-12-19T07:31:40Z

src/modality-specific-files/microelectrode-electrophysiology.md

+
+## Multi-part recordings
+
+Two different procedures are supported to handle multi-part recordings. In short, the two options are:


Suggested change

Two different procedures are supported to handle multi-part recordings. In short, the two options are:

Two different procedures are supported to handle multi-part recordings. The two options are:

bendichter · 2024-12-19T07:32:52Z

src/modality-specific-files/microelectrode-electrophysiology.md

+in the `*_events.tsv` file.
+These two options are made available to support different usages and habits of the experimenters, as
+well as to benefit from the capability of the supported data formats (NWB and NIX).
+They are described in the following subsections, and made explicit through some of the example data sets.


My read of the BIDS standard is that (i) _scans.tsv is the more canonical approach. What is the justification for supporting both methods?

bendichter · 2024-12-19T07:34:03Z

src/modality-specific-files/microelectrode-electrophysiology.md

+session (for example, recording start times in case the recording was paused and restarted)
+when the data from each of these different recordings is stored in separate files.
+Each data file should have a name that contains a `_task-XX` and/or `_run-XX` suffix, and
+should be described by at most one row in the `*_scans.tsv` file. See also the BIDS Scans


Suggested change

should be described by at most one row in the `*_scans.tsv` file. See also the BIDS Scans

should be described by one row in the `*_scans.tsv` file. See also the BIDS Scans

why "at most"? Shouldn't it be exactly 1, so this scan file can specify the start time?

bendichter · 2024-12-19T07:37:12Z

src/modality-specific-files/microelectrode-electrophysiology.md

+be expressed in the following format 2009-06-15T13:45:30 (year, month, day, hour (24h),
+minute, second; this is equivalent to the RFC3339 “date-time” format, time zone is always
+assumed as local time).


Suggested change

be expressed in the following format 2009-06-15T13:45:30 (year, month, day, hour (24h),

minute, second; this is equivalent to the RFC3339 “date-time” format, time zone is always

assumed as local time).

be expressed in the RFC3339 “date-time” format, for example 2009-06-15T13:45:30 (year, month, day, hour (24h), minute, second. Time zone is always assumed as local time.

Add first BEP032 objects

7ec5d5e

yarikoptic requested review from tsalo and erdalkaraca as code owners February 14, 2024 15:24

yarikoptic mentioned this pull request Feb 14, 2024

[ENH] Add BEP032 (microelectrode electrophysiology) specification #1352

Closed

3 tasks

Add ephys modality, start adding _channels but ran into a question on…

2f393ca

… _acq-

yarikoptic commented Feb 14, 2024

View reviewed changes

src/schema/rules/files/raw/channels.yaml Show resolved Hide resolved

tsalo added the BEP label Feb 15, 2024

yarikoptic mentioned this pull request Feb 29, 2024

Allow for using an arbitrary suffix and/or arbitrary entity bids-standard/bids-2-devel#63

Open

yarikoptic and others added 2 commits March 5, 2024 19:56

Add _acq- to ephys, add _channels and _coords

648781a

Draft for probes TSV file

d3c9996

TheChymera reviewed Mar 6, 2024

View reviewed changes

TheChymera closed this Mar 7, 2024

TheChymera deleted the bep032 branch March 7, 2024 22:13

TheChymera restored the bep032 branch March 7, 2024 22:15

TheChymera reopened this Mar 11, 2024

Fix preferably typo

08c92cd

yarikoptic mentioned this pull request Mar 15, 2024

[SCHEMA] Remove stale test comment -- test no longer fails! #1727

Merged

yarikoptic commented Mar 15, 2024

View reviewed changes

src/schema/rules/files/raw/ephys.yaml Outdated Show resolved Hide resolved

robertoostenveld mentioned this pull request Mar 21, 2024

Draft examples for BEP032 on animal electrophysiology bids-standard/bids-examples#430

Draft

yarikoptic mentioned this pull request Apr 2, 2024

Automatic generation of extra-fields template forms from yaml-based schema (BIDS standard) elabftw/elabftw#5023

Open

Comment out not yet defined iephys and add ephys to datatypes where i…

e2030ac

…t was missing

yarikoptic commented Apr 17, 2024

View reviewed changes

src/schema/objects/datatypes.yaml Outdated Show resolved Hide resolved

dorahermes reviewed Apr 17, 2024

View reviewed changes

src/schema/objects/modalities.yaml Outdated Show resolved Hide resolved

dorahermes reviewed Apr 17, 2024

View reviewed changes

src/schema/objects/suffixes.yaml Outdated Show resolved Hide resolved

TheChymera force-pushed the bep032 branch from cde80a6 to e2030ac Compare April 17, 2024 21:26

Added probes file

53f1087

Remi-Gau changed the title ~~[ENH] Add BEP032 (microelectrode electrophysiology) specification~~ [ENH] microelectrode electrophysiology specification (BEP032) Apr 18, 2024

bendichter reviewed Dec 18, 2024

View reviewed changes

src/modality-specific-files/microelectrode-electrophysiology.md Outdated Show resolved Hide resolved

bendichter reviewed Dec 18, 2024

View reviewed changes

src/schema/objects/columns.yaml Outdated Show resolved Hide resolved

bendichter reviewed Dec 18, 2024

View reviewed changes

src/schema/objects/columns.yaml Outdated Show resolved Hide resolved

bendichter reviewed Dec 18, 2024

View reviewed changes

src/schema/objects/columns.yaml Outdated Show resolved Hide resolved

bendichter reviewed Dec 18, 2024

View reviewed changes

src/schema/objects/columns.yaml Outdated Show resolved Hide resolved

bendichter reviewed Dec 18, 2024

View reviewed changes

This was referenced Dec 18, 2024

Formalize specification of shape(s) (AKA contour(s)) #2013

Open

NF: tools/text2filetree.py helper to convert some indented text into our filetree example spec #2014

Draft

yarikoptic and others added 7 commits December 18, 2024 14:47

Fix tabulation in a .tsv + convert to use yaw, pitch, roll

46c91b0

Co-authored-by: Ben Dichter <[email protected]>

Add types and min/max for roll,pitch,yaw + fix typo for microephysCha…

a884604

…nnels

Replace Pharmaceuticals with already existing fields and fix few more…

262d5b1

… references

Make use of merged fancing for tsvs and add a note about spotted Phar…

7bc17cc

…maceuticals

Remove some "bad latin"

d5fef1f

Fix a link to be final, before redirects

039fbb3

yarikoptic force-pushed the bep032 branch from fdb0300 to 039fbb3 Compare December 19, 2024 02:56

bendichter reviewed Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ENH] microelectrode electrophysiology specification (BEP032) #1705

[ENH] microelectrode electrophysiology specification (BEP032) #1705

yarikoptic commented Feb 14, 2024 •

edited

Loading

TheChymera Mar 6, 2024

yarikoptic Mar 7, 2024

TheChymera Mar 11, 2024

robertoostenveld Mar 12, 2024

TheChymera Apr 17, 2024 •

edited

Loading

lzehl Oct 16, 2024

TheChymera Mar 6, 2024

TheChymera commented Mar 12, 2024 •

edited

Loading

TheChymera commented Mar 12, 2024

bendichter Dec 18, 2024

yarikoptic Dec 18, 2024

bendichter Dec 18, 2024

bendichter Dec 19, 2024 •

edited

Loading

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024 •

edited

Loading

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

bendichter Dec 19, 2024

		It follows the [general BIDS specifications to describe participants](../modality-agnostic-files.md#participants-file).

		On top of the existing columns that can be present in this file and that are described in the BIDS specifications (`participant_id`, `species`, `strain`, `strain_rrid`, `sex`, `handedness`, and `age`), we propose to allow adding the following ones:


		## Coordinate System JSON (`*_coordsystem.json`) & Photos of electrode positions (`_photo.jpg`)

		This file provides metadata on the global coordinate system in which the electrodes are placed.

	This file provides metadata on the global coordinate system in which the electrodes are placed.
	This file provides metadata on the coordinate system in which the electrodes are placed.

	- The electrode coordinates in 3 columns (`xyz`)
	- The electrode coordinates in 2 or 3 columns (`xy[z]`)

-The coordinates are relative to the position on the probe.
+When the 'space' label is not defined, the coordinates are relative position on the probe. To specify electrode positions in subject-space, an atlas space, or some other space, use another electrodes.tsv file with the [space-<label>](https://bids-specification.readthedocs.io/en/stable/appendices/entities.html#space) entity: (`*[_space-<label>]_electrodes.tsv`). Each of these additional electrodes.tsv files should be accompanied by a matching *_space-<label>_coordsystem.json file.
+For example:
+└─ sub-01/
+   ├─ sub-01_space-CCFv3_electrodes.tsv
+   ├─ sub-01_space-CCFv3_coordsystem.json
+   └─ ...


		We propose to store all metadata that is not directly related to one of the other metadata files (probe/electrode/channel information) into a single JSON file: `_ephys.json`.

		There should be one such JSON file for each data file.


		## Multi-part recordings

		Two different procedures are supported to handle multi-part recordings. In short, the two options are:

	should be described by at most one row in the `*_scans.tsv` file. See also the BIDS Scans
	should be described by one row in the `*_scans.tsv` file. See also the BIDS Scans

[ENH] microelectrode electrophysiology specification (BEP032) #1705

Are you sure you want to change the base?

[ENH] microelectrode electrophysiology specification (BEP032) #1705

Conversation

yarikoptic commented Feb 14, 2024 • edited Loading

TODOs

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheChymera Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheChymera commented Mar 12, 2024 • edited Loading

TheChymera commented Mar 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bendichter Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bendichter Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yarikoptic commented Feb 14, 2024 •

edited

Loading

TheChymera Apr 17, 2024 •

edited

Loading

TheChymera commented Mar 12, 2024 •

edited

Loading

bendichter Dec 19, 2024 •

edited

Loading

bendichter Dec 19, 2024 •

edited

Loading