From 1ba5b867545a45bf2690ba9dda5b19ae4589cf84 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Thu, 7 Dec 2023 23:48:59 +0000 Subject: [PATCH 01/17] Split by recording first commit. --- doc/how_to/index.rst | 1 + doc/how_to/process_by_channel_group.rst | 128 ++++++++++++++++++++++++ 2 files changed, 129 insertions(+) create mode 100644 doc/how_to/process_by_channel_group.rst diff --git a/doc/how_to/index.rst b/doc/how_to/index.rst index da94cf549c..8c3a4fbb35 100644 --- a/doc/how_to/index.rst +++ b/doc/how_to/index.rst @@ -8,3 +8,4 @@ How to guides analyse_neuropixels handle_drift load_matlab_data + process_by_channel_group diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst new file mode 100644 index 0000000000..1e186d4e6b --- /dev/null +++ b/doc/how_to/process_by_channel_group.rst @@ -0,0 +1,128 @@ +Processing a Recording by Channel Group (e.g. Shank) +=========================================================== + +In this tutorial, we will walk through how to preprocess and sorting a recording +per channel group. A channel group is a subset of channels grouped by some +feature - a typical example is grouping channels by shank in a Neuropixels recording. + +First, we can check the channels on our recording are grouped as we expect. For example, +in the below example we have a X-shank neuropixels recording. We can visualise the +channel groupings with XXX. + + +On our SpikeInterface recording, we can also access the channel groups per-channel + +1) channel groups +2) index channel ids by groups + +Why would you want to preprocess or sort by group? + +1) +2) +3) + +Splitting a Recording by Channel Group +-------------------------------------- + +We can split a recording into mutliply recordings, one for each channel group, with the `split_by` method. + +Here, we split a recording by channel group to get a `split_recording_dict`. This is a dictionary +containing the recordings, split by channel group: + +``` +split_recording_dict = recording.split_by("group") +print(split_recording_dict) +XXXXX +``` + +Now, we are ready to preprocess and sort by channel group. + +Preprocessing a Recording by Channel Group +------------------------------------------ + +The essense of preprocessing by channel group is to first split the recording +into separate recordings, perform the preprocessing steps, then aggregate +the channels back together. + +Here, we loop over the split recordings, preprocessing each shank group +individually. At the end, we use the `aggregate_channels` function +to combine the per-shank recordings back together. + +``` +preprocessed_recordings = [] +for recording in split_recordings_dict.values(): + + + shifted_recording = spre.phase_shift(recording) + + filtered_recording = spre.bandpass_filter(shifted_recording) + + referenced_recording = spre.common_reference(filtered_recording) + + preprocessed_recordings.append(referenced_recording) + +combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) + +``` + +It is strongly recommended to use the above structure to preprocess by group. +A discussion of the subtleties of the approach may be found in the below +Notes section for the interested reader. + +# Preprocessing channel in depth + +Preprocessing and aggregation of channels is very flexible. Under the hood, +`aggregate_channels` keeps track of when a recording was split. When `get_traces()` +is called, the preprocessing is still performed per-group, even though the +recording is now aggregated. + +However, to ensure data is preprocessed by shank, the preprocessing step must be +applied per-group. For example, the below example will NOT preprocess by shank: + +``` +split_recording = recording.split_by("group") +split_recording_as_list = list(**split_recording.values()) +combined_recording = aggregate_channels(split_recording_as_list) + +# will NOT preprocess by channel group. +filtered_recording = common_reference(combined_recording) + +``` + +Similarly, in the below example the first preprocessing step (bandpass filter) +would applied by group (although, this would have no effect in practice +as this preprocessing step is always performed per channel). However, +common referencing (which is effected by splitting by group) will +not be applied per group: + +``` +split_recording = recording.split_by("group") + +filtered_recording = [] +for recording in split_recording.values() + filtered_recording.append(spre.bandpass_filtered(recording)) + +combined_recording = aggregate_channels(filtered_recording) + +# As the recording has been combined, common referencing +# will NOT be applied per channel group. +referenced_recording = spre.common_reference(combined_recording). +``` + +Finally, it is not recommended to apply `aggregate_channels` more than once +as this will slow down `get_traces()` and may result in unpredictable behaviour. + + +Sorting a Recording by Channel Group +------------------------------------ + +Sorting a recording can be performed in two ways. One, is to split the +recording by group and use `run_sorter` (LINK) separately on each preprocessed +channel group. + +Altearntively, SpikeInterface proves a conveniecne function XXX +to this. + +Example + +Done! From 9a14de55af43301760b3e1854c4aaca6f55ffc26 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Fri, 8 Dec 2023 12:05:27 +0000 Subject: [PATCH 02/17] Reword and tidy up, currently fixing recording issue. --- doc/how_to/process_by_channel_group.rst | 239 ++++++++++++++++-------- doc/sg_execution_times.rst | 82 ++++++++ 2 files changed, 240 insertions(+), 81 deletions(-) create mode 100644 doc/sg_execution_times.rst diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index 1e186d4e6b..8e33afa207 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -1,128 +1,205 @@ -Processing a Recording by Channel Group (e.g. Shank) -=========================================================== +Processing a Recording by Channel Group +======================================= -In this tutorial, we will walk through how to preprocess and sorting a recording -per channel group. A channel group is a subset of channels grouped by some -feature - a typical example is grouping channels by shank in a Neuropixels recording. +In this tutorial, we will walk through how to preprocess and sort a recording +separately for *channel groups*. A channel group is a subset of channels grouped by some +feature - for example a multi-shank Neuropixels recording in which the channels +are grouped by shank. -First, we can check the channels on our recording are grouped as we expect. For example, -in the below example we have a X-shank neuropixels recording. We can visualise the -channel groupings with XXX. +**Why preprocess by channel group?** +Certain preprocessing steps depend on the spatial arrangement of the channels. +For example, common average referencing (CAR) averages over channels (separately for each time point) +and subtracts the average. In such a scenario it may make sense to group channels so that +this averaging is performed only over spatially close channel groups. -On our SpikeInterface recording, we can also access the channel groups per-channel +**Why sort by channel group?** -1) channel groups -2) index channel ids by groups +When sorting, we may want to completely separately channel groups so we can +consider their signals in isolation. If recording from a long +silicon probe, we might want to sort different brain areas separately, +for example using a different sorter for the hippocampus, the thalamus, or the cerebellum. -Why would you want to preprocess or sort by group? - -1) -2) -3) Splitting a Recording by Channel Group -------------------------------------- -We can split a recording into mutliply recordings, one for each channel group, with the `split_by` method. +In this example, we create a 16-channel recording with 4 tetrodes. However this could +be any recording in which the channel are grouped in some way, for example +a 384 channel, 4 shank Neuropixels recording in which channel grouping represents the separate shanks. + +First, let's import the parts of SpikeInterface we need into Python, and generate our toy recording: + +.. code-block:: python + + import spikeinterface.extractors as se + import spikeinterface.preprocessing as spre + from spikeinterface import aggregate_channels + from probeinterface import generate_tetrode, ProbeGroup + import numpy as np + + recording, _ = se.toy_example(duration=[10.], num_segments=1, num_channels=16) + print(recording.get_channel_groups()) + # >>> [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] (here, all channels are in the same group) + + # create 4 tetrodes as a 'probegroup' + probegroup = ProbeGroup() + for i in range(4): + tetrode = generate_tetrode() + tetrode.set_device_channel_indices(np.arange(4) + i * 4) + probegroup.add_probe(tetrode) + + # set this to the recording + recording_4_tetrodes = recording.set_probegroup(probegroup, group_mode='by_probe') -Here, we split a recording by channel group to get a `split_recording_dict`. This is a dictionary -containing the recordings, split by channel group: + print(recording_4_tetrodes.get_property('group')) + # >>> [0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3] (now, channels are grouped by tetrode index) -``` -split_recording_dict = recording.split_by("group") -print(split_recording_dict) -XXXXX -``` +We can split a recording into multiple recordings, one for each channel group, with the :py:func:`~split_by` method. -Now, we are ready to preprocess and sort by channel group. +.. code-block:: python + split_recording_dict = recording.split_by("group") + +Splitting a recording by channel group returns a dictionary containing separate recordings, one for each channel group: + +.. code-block:: python + + print(split_recording_dict) + # {0: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB, 1: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB, 2: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB, 3: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s + # float32 dtype - 4.58 MiB} Preprocessing a Recording by Channel Group ------------------------------------------ -The essense of preprocessing by channel group is to first split the recording +The essence of preprocessing by channel group is to first split the recording into separate recordings, perform the preprocessing steps, then aggregate the channels back together. -Here, we loop over the split recordings, preprocessing each shank group -individually. At the end, we use the `aggregate_channels` function -to combine the per-shank recordings back together. +In the below example, we loop over the split recordings, preprocessing each channel group +individually. At the end, we use the :py:func:`~aggregate_channels` function +to combine the separate channel group recordings back together. -``` -preprocessed_recordings = [] -for recording in split_recordings_dict.values(): +.. code-block:: python + preprocessed_recordings = [] - shifted_recording = spre.phase_shift(recording) + # loop over the recordings contained in the dictionary + for chan_group_rec in split_recordings_dict.values(): - filtered_recording = spre.bandpass_filter(shifted_recording) + # Apply the preprocessing steps to the channel group in isolation + shifted_recording = spre.phase_shift(chan_group_rec) - referenced_recording = spre.common_reference(filtered_recording) + filtered_recording = spre.bandpass_filter(shifted_recording) - preprocessed_recordings.append(referenced_recording) + referenced_recording = spre.common_reference(filtered_recording) -combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) + preprocessed_recordings.append(referenced_recording) -``` + # Combine our preprocessed channel groups back together + combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) -It is strongly recommended to use the above structure to preprocess by group. -A discussion of the subtleties of the approach may be found in the below -Notes section for the interested reader. +Now, when this recording is used in sorting, plotting, or whenever +calling it's :py:func:`~get_traces` method, the data will have been +preprocessed separately per-channel group (then concatenated +back together under the hood). -# Preprocessing channel in depth +It is strongly recommended to use the above structure to preprocess by channel group. +A discussion of the subtleties of the :py:func:`~aggregate_channels` may be found in +the below section for the interested reader. -Preprocessing and aggregation of channels is very flexible. Under the hood, -`aggregate_channels` keeps track of when a recording was split. When `get_traces()` -is called, the preprocessing is still performed per-group, even though the -recording is now aggregated. +.. warning:: + It is not recommended to apply :py:func:`~aggregate_channels` more than once + as this will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. -However, to ensure data is preprocessed by shank, the preprocessing step must be -applied per-group. For example, the below example will NOT preprocess by shank: -``` -split_recording = recording.split_by("group") -split_recording_as_list = list(**split_recording.values()) -combined_recording = aggregate_channels(split_recording_as_list) +Sorting a Recording by Channel Group +------------------------------------ -# will NOT preprocess by channel group. -filtered_recording = common_reference(combined_recording) +We can also sort a recording for each channel group separately. It is not necessary to preprocess +a recording by channel group in order to sort by channel group. We can perform sorting on a recording +whichever way it was preprocessed. -``` +There are two ways to sort a recording by channel group. First, we can split the preprocessed +recording (or, if it was already split during preprocessing as above, skip the :py:func:`~aggregate_channels` step +directly use the :py:func:`~split_recording_dict`). -Similarly, in the below example the first preprocessing step (bandpass filter) -would applied by group (although, this would have no effect in practice -as this preprocessing step is always performed per channel). However, -common referencing (which is effected by splitting by group) will -not be applied per group: +**Option 1: Manual splitting** -``` -split_recording = recording.split_by("group") +In this example, similar to above we loop over all preprocessed recordings that +are groups by channel, and apply the sorting separately. We store the +sorting objects in a dictionary for later use. -filtered_recording = [] -for recording in split_recording.values() - filtered_recording.append(spre.bandpass_filtered(recording)) +.. code-block:: python -combined_recording = aggregate_channels(filtered_recording) + split_preprocessed_recording = preprocessed_recording.split_by("group") -# As the recording has been combined, common referencing -# will NOT be applied per channel group. -referenced_recording = spre.common_reference(combined_recording). -``` + sortings = {} + for group, sub_recording in split_preprocessed_recording.items(): + sorting = run_sorter( + sorter_name='kilosort2', + recording=split_preprocessed_recording, + output_folder=f"folder_KS2_group{group}" + ) + sortings[group] = sorting -Finally, it is not recommended to apply `aggregate_channels` more than once -as this will slow down `get_traces()` and may result in unpredictable behaviour. +**Option 2 : Automatic splitting** +Alternatively, SpikeInterface provides a convenience function to sort the recording by property: -Sorting a Recording by Channel Group ------------------------------------- +.. code-block:: python + + aggregate_sorting = run_sorter_by_property( + sorter_name='kilosort2', + recording=recording_4_tetrodes, + grouping_property='group', + working_folder='working_path' + ) + + +Further notes on preprocessing by channel group +----------------------------------------------- + +.. note:: + + The splitting and aggregation of channels for preprocessing is flexible. + Under the hood, :py:func:`~aggregate_channels` keeps track of when a recording was split. When + :py:func:`~get_traces` is called, the preprocessing is still performed per-group, + even though the recording is now aggregated. + + However, to ensure data is preprocessed by channel group, the preprocessing step must be + applied separately to each split channel group recording. + For example, the below example will NOT preprocess by shank: + + .. code-block:: python + + split_recording = recording.split_by("group") + split_recording_as_list = list(**split_recording.values()) + combined_recording = aggregate_channels(split_recording_as_list) + + # will NOT preprocess by channel group. + filtered_recording = common_reference(combined_recording) + + Similarly, in the below example the first preprocessing step (bandpass filter) + would applied by group (although, this would have no effect in practice + as this preprocessing step is always separately for each individual channel). + + However, in the below example common referencing (does operate across + separate channels) will not be applied per channel group: + + .. code-block:: python -Sorting a recording can be performed in two ways. One, is to split the -recording by group and use `run_sorter` (LINK) separately on each preprocessed -channel group. + split_recording = recording.split_by("group") -Altearntively, SpikeInterface proves a conveniecne function XXX -to this. + filtered_recording = [] + for recording in split_recording.values() + filtered_recording.append(spre.bandpass_filtered(recording)) -Example + combined_recording = aggregate_channels(filtered_recording) -Done! + # As the recording has been combined, common referencing + # will NOT be applied per channel group. But, the bandpass + # filter step will be applied per channel group. + referenced_recording = spre.common_reference(combined_recording). diff --git a/doc/sg_execution_times.rst b/doc/sg_execution_times.rst new file mode 100644 index 0000000000..ac9bb37470 --- /dev/null +++ b/doc/sg_execution_times.rst @@ -0,0 +1,82 @@ + +:orphan: + +.. _sphx_glr_sg_execution_times: + + +Computation times +================= +**00:04.653** total execution time for 16 files **from all galleries**: + +.. container:: + + .. raw:: html + + + + + + + + .. list-table:: + :header-rows: 1 + :class: table table-striped sg-datatable + + * - Example + - Time + - Mem (MB) + * - :ref:`sphx_glr_modules_gallery_core_plot_4_waveform_extractor.py` (``..\examples\modules_gallery\core\plot_4_waveform_extractor.py``) + - 00:02.203 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) + - 00:00.856 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_extractors_plot_1_read_various_formats.py` (``..\examples\modules_gallery\extractors\plot_1_read_various_formats.py``) + - 00:00.675 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_3_quality_mertics.py` (``..\examples\modules_gallery\qualitymetrics\plot_3_quality_mertics.py``) + - 00:00.337 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_3_waveforms_gallery.py` (``..\examples\modules_gallery\widgets\plot_3_waveforms_gallery.py``) + - 00:00.335 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) + - 00:00.247 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_comparison_generate_erroneous_sorting.py` (``..\examples\modules_gallery\comparison\generate_erroneous_sorting.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_comparison_plot_5_comparison_sorter_weaknesses.py` (``..\examples\modules_gallery\comparison\plot_5_comparison_sorter_weaknesses.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_1_recording_extractor.py` (``..\examples\modules_gallery\core\plot_1_recording_extractor.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_2_sorting_extractor.py` (``..\examples\modules_gallery\core\plot_2_sorting_extractor.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_3_handle_probe_info.py` (``..\examples\modules_gallery\core\plot_3_handle_probe_info.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_5_append_concatenate_segments.py` (``..\examples\modules_gallery\core\plot_5_append_concatenate_segments.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_core_plot_6_handle_times.py` (``..\examples\modules_gallery\core\plot_6_handle_times.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_extractors_plot_2_working_with_unscaled_traces.py` (``..\examples\modules_gallery\extractors\plot_2_working_with_unscaled_traces.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_1_rec_gallery.py` (``..\examples\modules_gallery\widgets\plot_1_rec_gallery.py``) + - 00:00.000 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_2_sort_gallery.py` (``..\examples\modules_gallery\widgets\plot_2_sort_gallery.py``) + - 00:00.000 + - 0.0 From b4faa9af698617ba0e1cea90e13f1372699828e5 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Mon, 11 Dec 2023 13:56:13 +0000 Subject: [PATCH 03/17] Update example to be neuropixels-like. --- doc/how_to/process_by_channel_group.rst | 42 ++++++++++++++++--------- 1 file changed, 27 insertions(+), 15 deletions(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index 8e33afa207..4530b4129b 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -38,22 +38,34 @@ First, let's import the parts of SpikeInterface we need into Python, and generat from probeinterface import generate_tetrode, ProbeGroup import numpy as np - recording, _ = se.toy_example(duration=[10.], num_segments=1, num_channels=16) - print(recording.get_channel_groups()) - # >>> [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] (here, all channels are in the same group) - - # create 4 tetrodes as a 'probegroup' - probegroup = ProbeGroup() - for i in range(4): - tetrode = generate_tetrode() - tetrode.set_device_channel_indices(np.arange(4) + i * 4) - probegroup.add_probe(tetrode) + # Create a toy 384 channel recording with 4 shanks (each shank contain 96 channels) + recording, _ = se.toy_example(duration=[1.00], num_segments=1, num_channels=384) + four_shank_groupings = np.repeat([0, 1, 2, 3], 96) + recording.set_property("group", four_shank_groupings) - # set this to the recording - recording_4_tetrodes = recording.set_probegroup(probegroup, group_mode='by_probe') + recording.get_property("location") - print(recording_4_tetrodes.get_property('group')) - # >>> [0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3] (now, channels are grouped by tetrode index) + print(recording.get_channel_groups()) + """ + array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, + 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, + 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]) + """ We can split a recording into multiple recordings, one for each channel group, with the :py:func:`~split_by` method. @@ -153,7 +165,7 @@ Alternatively, SpikeInterface provides a convenience function to sort the record aggregate_sorting = run_sorter_by_property( sorter_name='kilosort2', - recording=recording_4_tetrodes, + recording=preprocessed_recording, grouping_property='group', working_folder='working_path' ) From 1506fdad464a68e323d1849f1461439f3276bc11 Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Mon, 11 Dec 2023 14:05:35 +0000 Subject: [PATCH 04/17] Tidy up and check. --- doc/how_to/process_by_channel_group.rst | 41 ++++++++++++------------- doc/sg_execution_times.rst | 20 ++++++------ 2 files changed, 30 insertions(+), 31 deletions(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index 4530b4129b..f3a8db615c 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -24,9 +24,9 @@ for example using a different sorter for the hippocampus, the thalamus, or the c Splitting a Recording by Channel Group -------------------------------------- -In this example, we create a 16-channel recording with 4 tetrodes. However this could +In this example, we create a 384-channel recording with 4 shanks. However this could be any recording in which the channel are grouped in some way, for example -a 384 channel, 4 shank Neuropixels recording in which channel grouping represents the separate shanks. +a multi-tetrode recording with channel groups representing the channels on each individual tetrodes. First, let's import the parts of SpikeInterface we need into Python, and generate our toy recording: @@ -43,8 +43,6 @@ First, let's import the parts of SpikeInterface we need into Python, and generat four_shank_groupings = np.repeat([0, 1, 2, 3], 96) recording.set_property("group", four_shank_groupings) - recording.get_property("location") - print(recording.get_channel_groups()) """ array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, @@ -78,11 +76,13 @@ Splitting a recording by channel group returns a dictionary containing separate .. code-block:: python print(split_recording_dict) - # {0: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB, 1: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB, 2: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB, 3: ChannelSliceRecording: 4 channels - 30.0kHz - 1 segments - 300,000 samples - 10.00s - # float32 dtype - 4.58 MiB} + """ + {0: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB, 1: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB, 2: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB, 3: ChannelSliceRecording: 96 channels - 30.0kHz - 1 segments - 30,000 samples - 1.00s - float32 dtype + 10.99 MiB} + """ Preprocessing a Recording by Channel Group ------------------------------------------ @@ -120,19 +120,18 @@ back together under the hood). It is strongly recommended to use the above structure to preprocess by channel group. A discussion of the subtleties of the :py:func:`~aggregate_channels` may be found in -the below section for the interested reader. +the end of this tutorial for the interested reader. .. warning:: - It is not recommended to apply :py:func:`~aggregate_channels` more than once - as this will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. + It is not recommended to apply :py:func:`~aggregate_channels` more than once. + This will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. Sorting a Recording by Channel Group ------------------------------------ We can also sort a recording for each channel group separately. It is not necessary to preprocess -a recording by channel group in order to sort by channel group. We can perform sorting on a recording -whichever way it was preprocessed. +a recording by channel group in order to sort by channel group. There are two ways to sort a recording by channel group. First, we can split the preprocessed recording (or, if it was already split during preprocessing as above, skip the :py:func:`~aggregate_channels` step @@ -141,7 +140,7 @@ directly use the :py:func:`~split_recording_dict`). **Option 1: Manual splitting** In this example, similar to above we loop over all preprocessed recordings that -are groups by channel, and apply the sorting separately. We store the +are grouped by channel, and apply the sorting separately. We store the sorting objects in a dictionary for later use. .. code-block:: python @@ -181,9 +180,9 @@ Further notes on preprocessing by channel group :py:func:`~get_traces` is called, the preprocessing is still performed per-group, even though the recording is now aggregated. - However, to ensure data is preprocessed by channel group, the preprocessing step must be + To ensure data is preprocessed by channel group, the preprocessing step must be applied separately to each split channel group recording. - For example, the below example will NOT preprocess by shank: + For example, the below example will NOT preprocess by channel group: .. code-block:: python @@ -194,11 +193,11 @@ Further notes on preprocessing by channel group # will NOT preprocess by channel group. filtered_recording = common_reference(combined_recording) - Similarly, in the below example the first preprocessing step (bandpass filter) - would applied by group (although, this would have no effect in practice - as this preprocessing step is always separately for each individual channel). + In the below example the first preprocessing step (bandpass filter) + would applied by channel group (although, in practice this would have no effect + as filtering is always applied separately to each individual channel). - However, in the below example common referencing (does operate across + However, common referencing (does operate across separate channels) will not be applied per channel group: .. code-block:: python diff --git a/doc/sg_execution_times.rst b/doc/sg_execution_times.rst index ac9bb37470..38453ec0e7 100644 --- a/doc/sg_execution_times.rst +++ b/doc/sg_execution_times.rst @@ -6,7 +6,7 @@ Computation times ================= -**00:04.653** total execution time for 16 files **from all galleries**: +**00:03.838** total execution time for 16 files **from all galleries**: .. container:: @@ -33,22 +33,22 @@ Computation times - Time - Mem (MB) * - :ref:`sphx_glr_modules_gallery_core_plot_4_waveform_extractor.py` (``..\examples\modules_gallery\core\plot_4_waveform_extractor.py``) - - 00:02.203 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) - - 00:00.856 + - 00:02.038 - 0.0 * - :ref:`sphx_glr_modules_gallery_extractors_plot_1_read_various_formats.py` (``..\examples\modules_gallery\extractors\plot_1_read_various_formats.py``) - - 00:00.675 + - 00:00.640 + - 0.0 + * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) + - 00:00.303 - 0.0 * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_3_quality_mertics.py` (``..\examples\modules_gallery\qualitymetrics\plot_3_quality_mertics.py``) - - 00:00.337 + - 00:00.301 - 0.0 * - :ref:`sphx_glr_modules_gallery_widgets_plot_3_waveforms_gallery.py` (``..\examples\modules_gallery\widgets\plot_3_waveforms_gallery.py``) - - 00:00.335 + - 00:00.285 - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) - - 00:00.247 + * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) + - 00:00.271 - 0.0 * - :ref:`sphx_glr_modules_gallery_comparison_generate_erroneous_sorting.py` (``..\examples\modules_gallery\comparison\generate_erroneous_sorting.py``) - 00:00.000 From 47049c92e4e38e84f18443a72374d414802598cf Mon Sep 17 00:00:00 2001 From: Alessio Buccino Date: Fri, 19 Jan 2024 11:17:34 +0100 Subject: [PATCH 05/17] Update doc/how_to/process_by_channel_group.rst Co-authored-by: Heberto Mayorquin --- doc/how_to/process_by_channel_group.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index f3a8db615c..c5c60ab889 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -194,7 +194,7 @@ Further notes on preprocessing by channel group filtered_recording = common_reference(combined_recording) In the below example the first preprocessing step (bandpass filter) - would applied by channel group (although, in practice this would have no effect + would be applied by channel group (although, in practice this would have no effect as filtering is always applied separately to each individual channel). However, common referencing (does operate across From 9feee09067b7954225ec0c48bbbb8fd75475a9bc Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Tue, 13 Feb 2024 11:47:43 +0000 Subject: [PATCH 06/17] Remove execution times .rst --- doc/sg_execution_times.rst | 82 -------------------------------------- 1 file changed, 82 deletions(-) delete mode 100644 doc/sg_execution_times.rst diff --git a/doc/sg_execution_times.rst b/doc/sg_execution_times.rst deleted file mode 100644 index 38453ec0e7..0000000000 --- a/doc/sg_execution_times.rst +++ /dev/null @@ -1,82 +0,0 @@ - -:orphan: - -.. _sphx_glr_sg_execution_times: - - -Computation times -================= -**00:03.838** total execution time for 16 files **from all galleries**: - -.. container:: - - .. raw:: html - - - - - - - - .. list-table:: - :header-rows: 1 - :class: table table-striped sg-datatable - - * - Example - - Time - - Mem (MB) - * - :ref:`sphx_glr_modules_gallery_core_plot_4_waveform_extractor.py` (``..\examples\modules_gallery\core\plot_4_waveform_extractor.py``) - - 00:02.038 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_extractors_plot_1_read_various_formats.py` (``..\examples\modules_gallery\extractors\plot_1_read_various_formats.py``) - - 00:00.640 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_4_peaks_gallery.py` (``..\examples\modules_gallery\widgets\plot_4_peaks_gallery.py``) - - 00:00.303 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_3_quality_mertics.py` (``..\examples\modules_gallery\qualitymetrics\plot_3_quality_mertics.py``) - - 00:00.301 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_3_waveforms_gallery.py` (``..\examples\modules_gallery\widgets\plot_3_waveforms_gallery.py``) - - 00:00.285 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_qualitymetrics_plot_4_curation.py` (``..\examples\modules_gallery\qualitymetrics\plot_4_curation.py``) - - 00:00.271 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_comparison_generate_erroneous_sorting.py` (``..\examples\modules_gallery\comparison\generate_erroneous_sorting.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_comparison_plot_5_comparison_sorter_weaknesses.py` (``..\examples\modules_gallery\comparison\plot_5_comparison_sorter_weaknesses.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_1_recording_extractor.py` (``..\examples\modules_gallery\core\plot_1_recording_extractor.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_2_sorting_extractor.py` (``..\examples\modules_gallery\core\plot_2_sorting_extractor.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_3_handle_probe_info.py` (``..\examples\modules_gallery\core\plot_3_handle_probe_info.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_5_append_concatenate_segments.py` (``..\examples\modules_gallery\core\plot_5_append_concatenate_segments.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_core_plot_6_handle_times.py` (``..\examples\modules_gallery\core\plot_6_handle_times.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_extractors_plot_2_working_with_unscaled_traces.py` (``..\examples\modules_gallery\extractors\plot_2_working_with_unscaled_traces.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_1_rec_gallery.py` (``..\examples\modules_gallery\widgets\plot_1_rec_gallery.py``) - - 00:00.000 - - 0.0 - * - :ref:`sphx_glr_modules_gallery_widgets_plot_2_sort_gallery.py` (``..\examples\modules_gallery\widgets\plot_2_sort_gallery.py``) - - 00:00.000 - - 0.0 From 738dc6a576e7d56fa0413d5df5d225f407ee667f Mon Sep 17 00:00:00 2001 From: JoeZiminski Date: Tue, 13 Feb 2024 12:29:37 +0000 Subject: [PATCH 07/17] Integrate the end section of the page into the text. --- doc/how_to/process_by_channel_group.rst | 72 ++++++++----------------- 1 file changed, 22 insertions(+), 50 deletions(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index c5c60ab889..d88f4d3120 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -119,11 +119,29 @@ preprocessed separately per-channel group (then concatenated back together under the hood). It is strongly recommended to use the above structure to preprocess by channel group. -A discussion of the subtleties of the :py:func:`~aggregate_channels` may be found in -the end of this tutorial for the interested reader. -.. warning:: - It is not recommended to apply :py:func:`~aggregate_channels` more than once. +.. note:: + + The splitting and aggregation of channels for preprocessing is flexible. + Under the hood, :py:func:`~aggregate_channels` keeps track of when a recording was split. When + :py:func:`~get_traces` is called, the preprocessing is still performed per-group, + even though the recording is now aggregated. + + To ensure data is preprocessed by channel group, the preprocessing step must be + applied separately to each split channel group recording. + For example, the below example will NOT preprocess by channel group: + + .. code-block:: python + + split_recording = recording.split_by("group") + split_recording_as_list = list(**split_recording.values()) + combined_recording = aggregate_channels(split_recording_as_list) + + # will NOT preprocess by channel group. + filtered_recording = common_reference(combined_recording) + + + In general, it is not recommended to apply :py:func:`~aggregate_channels` more than once. This will slow down :py:func:`~get_traces` calls and may result in unpredictable behaviour. @@ -168,49 +186,3 @@ Alternatively, SpikeInterface provides a convenience function to sort the record grouping_property='group', working_folder='working_path' ) - - -Further notes on preprocessing by channel group ------------------------------------------------ - -.. note:: - - The splitting and aggregation of channels for preprocessing is flexible. - Under the hood, :py:func:`~aggregate_channels` keeps track of when a recording was split. When - :py:func:`~get_traces` is called, the preprocessing is still performed per-group, - even though the recording is now aggregated. - - To ensure data is preprocessed by channel group, the preprocessing step must be - applied separately to each split channel group recording. - For example, the below example will NOT preprocess by channel group: - - .. code-block:: python - - split_recording = recording.split_by("group") - split_recording_as_list = list(**split_recording.values()) - combined_recording = aggregate_channels(split_recording_as_list) - - # will NOT preprocess by channel group. - filtered_recording = common_reference(combined_recording) - - In the below example the first preprocessing step (bandpass filter) - would be applied by channel group (although, in practice this would have no effect - as filtering is always applied separately to each individual channel). - - However, common referencing (does operate across - separate channels) will not be applied per channel group: - - .. code-block:: python - - split_recording = recording.split_by("group") - - filtered_recording = [] - for recording in split_recording.values() - filtered_recording.append(spre.bandpass_filtered(recording)) - - combined_recording = aggregate_channels(filtered_recording) - - # As the recording has been combined, common referencing - # will NOT be applied per channel group. But, the bandpass - # filter step will be applied per channel group. - referenced_recording = spre.common_reference(combined_recording). From 0c1159885752676128581006424ba63bcf916d1e Mon Sep 17 00:00:00 2001 From: Samuel Garcia Date: Wed, 6 Mar 2024 14:06:03 +0100 Subject: [PATCH 08/17] Update yaml files for installation + add a new one --- installation_tips/README.md | 11 ++++-- ...interface_environement_rolling_updates.yml | 35 +++++++++++++++++++ ...spikeinterface_environment_linux_dandi.yml | 31 ++++++++-------- .../full_spikeinterface_environment_mac.yml | 27 +++++++------- ...ull_spikeinterface_environment_windows.yml | 20 ++++++----- 5 files changed, 82 insertions(+), 42 deletions(-) create mode 100644 installation_tips/full_spikeinterface_environement_rolling_updates.yml diff --git a/installation_tips/README.md b/installation_tips/README.md index 66d3f94a02..f0090a4790 100644 --- a/installation_tips/README.md +++ b/installation_tips/README.md @@ -9,8 +9,6 @@ This environment will install: * spikeinterface-gui * phy * tridesclous - * spyking-circus (not on mac) - * herdingspikes (not on windows) Kilosort, Ironclust and HDSort are MATLAB based and need to be installed from source. Klusta does not work anymore with python3.8 you should create a similar environment with python3.6. @@ -19,7 +17,7 @@ Klusta does not work anymore with python3.8 you should create a similar environm Steps: -1. Download anaconda individual edition [here](https://www.anaconda.com/products/individual) +1. Download anaconda individual edition [here](https://www.anaconda.com/download) 2. Run the installer. Check the box “Add anaconda3 to my Path environment variable”. It makes life easier for beginners. 3. Download with right click + save the file corresponding to your OS, and put it in "Documents" folder * [`full_spikeinterface_environment_windows.yml`](https://raw.githubusercontent.com/SpikeInterface/spikeinterface/master/installation_tips/full_spikeinterface_environment_windows.yml) @@ -64,3 +62,10 @@ This script tests the following: * running herdingspikes (not on windows) * opening the spikeinterface-gui * exporting to Phy + + +## Installing before release + +Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, python-neo, sortingview). +We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `full_spikeinterface_environement_rolling_updates.yml` file to create the environement. This will install packages of the ecosystem from source. +This is a good way to test if patch fix your issue. diff --git a/installation_tips/full_spikeinterface_environement_rolling_updates.yml b/installation_tips/full_spikeinterface_environement_rolling_updates.yml new file mode 100644 index 0000000000..8eff19c8c3 --- /dev/null +++ b/installation_tips/full_spikeinterface_environement_rolling_updates.yml @@ -0,0 +1,35 @@ +name: si_env +channels: + - conda-forge + - defaults +dependencies: + - python=3.11 + - pip + - numpy + - scipy + - joblib + - tqdm + - matplotlib + - h5py + - pandas + - xarray + - hdbscan + - scikit-learn + - networkx + - pybind11 + - loky + - numba + - jupyter + - pyqt=5 + - pyqtgraph + - ipywidgets + - ipympl + - pip: + - ephyviewer + - docker + - https://github.com/SpikeInterface/MEArec/archive/main.zip + - https://github.com/NeuralEnsemble/python-neo/archive/master.zip + - https://github.com/SpikeInterface/probeinterface/archive/main.zip + - https://github.com/SpikeInterface/spikeinterface/archive/main.zip + - https://github.com/SpikeInterface/spikeinterface-gui/archive/main.zip + - https://github.com/magland/sortingview/archive/main.zip diff --git a/installation_tips/full_spikeinterface_environment_linux_dandi.yml b/installation_tips/full_spikeinterface_environment_linux_dandi.yml index 2ed176b16c..0e885d2a8d 100755 --- a/installation_tips/full_spikeinterface_environment_linux_dandi.yml +++ b/installation_tips/full_spikeinterface_environment_linux_dandi.yml @@ -3,37 +3,38 @@ channels: - conda-forge - defaults dependencies: - - python=3.10 - - pip>=21.0 - - mamba - - numpy<1.22 + - python=3.11 + - pip + - numpy + - scipy - joblib - tqdm - matplotlib - h5py - pandas + - xarray + - zarr - scikit-learn + - hdbscan - networkx - pybind11 - loky - - hdbscan - numba - jupyter - - mpi4py - - cxx-compiler - - libxcb - pyqt=5 - pyqtgraph - - htop - ipywidgets - ipympl + - htop + - cxx-compiler + - libxcb - pip: - ephyviewer - - neo>=0.12 - - probeinterface>=0.2.11 - - MEArec>=1.8 - - spikeinterface[full, widgets] + - neo>=0.13 + - probeinterface + - MEArec + - spikeinterface[full,widgets] - spikeinterface-gui - - tridesclous>=1.6.8 - - spyking-circus>=1.1.0 + - tridesclous # - phy==2.0b5 + \ No newline at end of file diff --git a/installation_tips/full_spikeinterface_environment_mac.yml b/installation_tips/full_spikeinterface_environment_mac.yml index 7ce4a149cc..e5a900e54f 100755 --- a/installation_tips/full_spikeinterface_environment_mac.yml +++ b/installation_tips/full_spikeinterface_environment_mac.yml @@ -3,37 +3,34 @@ channels: - conda-forge - defaults dependencies: - - python=3.10 - - pip>=21.0 + - python=3.11 + - pip - numpy + - scipy - joblib - tqdm - matplotlib - h5py - pandas + - xarray + - zarr - scikit-learn + - hdbscan - networkx - pybind11 - loky - - hdbscan - numba -# - jupyter - - mpi - - mpi4py - - compilers + - jupyter - pyqt=5 - pyqtgraph - ipywidgets - ipympl - pip: -# - PyQt5 - ephyviewer - - neo>=0.12 - - probeinterface>=0.2.11 - - MEArec>=1.8 - - spikeinterface[full, widgets] + - neo>=0.13 + - probeinterface + - MEArec + - spikeinterface[full,widgets] - spikeinterface-gui - - tridesclous>=1.6.8 + - tridesclous # - phy==2.0b5 - # - mountainsort4>=1.0.0 isosplit5 fails on pip install for mac - # - mountainsort5>=0.3.0 diff --git a/installation_tips/full_spikeinterface_environment_windows.yml b/installation_tips/full_spikeinterface_environment_windows.yml index 38c26e6a78..e5a900e54f 100755 --- a/installation_tips/full_spikeinterface_environment_windows.yml +++ b/installation_tips/full_spikeinterface_environment_windows.yml @@ -3,20 +3,22 @@ channels: - conda-forge - defaults dependencies: - - python=3.10 - - pip>=21.0 - # numpy 1.21 break numba which break tridesclous + - python=3.11 + - pip - numpy + - scipy - joblib - tqdm - matplotlib - h5py - pandas + - xarray + - zarr - scikit-learn + - hdbscan - networkx - pybind11 - loky - - hdbscan - numba - jupyter - pyqt=5 @@ -25,10 +27,10 @@ dependencies: - ipympl - pip: - ephyviewer - - neo>=0.12 - - probeinterface>=0.2.11 - - MEArec>=1.8 - - spikeinterface[full, widgets] + - neo>=0.13 + - probeinterface + - MEArec + - spikeinterface[full,widgets] - spikeinterface-gui - - tridesclous>=1.6.8 + - tridesclous # - phy==2.0b5 From d7a904e2d3c9285e6d85dc3accb3ae158df77f6d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 6 Mar 2024 13:07:01 +0000 Subject: [PATCH 09/17] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- .../full_spikeinterface_environment_linux_dandi.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/installation_tips/full_spikeinterface_environment_linux_dandi.yml b/installation_tips/full_spikeinterface_environment_linux_dandi.yml index 0e885d2a8d..321ccc432a 100755 --- a/installation_tips/full_spikeinterface_environment_linux_dandi.yml +++ b/installation_tips/full_spikeinterface_environment_linux_dandi.yml @@ -37,4 +37,4 @@ dependencies: - spikeinterface-gui - tridesclous # - phy==2.0b5 - \ No newline at end of file + From 6cd6344750be2d44997eb58aa5b00d6a068f0892 Mon Sep 17 00:00:00 2001 From: Garcia Samuel Date: Wed, 6 Mar 2024 14:37:37 +0100 Subject: [PATCH 10/17] Apply suggestions from code review Co-authored-by: Zach McKenzie <92116279+zm711@users.noreply.github.com> --- installation_tips/README.md | 2 +- .../full_spikeinterface_environement_rolling_updates.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/installation_tips/README.md b/installation_tips/README.md index f0090a4790..426d1e58db 100644 --- a/installation_tips/README.md +++ b/installation_tips/README.md @@ -67,5 +67,5 @@ This script tests the following: ## Installing before release Some tools in the spikeinteface ecosystem are getting regular bug fixes (spikeinterface, spikeinterface-gui, probeinterface, python-neo, sortingview). -We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `full_spikeinterface_environement_rolling_updates.yml` file to create the environement. This will install packages of the ecosystem from source. +We are making releases 2 to 4 times a year. In between releases if you want to install from source you can use the `full_spikeinterface_environment_rolling_updates.yml` file to create the environment. This will install the packages of the ecosystem from source. This is a good way to test if patch fix your issue. diff --git a/installation_tips/full_spikeinterface_environement_rolling_updates.yml b/installation_tips/full_spikeinterface_environement_rolling_updates.yml index 8eff19c8c3..b4479aa20f 100644 --- a/installation_tips/full_spikeinterface_environement_rolling_updates.yml +++ b/installation_tips/full_spikeinterface_environement_rolling_updates.yml @@ -1,4 +1,4 @@ -name: si_env +name: si_env_rolling channels: - conda-forge - defaults From b253fa08880703b1ea03b7f70f241254bd5bfb85 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 6 Mar 2024 13:37:54 +0000 Subject: [PATCH 11/17] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- .../full_spikeinterface_environment_linux_dandi.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/installation_tips/full_spikeinterface_environment_linux_dandi.yml b/installation_tips/full_spikeinterface_environment_linux_dandi.yml index 321ccc432a..b16b1bd4ae 100755 --- a/installation_tips/full_spikeinterface_environment_linux_dandi.yml +++ b/installation_tips/full_spikeinterface_environment_linux_dandi.yml @@ -37,4 +37,3 @@ dependencies: - spikeinterface-gui - tridesclous # - phy==2.0b5 - From 3ce821b3dbdcb9c5f6ab6cf5dcdca37d02cafefe Mon Sep 17 00:00:00 2001 From: Samuel Garcia Date: Wed, 6 Mar 2024 15:55:00 +0100 Subject: [PATCH 12/17] Update check_your_install.py and yaml --- installation_tips/README.md | 1 - installation_tips/check_your_install.py | 76 +++++++++++-------- ...spikeinterface_environment_linux_dandi.yml | 2 - .../full_spikeinterface_environment_mac.yml | 2 - ...ull_spikeinterface_environment_windows.yml | 2 - 5 files changed, 45 insertions(+), 38 deletions(-) diff --git a/installation_tips/README.md b/installation_tips/README.md index 426d1e58db..59209d01bc 100644 --- a/installation_tips/README.md +++ b/installation_tips/README.md @@ -11,7 +11,6 @@ This environment will install: * tridesclous Kilosort, Ironclust and HDSort are MATLAB based and need to be installed from source. -Klusta does not work anymore with python3.8 you should create a similar environment with python3.6. ### Quick installation diff --git a/installation_tips/check_your_install.py b/installation_tips/check_your_install.py index 4e01beb901..f3f80961e8 100644 --- a/installation_tips/check_your_install.py +++ b/installation_tips/check_your_install.py @@ -17,41 +17,54 @@ def _create_recording(): rec.save(folder='./toy_example_recording') -def _run_one_sorter_and_exctract_wf(sorter_name): +def _run_one_sorter_and_analyzer(sorter_name): + job_kwargs = dict(n_jobs=-1, progress_bar=True, chunk_duration="1s") import spikeinterface.full as si - rec = si.load_extractor('./toy_example_recording') - sorting = si.run_sorter(sorter_name, rec, output_folder=f'{sorter_name}_output', verbose=False) - si.extract_waveforms(rec, sorting, f'{sorter_name}_waveforms', - n_jobs=1, total_memory="10M", max_spikes_per_unit=500, return_scaled=False) - -def run_tridesclous(): - _run_one_sorter_and_exctract_wf('tridesclous') + recording = si.load_extractor('./toy_example_recording') + sorting = si.run_sorter(sorter_name, recording, output_folder=f'./sorter_with_{sorter_name}', verbose=False) + + sorting_analyzer = si.create_sorting_analyzer(sorting, recording, + format="binary_folder", folder=f"./analyzer_with_{sorter_name}", + **job_kwargs) + sorting_analyzer.compute("random_spikes", method="uniform", max_spikes_per_unit=500) + sorting_analyzer.compute("waveforms", **job_kwargs) + sorting_analyzer.compute("templates") + sorting_analyzer.compute("noise_levels") + sorting_analyzer.compute("unit_locations", method="monopolar_triangulation") + sorting_analyzer.compute("correlograms", window_ms=100, bin_ms=5.) + sorting_analyzer.compute("principal_components", n_components=3, mode='by_channel_global', whiten=True, **job_kwargs) + sorting_analyzer.compute("quality_metrics", metric_names=["snr", "firing_rate"]) -def run_spykingcircus(): - _run_one_sorter_and_exctract_wf('spykingcircus') +def run_tridesclous(): + _run_one_sorter_and_analyzer('tridesclous') +def run_tridesclous2(): + _run_one_sorter_and_analyzer('tridesclous2') -def run_herdingspikes(): - _run_one_sorter_and_exctract_wf('herdingspikes') def open_sigui(): import spikeinterface.full as si import spikeinterface_gui app = spikeinterface_gui.mkQApp() - waveform_forlder = 'tridesclous_waveforms' - we = si.WaveformExtractor.load_from_folder(waveform_forlder) - pc = si.compute_principal_components(we, n_components=3, mode='by_channel_local', whiten=True, dtype='float32') - win = spikeinterface_gui.MainWindow(we) + + sorter_name = "tridesclous2" + folder = f"./analyzer_with_{sorter_name}" + analyzer = si.load_sorting_analyzer(folder) + + win = spikeinterface_gui.MainWindow(analyzer) win.show() app.exec_() def export_to_phy(): import spikeinterface.full as si - we = si.WaveformExtractor.load_from_folder("tridesclous_waveforms") + sorter_name = "tridesclous2" + folder = f"./analyzer_with_{sorter_name}" + analyzer = si.load_sorting_analyzer(folder) + phy_folder = "./phy_example" - si.export_to_phy(we, output_folder=phy_folder, verbose=False) + si.export_to_phy(analyzer, output_folder=phy_folder, verbose=False) def open_phy(): @@ -61,10 +74,12 @@ def open_phy(): def _clean(): # clean folders = [ - 'toy_example_recording', - "tridesclous_output", "tridesclous_waveforms", - "spykingcircus_output", "spykingcircus_waveforms", - "phy_example" + "./toy_example_recording", + "./sorter_with_tridesclous", + "./analyzer_with_tridesclous", + "./sorter_with_tridesclous2", + "./analyzer_with_tridesclous2", + "./phy_example" ] for folder in folders: if Path(folder).exists(): @@ -86,25 +101,24 @@ def _clean(): ('Import spikeinterface', check_import_si), ('Import spikeinterface.full', check_import_si_full), ('Run tridesclous', run_tridesclous), + ('Run tridesclous2', run_tridesclous2), ] # backwards logic because default is True for end-user if args.ci: - steps.insert(3, ('Open spikeinterface-gui', open_sigui) ) + steps.append(('Open spikeinterface-gui', open_sigui)) steps.append(('Export to phy', export_to_phy)), # phy is removed from the env because it force a pip install PyQt5 # which break the conda env # ('Open phy', open_phy), - if platform.system() == "Windows": - pass - # steps.insert(3, ('Run spykingcircus', run_spykingcircus)) - elif platform.system() == "Darwin": - steps.insert(3, ('Run herdingspikes', run_herdingspikes)) - else: - steps.insert(3, ('Run spykingcircus', run_spykingcircus)) - steps.insert(4, ('Run herdingspikes', run_herdingspikes)) + # if platform.system() == "Windows": + # pass + # elif platform.system() == "Darwin": + # pass + # else: + # pass for label, func in steps: try: diff --git a/installation_tips/full_spikeinterface_environment_linux_dandi.yml b/installation_tips/full_spikeinterface_environment_linux_dandi.yml index b16b1bd4ae..5f276b0d20 100755 --- a/installation_tips/full_spikeinterface_environment_linux_dandi.yml +++ b/installation_tips/full_spikeinterface_environment_linux_dandi.yml @@ -30,8 +30,6 @@ dependencies: - libxcb - pip: - ephyviewer - - neo>=0.13 - - probeinterface - MEArec - spikeinterface[full,widgets] - spikeinterface-gui diff --git a/installation_tips/full_spikeinterface_environment_mac.yml b/installation_tips/full_spikeinterface_environment_mac.yml index e5a900e54f..2522fda78d 100755 --- a/installation_tips/full_spikeinterface_environment_mac.yml +++ b/installation_tips/full_spikeinterface_environment_mac.yml @@ -27,8 +27,6 @@ dependencies: - ipympl - pip: - ephyviewer - - neo>=0.13 - - probeinterface - MEArec - spikeinterface[full,widgets] - spikeinterface-gui diff --git a/installation_tips/full_spikeinterface_environment_windows.yml b/installation_tips/full_spikeinterface_environment_windows.yml index e5a900e54f..2522fda78d 100755 --- a/installation_tips/full_spikeinterface_environment_windows.yml +++ b/installation_tips/full_spikeinterface_environment_windows.yml @@ -27,8 +27,6 @@ dependencies: - ipympl - pip: - ephyviewer - - neo>=0.13 - - probeinterface - MEArec - spikeinterface[full,widgets] - spikeinterface-gui From 4d71d43e0395926351d6fd6fe4d670cbbe314591 Mon Sep 17 00:00:00 2001 From: Samuel Garcia Date: Wed, 6 Mar 2024 16:09:35 +0100 Subject: [PATCH 13/17] Add strict_gap_mode in read_neuralynx to reflect neo. --- .../extractors/neoextractors/neuralynx.py | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/src/spikeinterface/extractors/neoextractors/neuralynx.py b/src/spikeinterface/extractors/neoextractors/neuralynx.py index 88ecc69c53..3df8a8d255 100644 --- a/src/spikeinterface/extractors/neoextractors/neuralynx.py +++ b/src/spikeinterface/extractors/neoextractors/neuralynx.py @@ -3,11 +3,15 @@ from typing import Optional from pathlib import Path +from importlib.metadata import version + from spikeinterface.core.core_tools import define_function_from_class from .neobaseextractor import NeoBaseRecordingExtractor, NeoBaseSortingExtractor + + class NeuralynxRecordingExtractor(NeoBaseRecordingExtractor): """ Class for reading neuralynx folder @@ -27,22 +31,33 @@ class NeuralynxRecordingExtractor(NeoBaseRecordingExtractor): exlude_filename: list[str], default: None List of filename to exclude from the loading. For example, use `exclude_filename=["events.nev"]` to skip loading the event file. + strict_gap_mode: bool, default: False + See neo documentation. + Detect gaps using strict mode or not. + * strict_gap_mode = True then a gap is consider when timstamp difference between two + consequtive data packet is more than one sample interval. + * strict_gap_mode = False then a gap has an increased tolerance. Some new system with different clock need this option + otherwise, too many gaps are detected + Note hat here the default is False contrary to neo. """ mode = "folder" NeoRawIOClass = "NeuralynxRawIO" name = "neuralynx" - def __init__(self, folder_path, stream_id=None, stream_name=None, all_annotations=False, exclude_filename=None): - neo_kwargs = self.map_to_neo_kwargs(folder_path, exclude_filename) + def __init__(self, folder_path, stream_id=None, stream_name=None, all_annotations=False, exclude_filename=None, strict_gap_mode=False): + neo_kwargs = self.map_to_neo_kwargs(folder_path, exclude_filename, strict_gap_mode) NeoBaseRecordingExtractor.__init__( self, stream_id=stream_id, stream_name=stream_name, all_annotations=all_annotations, **neo_kwargs ) - self._kwargs.update(dict(folder_path=str(Path(folder_path).absolute()), exclude_filename=exclude_filename)) + self._kwargs.update(dict(folder_path=str(Path(folder_path).absolute()), exclude_filename=exclude_filename), strict_gap_mode=strict_gap_mode) @classmethod - def map_to_neo_kwargs(cls, folder_path, exclude_filename): + def map_to_neo_kwargs(cls, folder_path, exclude_filename, strict_gap_mode): neo_kwargs = {"dirname": str(folder_path), "exclude_filename": exclude_filename} + if version("neo") >= "0.13.1": + neo_kwargs["strict_gap_mode"] = strict_gap_mode + return neo_kwargs From 254985cee62d1487f52d57380a1004650b2de610 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 6 Mar 2024 15:10:47 +0000 Subject: [PATCH 14/17] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- .../extractors/neoextractors/neuralynx.py | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/src/spikeinterface/extractors/neoextractors/neuralynx.py b/src/spikeinterface/extractors/neoextractors/neuralynx.py index 3df8a8d255..cc270d8029 100644 --- a/src/spikeinterface/extractors/neoextractors/neuralynx.py +++ b/src/spikeinterface/extractors/neoextractors/neuralynx.py @@ -10,8 +10,6 @@ from .neobaseextractor import NeoBaseRecordingExtractor, NeoBaseSortingExtractor - - class NeuralynxRecordingExtractor(NeoBaseRecordingExtractor): """ Class for reading neuralynx folder @@ -34,7 +32,7 @@ class NeuralynxRecordingExtractor(NeoBaseRecordingExtractor): strict_gap_mode: bool, default: False See neo documentation. Detect gaps using strict mode or not. - * strict_gap_mode = True then a gap is consider when timstamp difference between two + * strict_gap_mode = True then a gap is consider when timstamp difference between two consequtive data packet is more than one sample interval. * strict_gap_mode = False then a gap has an increased tolerance. Some new system with different clock need this option otherwise, too many gaps are detected @@ -45,12 +43,23 @@ class NeuralynxRecordingExtractor(NeoBaseRecordingExtractor): NeoRawIOClass = "NeuralynxRawIO" name = "neuralynx" - def __init__(self, folder_path, stream_id=None, stream_name=None, all_annotations=False, exclude_filename=None, strict_gap_mode=False): + def __init__( + self, + folder_path, + stream_id=None, + stream_name=None, + all_annotations=False, + exclude_filename=None, + strict_gap_mode=False, + ): neo_kwargs = self.map_to_neo_kwargs(folder_path, exclude_filename, strict_gap_mode) NeoBaseRecordingExtractor.__init__( self, stream_id=stream_id, stream_name=stream_name, all_annotations=all_annotations, **neo_kwargs ) - self._kwargs.update(dict(folder_path=str(Path(folder_path).absolute()), exclude_filename=exclude_filename), strict_gap_mode=strict_gap_mode) + self._kwargs.update( + dict(folder_path=str(Path(folder_path).absolute()), exclude_filename=exclude_filename), + strict_gap_mode=strict_gap_mode, + ) @classmethod def map_to_neo_kwargs(cls, folder_path, exclude_filename, strict_gap_mode): From 0b420e9f60bca9d8b61651b52687265272392824 Mon Sep 17 00:00:00 2001 From: Garcia Samuel Date: Wed, 6 Mar 2024 19:27:18 +0100 Subject: [PATCH 15/17] Apply suggestions from code review Co-authored-by: Zach McKenzie <92116279+zm711@users.noreply.github.com> --- src/spikeinterface/extractors/neoextractors/neuralynx.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/spikeinterface/extractors/neoextractors/neuralynx.py b/src/spikeinterface/extractors/neoextractors/neuralynx.py index cc270d8029..cc89302e59 100644 --- a/src/spikeinterface/extractors/neoextractors/neuralynx.py +++ b/src/spikeinterface/extractors/neoextractors/neuralynx.py @@ -33,10 +33,10 @@ class NeuralynxRecordingExtractor(NeoBaseRecordingExtractor): See neo documentation. Detect gaps using strict mode or not. * strict_gap_mode = True then a gap is consider when timstamp difference between two - consequtive data packet is more than one sample interval. - * strict_gap_mode = False then a gap has an increased tolerance. Some new system with different clock need this option + consecutive data packets is more than one sample interval. + * strict_gap_mode = False then a gap has an increased tolerance. Some new systems with different clocks need this option otherwise, too many gaps are detected - Note hat here the default is False contrary to neo. + Note that here the default is False contrary to neo. """ mode = "folder" From 8aaddec05133b2ecc65a017531544a95e8622640 Mon Sep 17 00:00:00 2001 From: Samuel Garcia Date: Wed, 6 Mar 2024 20:53:37 +0100 Subject: [PATCH 16/17] oups --- ...es.yml => full_spikeinterface_environment_rolling_updates.yml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename installation_tips/{full_spikeinterface_environement_rolling_updates.yml => full_spikeinterface_environment_rolling_updates.yml} (100%) diff --git a/installation_tips/full_spikeinterface_environement_rolling_updates.yml b/installation_tips/full_spikeinterface_environment_rolling_updates.yml similarity index 100% rename from installation_tips/full_spikeinterface_environement_rolling_updates.yml rename to installation_tips/full_spikeinterface_environment_rolling_updates.yml From 649b807537c149eae431fb3e5f86ca262b93a305 Mon Sep 17 00:00:00 2001 From: Garcia Samuel Date: Fri, 8 Mar 2024 13:12:29 +0100 Subject: [PATCH 17/17] Update doc/how_to/process_by_channel_group.rst Co-authored-by: Zach McKenzie <92116279+zm711@users.noreply.github.com> --- doc/how_to/process_by_channel_group.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/how_to/process_by_channel_group.rst b/doc/how_to/process_by_channel_group.rst index d88f4d3120..439e757e59 100644 --- a/doc/how_to/process_by_channel_group.rst +++ b/doc/how_to/process_by_channel_group.rst @@ -114,7 +114,7 @@ to combine the separate channel group recordings back together. combined_preprocessed_recording = aggregate_channels(preprocessed_recordings) Now, when this recording is used in sorting, plotting, or whenever -calling it's :py:func:`~get_traces` method, the data will have been +calling its :py:func:`~get_traces` method, the data will have been preprocessed separately per-channel group (then concatenated back together under the hood).