Question: how to ONLY add "manual_labels" when using "apply_curation"

[borrepp]

Hello,
I'm trying to add some labels ["good", "noise", "mua", "artifact"] to a sorting_analyzer, but when I wanted to do it by creating a dictionary with empty "removed_units" & "merge_units_groups" it throws an error saying that merge_unit_groups must contain at least two units. But I don't want to merge anything. I only want to add labels. Is there a way to do it by "curation_dict", The alternative I found was by setting a property to the "sorting_analyzer.sorting.set_property('quality', ['mua'])". I only have one unit, so it was not complicated, but it might be useful to allow "apply_curation" when there are no merge_units_groups.

A related question after I set the property is: What is the best practice for saving this change?
I did it by saving the sorting analyzer again:
"sorting_analyzer.save_as(format="binary_folder", folder=sorter_analyzer_folder+'_curated')"
Is there a way to avoid rewriting the folder?

Thanks in advance for your help.

Here is the dictionary that I created for curation:

# Sample Dictionary to be applied on the result to have a “clean” result
curation_dict = dict(
    format_version = "1",
    # Define LABELS
    # For label category with exclusive=True : a column is created and values are the unique label.
    # For label category with exclusive=False : one column per possible is created and values are boolean.
    label_definitions = dict(
        quality = dict(label_options = ["good", "noise", "mua", "artifact"], exclusive = True)
        # Keep adding custom labels. Example:
        # putative_type = dict(label_options = ["excitatory", "inhibitory", "pyramidal", "mitral"], exclusive = False)
    ),
    # 
    unit_ids = sorting_analyzer.unit_ids,
    removed_units = [], # List of units to remove. Example: [31, 42]
    merge_unit_groups = [[], []], # List of Lists of units to merge (at least 2 units require). Example: [[3, 6], [10, 14, 20]]
    manual_labels = [
        dict(unit_id = 1, quality = ["mua"]),
        # Keep adding neurons' labels. Example:
        # dict(unit_id = 2, quality = ["noise"], putative_type = ["excitatory", "pyramidal"])
    ],
)

Here is the error when trying "apply_curation" with that dictionary:

--> [343](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:343) analyzer, new_unit_ids = analyzer.merge_units(
    [344](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:344)     curation_dict["merge_unit_groups"],
    [345](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:345)     censor_ms=censor_ms,
    [346](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:346)     merging_mode=merging_mode,
    [347](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:347)     sparsity_overlap=sparsity_overlap,
    [348](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:348)     new_id_strategy=new_id_strategy,
    [349](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:349)     return_new_unit_ids=True,
    [350](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:350)     format="memory",
    [351](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:351)     verbose=verbose,
    [352](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:352)     **job_kwargs,
    [353](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:353) )
    [354](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:354) apply_curation_labels(analyzer.sorting, new_unit_ids, curation_dict)
    [355](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:355) return analyzer

File m:\Monkey_Python\SIenv\Lib\site-packages\spikeinterface\core\sortinganalyzer.py:1101, in SortingAnalyzer.merge_units(self, merge_unit_groups, new_unit_ids, censor_ms, merging_mode, sparsity_overlap, new_id_strategy, return_new_unit_ids, format, folder, verbose, **job_kwargs)
   [1098](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1098) for units in merge_unit_groups:
   [1099](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1099)     # TODO more checks like one units is only in one group
   [1100](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1100)     if len(units) < 2:
-> [1101](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1101)         raise ValueError("Merging requires at least two units to merge")
   [1103](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1103) # TODO : no this function did not exists before
   [1104](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1104) if not isinstance(merge_unit_groups[0], (list, tuple)):
   [1105](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/core/sortinganalyzer.py:1105)     # keep backward compatibility : the previous behavior was only one merge

ValueError: Merging requires at least two units to merge

[zm711]

We are going to do a big round of curation troubleshooting and bug squashing soon. :) Thanks for reporting this. We will look into this and try to see what is happening soon.

[borrepp]

Thanks @zm711

[samuelgarcia]

Hi thnaks.
On my todo list for sure.

[samuelgarcia]

merge_unit_groups = [[], []] is wrong if you do not have merges you should have merge_unit_groups = []
I will add a better message in validate_curation_dict() that is internally called before apply_curation()

[samuelgarcia]

Hi could you try this branch #3601 ?

[borrepp]

Hello @samuelgarcia

I don't get the error related to validate_curation_dict() : ValueError: Merging requires at least two units to merge when I set merge_unit_groups = []

But, now I'm getting a different type of error. Might it be related to the fact that is trying to merge when there is nothing to merge?

I got different errors depending on how I tried to apply curation.

Here is the error when I used the sorting_analyzer:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[6], [line 1](vscode-notebook-cell:?execution_count=6&line=1)
----> [1](vscode-notebook-cell:?execution_count=6&line=1) scur.apply_curation(sorting_or_analyzer=sorting_analyzer, 
      [2](vscode-notebook-cell:?execution_count=6&line=2)     curation_dict = curation_dict,
      [3](vscode-notebook-cell:?execution_count=6&line=3)     censor_ms=0.25,
      [4](vscode-notebook-cell:?execution_count=6&line=4)     new_id_strategy="append",
      [5](vscode-notebook-cell:?execution_count=6&line=5)     merging_mode="soft",
      [6](vscode-notebook-cell:?execution_count=6&line=6)     sparsity_overlap=0.[7](vscode-notebook-cell:?execution_count=6&line=7)5,
      7     verbose=False,
      [8](vscode-notebook-cell:?execution_count=6&line=8)     **job_kwargs
      [9](vscode-notebook-cell:?execution_count=6&line=9)     )

File m:\Monkey_Python\SIenv\Lib\site-packages\spikeinterface\curation\curation_format.py:343, in apply_curation(sorting_or_analyzer, curation_dict, censor_ms, new_id_strategy, merging_mode, sparsity_overlap, verbose, **job_kwargs)
    [341](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:341) analyzer = sorting_or_analyzer
    [342](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:342) analyzer = analyzer.remove_units(curation_dict["removed_units"])
--> [343](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:343) analyzer, new_unit_ids = analyzer.merge_units(
    [344](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:344)     curation_dict["merge_unit_groups"],
    [345](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:345)     censor_ms=censor_ms,
    [346](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:346)     merging_mode=merging_mode,
    [347](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:347)     sparsity_overlap=sparsity_overlap,
    [348](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:348)     new_id_strategy=new_id_strategy,
    [349](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:349)     return_new_unit_ids=True,
    [350](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:350)     format="memory",
    [351](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:351)     verbose=verbose,
    [352](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:352)     **job_kwargs,
    [353](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:353) )
    [354](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:354) apply_curation_labels(analyzer.sorting, new_unit_ids, curation_dict)
    [355](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:355) return analyzer

TypeError: cannot unpack non-iterable SortingAnalyzer object

Here is the error when I used the sorting:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[7], [line 1](vscode-notebook-cell:?execution_count=7&line=1)
----> [1](vscode-notebook-cell:?execution_count=7&line=1) scur.apply_curation(sorting_or_analyzer=sorting_analyzer.sorting, 
      [2](vscode-notebook-cell:?execution_count=7&line=2)     curation_dict = curation_dict,
      [3](vscode-notebook-cell:?execution_count=7&line=3)     censor_ms=0.25,
      [4](vscode-notebook-cell:?execution_count=7&line=4)     new_id_strategy="append",
      [5](vscode-notebook-cell:?execution_count=7&line=5)     merging_mode="soft",
      [6](vscode-notebook-cell:?execution_count=7&line=6)     sparsity_overlap=0.[7](vscode-notebook-cell:?execution_count=7&line=7)5,
      7     verbose=False,
      [8](vscode-notebook-cell:?execution_count=7&line=8)     **job_kwargs
      [9](vscode-notebook-cell:?execution_count=7&line=9)     )

File m:\Monkey_Python\SIenv\Lib\site-packages\spikeinterface\curation\curation_format.py:337, in apply_curation(sorting_or_analyzer, curation_dict, censor_ms, new_id_strategy, merging_mode, sparsity_overlap, verbose, **job_kwargs)
    [329](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:329)     sorting = sorting.remove_units(curation_dict["removed_units"])
    [330](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:330)     sorting, _, new_unit_ids = apply_merges_to_sorting(
    [331](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:331)         sorting,
    [332](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:332)         curation_dict["merge_unit_groups"],
   (...)
    [335](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:335)         new_id_strategy=new_id_strategy,
    [336](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:336)     )
--> [337](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:337)     apply_curation_labels(sorting, new_unit_ids, curation_dict)
    [338](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:338)     return sorting
    [340](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:340) elif isinstance(sorting_or_analyzer, SortingAnalyzer):

File m:\Monkey_Python\SIenv\Lib\site-packages\spikeinterface\curation\curation_format.py:241, in apply_curation_labels(sorting, new_unit_ids, curation_dict)
    [239](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:239) for unit_ind, unit_id in enumerate(sorting.unit_ids):
    [240](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:240)     if unit_id not in new_unit_ids:
--> [241](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:241)         ind = curation_dict["unit_ids"].index(unit_id)
    [242](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:242)         all_values[unit_ind] = values[ind]
    [243](file:///M:/Monkey_Python/SIenv/Lib/site-packages/spikeinterface/curation/curation_format.py:243) sorting.set_property(key, all_values)

AttributeError: 'numpy.ndarray' object has no attribute 'index'

Let me know if I should try another parameter.

Thanks
Pepe

[samuelgarcia]

Maybe we have an other error thanks for this. I will also fix it