Question: How to insert new data into a recording object. Purpose: "remove_artifacts" by inserting our own cleaned data

[borrepp]

Hello,

Contex:
We have a recording that includes Intracortical Microstimulation (ICMS) multiple times during the session. The data is in NWB format originally.
A collaborator has implemented an artifact removal using ERAASR (https://github.com/djoshea/eraasr) and provided a Matlab file with:

1. A matrix of [nSnippet X nChans X nStim] which contains snippets (length = nSnippet) of the high-pass filtered & centered cleaned traces for all the channels (nChans) from all the ICMS epochs (nStim)
2. An array of size [nStim] which contains the start_sample_index of each snippet.

Now, I'm working out how to insert these clean traces into the "filtered & centered" recording object that I got from the same NWB file. Afterward, I will use "neuroconv" to export this recording into a new NWB file (which I have already tested but without the snippet replacement).

Question:
Is there a function to insert these snippets into the recording object?

I hope my question makes sense, if not I can explain anything in further detail.

Thanks for any thoughts and comments on how to approach this.

Best
Pepe

[h-mayorquin]

Hi, man.

First the simplest answer:

If your data fits into memory the easier thing is to use the NumpyRecording:

https://github.com/h-mayorquin/spikeinterface/blob/0bf2b08248b836c6323524c1f54cf3690cd6c5f8/src/spikeinterface/core/numpyextractors.py#L23-L53

This will allow you to just put your data as a numpy array and run any spikeinterface routine your might want. This gives you a lot of flexibility but is limited by memory.

Now do I get your right that you want a recording object that has the following behavior:

get_traces(start_frame=a, end_frame=b)
1) normal recording object if the object is outside the snippet
2) the snippet data if the object on a snipet

The most similar thing that I can think off is InjectTemplatesRecording but it does not seem to fit perfectly.

@alejoe91 , what are your thoughts?

Also, I did this for you and some other users:
#3587

[borrepp]

Hello @h-mayorquin

Thanks for the tips.

It is a dataset with 32 channels at 30K, so it might not be feasible to load it into memory. But I can try to process individual channels, save them as a binary folder, and later combine them within spikeinterface. Is there a function to concatenate channels?

I saw your request to get non-electrical NWB timeseries¡ Thanks¡¡ that will be useful for filtering and processing our analog signals from behavioral channels. THANKS¡¡

Best
Pepe

[h-mayorquin]

from spikeinterface.core import aggregate_channels

You can ctrl + f this here to get a longer explanation.
https://github.com/SpikeInterface/SpikeInterface-Training-Edinburgh-May24/blob/main/hands_on/cookbooks/14_00_cookbook.ipynb

Be mindful that there might be a better way than the per channel processing, I would wait a bit for @alejoe91 or someone else to comment.

But at least that's a way.

[borrepp]

Cool,

Thanks,

Yes, I will wait. I'm still finishing our pipeline to export the curation from phy.
Reinserting the cleaned snippets will be the next step, so there is no rush.

Best
Pepe