Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for kilosort>=4.0.12 #3055

Merged
merged 1 commit into from
Jun 21, 2024
Merged

Conversation

alejoe91
Copy link
Member

Plus some safe casting in the InjectTemplateRecording

Fixes #3050 #3020 #3007

@alejoe91 alejoe91 added the sorters Related to sorters module label Jun 20, 2024
@samuelgarcia
Copy link
Member

cool.

Copy link
Collaborator

@h-mayorquin h-mayorquin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@@ -153,6 +160,11 @@ def _run_from_folder(cls, sorter_output_folder, params, verbose):
import torch
import numpy as np

if verbose:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And so it begins : )

@@ -194,11 +206,17 @@ def _run_from_folder(cls, sorter_output_folder, params, verbose):
data_dir = ""
results_dir = sorter_output_folder
filename, data_dir, results_dir, probe = set_files(settings, filename, probe, probe_name, data_dir, results_dir)
ops = initialize_ops(settings, probe, recording.get_dtype(), do_CAR, invert_sign, device)
if version.parse(cls.get_sorter_version()) >= version.parse("4.0.12"):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh my god.

Copy link
Collaborator

@chrishalcrow chrishalcrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable and it's working on our data.

Just curious, the verbose output looks worse than the previous version. Is there some reason for that? Doesn't look like much has changed on our end?

now:
Screenshot 2024-06-20 at 15 53 04

before:
Screenshot 2024-06-20 at 15 58 01

@JoeZiminski
Copy link
Collaborator

JoeZiminski commented Jun 21, 2024

Hey @alejoe91, just to check my understanding is this also related to the uint bug from #2839? This is casting to int deep in the get_traces machinery?

@alejoe91
Copy link
Member Author

Hey @alejoe91, just to check my understanding is this also related to the uint bug from #2839? This is casting to int deep in the get_traces machinery?

It's currently doing it only for the generated recording, but we could cast in the main get_traces. I think there's no downside

@alejoe91
Copy link
Member Author

Looks reasonable and it's working on our data.

Just curious, the verbose output looks worse than the previous version. Is there some reason for that? Doesn't look like much has changed on our end?

now: Screenshot 2024-06-20 at 15 53 04

before: Screenshot 2024-06-20 at 15 58 01

@chrishalcrow that's on the Kilosort side. Without the logger now only progress bars are printed...

@alejoe91
Copy link
Member Author

Hey @alejoe91, just to check my understanding is this also related to the uint bug from #2839? This is casting to int deep in the get_traces machinery?

It's currently doing it only for the generated recording, but we could cast in the main get_traces. I think there's no downside

I'll make a separate PR for this

@alejoe91 alejoe91 merged commit 5e32a9f into SpikeInterface:main Jun 21, 2024
11 checks passed
Copy link
Collaborator

@JoeZiminski JoeZiminski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good! An extra note to the comments, save_sorting has a new parameter save_preprocessed_copy in v4.0.12. I don't know if we need to / want to explicitly support this at this stage.

I had a look through how the signature of all imported KS4 functions change across versions and they have been generally stable (except for changes addressed already in the SpikeInterface code). However, further to my below comment I think I would advocate using explicitly keyword arguments for all function calls. Although unlikely (but definitely not impossible over the timescale of years), some nasty bugs could be introduced if on the KS4 end arguments are: a) changed in order b) one removed and one added at the same time. We are lucky if they cause a crash but they could feasibly cause non-breaking changes to results 😱 .

@@ -194,11 +206,17 @@ def _run_from_folder(cls, sorter_output_folder, params, verbose):
data_dir = ""
results_dir = sorter_output_folder
filename, data_dir, results_dir, probe = set_files(settings, filename, probe, probe_name, data_dir, results_dir)
ops = initialize_ops(settings, probe, recording.get_dtype(), do_CAR, invert_sign, device)
if version.parse(cls.get_sorter_version()) >= version.parse("4.0.12"):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this change was made in 4.0.11, could the condition be set to this version? kilosort code at 4.0.11

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, the change was made in 4.0.11 to get_run_parameters but in 4.012 to initialize_ops.

@@ -194,11 +206,17 @@ def _run_from_folder(cls, sorter_output_folder, params, verbose):
data_dir = ""
results_dir = sorter_output_folder
filename, data_dir, results_dir, probe = set_files(settings, filename, probe, probe_name, data_dir, results_dir)
ops = initialize_ops(settings, probe, recording.get_dtype(), do_CAR, invert_sign, device)
if version.parse(cls.get_sorter_version()) >= version.parse("4.0.12"):
ops = initialize_ops(settings, probe, recording.get_dtype(), do_CAR, invert_sign, device, False)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could save_preprocessed_copy=False be used for readability?

In general I wonder if it would be better to call the kilosort functions with the kwargs explicitly e.g. initialize_ops(settings=settings, ...). This is annoyingly verbose but will be robust against changes to these function signatures (e.g. if they remove one argument and add another at the same time).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you make a separate PR for this?

Copy link
Collaborator

@JoeZiminski JoeZiminski Jun 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure will do!

This was referenced Jun 21, 2024
alejoe91 added a commit to alejoe91/spikeinterface that referenced this pull request Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sorters Related to sorters module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

KS4: initialize_ops() missing 1 required positional argument: 'save_preprocessed_copy'
5 participants