Fix integer overflow in parallel computing #3426
Conversation
for more information, see https://pre-commit.ci
src/spikeinterface/core/job_tools.py
Outdated
| frame_starts = frame_starts.tolist() | ||
| frame_stops = frame_stops.tolist() | ||
| frame_starts = [i * chunk_size for i in range(n)] | ||
| frame_stops = [(i + 1) * chunk_size for i in range(n)] |
There was a problem hiding this comment.
This works. I tend to lean toward explicit so something like
frame_stops = [frame_start + chunk_size for frame_start in frame_starts]
Just because it is clearer rather than have to multiple things out. But I think this is a good protection against overflow. Your long recordings are really finding all the overflows :)
There was a problem hiding this comment.
Yeah... I am surprised no one has tried to deal with long recordings before. Are there any other long-recording users?
There was a problem hiding this comment.
The most I've heard of recently is closer to the 10-15 hour mark. But haven't really followed up on those. These things work until they don't. So we are trying to track them down, but they are easier to find when they break.
There was a problem hiding this comment.
Hope that long recording is not a bad idea :)
There was a problem hiding this comment.
@jiumao2 we are planning to run week long recordings. You are in good company :)
There was a problem hiding this comment.
Good to hear that! Not alone anymore :)
There was a problem hiding this comment.
Hi @alejoe91 ,
Are you beginning to work with the week-long recordings? I'm not satisfied with the results from Kilosort and have run into a number of issues. Could you share any strategies or tools you're using, such as different sorters, that might be helpful?
Thank you!
There was a problem hiding this comment.
Yeah, this is a good idea.
I also prefer @zm711 writing but this is your PR and that would firmly in style territory : P
pero somos elegantes, carnal. |
|
We all love @zm711 writing! |
frame_starts = np.arange(n) * chunk_sizewill overflow when the recording is ~24 hours long andchunk_size = "1s"Another integer overflow issue on long recordings following #3388