Patch to skip chunks without peaks or if enough peaks have been detected #2011
Conversation
|
Your patch is worst than before, you are loading (and computing) traces for all nodes!! |
|
Ok, so what should be done is simply checking if the first node is a PeakSource, and it it has spikes, then load traces otherwise skip. I'll give it a try |
for more information, see https://pre-commit.ci
|
Here we go, let me know if this works with the node_pipeline spirit! |
|
Yes. We check the first node and we skip the entire grpah if threre is no peak. |
| recording_segment, start_frame, end_frame, None, max_margin, add_zeros=True | ||
| ) | ||
| else: | ||
| traces_chunk, left_margin, right_margin = None, None, None |
There was a problem hiding this comment.
If we do not have traces maybe we can also skip the node loop no ?
yger
changed the title
|
I've extended the PR to cases where we want to stop everything when at least max_peaks have been discovered. This is working, and to allow shuffliung of the chunks, we need to add some seed arguments to the run() function of the ChunkRecordingExecutor. By doing so, chunks can be shuffled, and as soon as max_peaks have been detected, we stop loading traces (or if no peaks are in the chunks) |
|
@samuelgarcia @yger should we keep this for later? |
|
Yes, @samuelgarcia wants to redesign something, so I guess this will be postponed |
|
This has been done in #3356 |
Could such a solution work @samuelgarcia ? This is a patch to try to address #2001. The idea is that if we have a PeakSource node, and no peaks between start_frame and end_frame, then we should clearly reduce IO by not loading the traces