Merge branch 'main' into numba_similarity

doc/releases/0.101.0.rst

@@ -3,7 +3,7 @@
 SpikeInterface 0.101.0 release notes
 ------------------------------------
 
-Estimated: 19th July 2024
+19th July 2024
 
 Main changes:
 

doc/releases/0.101.1.rst

@@ -0,0 +1,144 @@
+.. _release0.101.1:
+
+SpikeInterface 0.101.1 release notes
+------------------------------------
+
+16th September 2024
+
+Main changes:
+
+* Enabled support for consolidated Zarr-backend for `SortingAnalyzer`, including cloud support (#3314, #3318, #3349, #3351)
+* Improved support for Kilosort4 **ONLY VERSIONS >= 4.0.16** (#3339, #3276)
+* Skip recomputation of quality and template metrics if already computed (#3292)
+* Improved `estimate_sparsity` with new `amplitude` method and deprecated `from_ptp` option (#3369)
+* Dropped support for Python<3.9 (#3267)
+
+core:
+
+* Update the method of creating an empty file with right size when saving binary files (#3408)
+* Refactor pandas save load and convert dtypes (#3412)
+* Check run info completed only if it exists (back-compatibility) (#3407)
+* Fix argument spelling in check for binary compatibility (#3409)
+* Fix proposal for channel location when probegroup (#3392)
+* Fix time handling test memory (#3379)
+* Add `BaseRecording.reset_times()` function (#3363, #3380, #3391)
+* Extend `estimate_sparsity` methods and update `from_ptp`` (#3369)
+* Add `load_sorting_analyzer_or_waveforms` function (#3352)
+* Fix zarr folder suffix handling (#3349)
+* Analyzer extension exit status (#3347)
+* Lazy loading of zarr timestamps (#3318)
+* Enable cloud-loading for analyzer Zarr (#3314, #3351, #3378)
+* Refactor `set_property` in base (#3287)
+* Job kwargs fix (#3259)
+* Add `is_filtered` to annotations in `binary.json` (#3245)
+* Add check for None in `NoiseGeneratorRecordingSegment`` get_traces() (#3230)
+
+extractors:
+
+* Load phy channel_group as group (#3368)
+* "quality" property to be read as string instead of object in `BasePhyKilosortSortingExtractor` (#3365)
+
+preprocessing:
+
+* Auto-cast recording to float prior to interpolation (#3415)
+* Update doc handle drift  + better preset (#3232)
+* Add causal filtering to filter.py (#3172)
+
+sorters:
+
+* Updates to kilosort 4: version >= 4.0.16, `bad_channels`, `clear_cache`, `use_binary_file` (#3339)
+* Download apptainer images without docker client (#3335)
+* Expose save preprocessing in ks4 (#3276)
+* Fix KS2/2.5/3 skip_kilosort_preprocessing (#3265)
+* HS: Added lowpass parameter, fixed verbose option (#3262)
+* Now exclusive support for HS v0.4 (Lightning) (#3210)
+
+postprocessing:
+
+* Add extra protection for template metrics (#3364)
+* Protect median against nans in get_prototype_spike (#3270)
+* Fix docstring and error for spike_amplitudes (#3269)
+
+qualitymetrics:
+
+* Do not delete quality and template metrics on recompute (#3292)
+* Refactor quality metrics tests to use fixture (#3249)
+
+
+curation:
+
+* Clean-up identity merges in `get_potential_auto_merges` (#3346)
+* Fix sortingview curation no merge case  (#3309)
+* Start apply_curation() (#3208)
+
+widgets:
+
+* Fix metrics widgets for convert_dtypes (#3417)
+* Fix plot motion for multi-segment (#3414)
+* Sortingview: only round float properties (#3406)
+* Fix widgets tests and add test on unit_table_properties (#3354)
+* Allow quality and template metrics in sortingview's unit table (#3299)
+* Add subwidget parameters for UnitSummaryWidget (#3242)
+* Fix `ipympl`/`widget` backend check (#3238)
+
+generators:
+
+* Handle case where channel count changes from probeA to probeB (#3237)
+
+sortingcomponents:
+
+* Update doc handle drift  + better preset (#3232)
+* Make `InterpolateMotionRecording`` not JSON-serializable (#3341)
+
+documentation:
+
+* Clarify meaning of `delta_time` in  `compare_sorter_to_ground_truth` (#3360)
+* Added sphinxcontrib-jquery (#3307)
+* Adding return type annotations (#3304)
+* More docstring updates for multiple modules (#3298)
+* Fix sampling frequency repr (#3294)
+* Proposal for adding Examples to docstrings (#3279)
+* More numpydoc fixes (#3275)
+* Fix docstring and error for spike_amplitudes (#3269)
+* Fix postprocessing docs (#3268)
+* Fix name of principal_components ext in qm docs (take 2) (#3261)
+* Update doc handle drift  + better preset (#3232)
+* Add `int` type to `num_samples` on `InjectTemplatesRecording`. (#3229)
+
+continuous integration:
+
+* Fix streaming extractor condition in the CI (#3362)
+
+packaging:
+
+* Minor typing fixes (#3374)
+* Drop python 3.8 in pyproject.toml (#3267)
+
+testing:
+
+* Fix time handling test memory (#3379)
+* Fix streaming extractor condition in the CI (#3362)
+* Test IBL skip when the setting up the one client fails (#3289)
+* Refactor `set_property` in base (#3287)
+* Refactor quality metrics tests to use fixture (#3249)
+* Add kilosort4 wrapper tests (#3085)
+* Test IBL skip when the setting up the one client fails (#3289)
+* Add kilosort4 wrapper tests (#3085)
+
+Contributors:
+
+* @Djoels
+* @JoeZiminski
+* @JuanPimientoCaicedo
+* @alejoe91
+* @chrishalcrow
+* @cwindolf
+* @florian6973
+* @h-mayorquin
+* @jiumao2
+* @jonahpearl
+* @mhhennig
+* @rkim48
+* @samuelgarcia
+* @tabedzki
+* @zm711

doc/scripts/auto-release-notes.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 if [ $# -eq 0 ]; then
-    echo "Usage: $0 START_DATE END_DATE [LABEL] [BRANCH1,BRANCH2] [LIMIT]"
+    echo "Usage: $0 START_DATE (format: YEAR-MM-DD) END_DATE [LABEL] [BRANCH1,BRANCH2] [LIMIT]"
     exit 1
 fi
 

doc/whatisnew.rst

@@ -8,6 +8,7 @@ Release notes
 .. toctree::
   :maxdepth: 1
 
+  releases/0.101.1.rst
   releases/0.101.0.rst
   releases/0.100.8.rst
   releases/0.100.7.rst
@@ -43,6 +44,16 @@ Release notes
   releases/0.9.1.rst
 
 
+Version 0.101.1
+===============
+
+* Enabled support for consolidated Zarr-backend for `SortingAnalyzer`, including cloud support (#3314, #3318, #3349, #3351)
+* Improved support for Kilosort4 **ONLY VERSIONS >= 4.0.16** (#3339, #3276)
+* Skip recomputation of quality and template metrics if already computed (#3292)
+* Improved `estimate_sparsity` with new `amplitude` method and deprecated `from_ptp` option (#3369)
+* Dropped support for Python<3.9 (#3267)
+
+
 Version 0.101.0
 ===============
 

pyproject.toml

@@ -94,7 +94,6 @@ preprocessing = [
 full = [
     "h5py",
     "pandas",
-    "xarray",
     "scipy",
     "scikit-learn",
     "networkx",
@@ -125,16 +124,16 @@ test_core = [
 
     # for github test : probeinterface and neo from master
     # for release we need pypi, so this need to be commented
-    "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",
-    "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",
+    # "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",
+    # "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",
 ]
 
 test_extractors = [
     # Functions to download data in neo test suite
     "pooch>=1.8.2",
     "datalad>=1.0.2",
-    "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",
-    "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",
+    # "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",
+    # "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",
 ]
 
 test_preprocessing = [
@@ -148,7 +147,6 @@ test = [
     "pytest-dependency",
     "pytest-cov",
 
-    "xarray",
     "huggingface_hub",
 
     # preprocessing
@@ -175,8 +173,8 @@ test = [
 
     # for github test : probeinterface and neo from master
     # for release we need pypi, so this need to be commented
-    "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",
-    "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",
+    # "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",
+    # "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",
 ]
 
 docs = [
@@ -193,15 +191,14 @@ docs = [
     "pandas", # in the modules gallery comparison tutorial
     "hdbscan>=0.8.33",   # For sorters spykingcircus2 + tridesclous
     "numba", # For many postprocessing functions
-    "xarray", # For use of SortingAnalyzer zarr format
     "networkx",
     # Download data
     "pooch>=1.8.2",
     "datalad>=1.0.2",
 
     # for release we need pypi, so this needs to be commented
-    "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",  # We always build from the latest version
-    "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",  # We always build from the latest version
+    # "probeinterface @ git+https://github.com/SpikeInterface/probeinterface.git",  # We always build from the latest version
+    # "neo @ git+https://github.com/NeuralEnsemble/python-neo.git",  # We always build from the latest version
 
 ]
 

src/spikeinterface/__init__.py

@@ -30,5 +30,5 @@
 # This flag must be set to False for release
 # This avoids using versioning that contains ".dev0" (and this is a better choice)
 # This is mainly useful when using run_sorter in a container and spikeinterface install
-DEV_MODE = True
-# DEV_MODE = False
+# DEV_MODE = True
+DEV_MODE = False

src/spikeinterface/core/baserecording.py

@@ -768,7 +768,13 @@ def get_binary_description(self):
             raise NotImplementedError
 
     def binary_compatible_with(
-        self, dtype=None, time_axis=None, file_paths_lenght=None, file_offset=None, file_suffix=None
+        self,
+        dtype=None,
+        time_axis=None,
+        file_paths_length=None,
+        file_offset=None,
+        file_suffix=None,
+        file_paths_lenght=None,
     ):
         """
         Check is the recording is binary compatible with some constrain on
@@ -779,6 +785,15 @@ def binary_compatible_with(
           * file_offset
           * file_suffix
         """
+
+        # spelling typo need to fix
+        if file_paths_lenght is not None:
+            warnings.warn(
+                "`file_paths_lenght` is deprecated and will be removed in 0.103.0 please use `file_paths_length`"
+            )
+            if file_paths_length is None:
+                file_paths_length = file_paths_lenght
+
         if not self.is_binary_compatible():
             return False
 
@@ -790,7 +805,7 @@ def binary_compatible_with(
         if time_axis is not None and time_axis != d["time_axis"]:
             return False
 
-        if file_paths_lenght is not None and file_paths_lenght != len(d["file_paths"]):
+        if file_paths_length is not None and file_paths_length != len(d["file_paths"]):
             return False
 
         if file_offset is not None and file_offset != d["file_offset"]:

src/spikeinterface/core/recording_tools.py

@@ -131,9 +131,12 @@ def write_binary_recording(
         data_size_bytes = dtype_size_bytes * num_frames * num_channels
         file_size_bytes = data_size_bytes + byte_offset
 
-        file = open(file_path, "wb+")
-        file.truncate(file_size_bytes)
-        file.close()
+        # Create an empty file with file_size_bytes
+        with open(file_path, "wb+") as file:
+            # The previous implementation `file.truncate(file_size_bytes)` was slow on Windows (#3408)
+            file.seek(file_size_bytes - 1)
+            file.write(b"\0")
+
         assert Path(file_path).is_file()
 
     # use executor (loop or workers)