fixed conversion factor and reordered builder

.github/workflows/PR-test.yml

@@ -0,0 +1,33 @@
+name: PR Test
+
+on: push
+
+jobs:
+  run-tests:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest]  #, macos-latest, windows-latest]
+    env:
+      OS: ${{ matrix.os }}
+      PYTHON: '3.8'
+    steps:
+    - name: Cancel Workflow Action
+      uses: styfle/[email protected]
+      with:
+        access_token: ${{ github.token }}
+    - name: Checkout
+      uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+    - name: Set up conda environment
+      uses: conda-incubator/setup-miniconda@v2
+      with:
+        activate-environment: rec_to_nwb
+        environment-file: environment.yml
+    - name: Install rec_to_nwb
+      shell: bash -l {0}
+      run: |
+        pip install -e .

rec_to_nwb/.gitignore → .gitignore

@@ -20,6 +20,8 @@ parts/
 sdist/
 var/
 wheels/
+pip-wheel-metadata/
+share/python-wheels/
 *.egg-info/
 .installed.cfg
 *.egg
@@ -38,6 +40,7 @@ pip-delete-this-directory.txt
 # Unit test / coverage reports
 htmlcov/
 .tox/
+.nox/
 .coverage
 .coverage.*
 .cache
@@ -72,9 +75,20 @@ target/
 # Jupyter Notebook
 .ipynb_checkpoints
 
+# IPython
+profile_default/
+ipython_config.py
+
 # pyenv
 .python-version
 
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
 # celery beat schedule file
 celerybeat-schedule
 
@@ -102,9 +116,18 @@ venv.bak/
 
 # mypy
 .mypy_cache/
-
-# idea pycharm
-../.idea/
-
-#testexample
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+.DS_Store
+*.mat
+*.lock
+*.dirlock
+.vscode
+*.dat
+*.xml
+*.h264
+*.mda
 *.nwb

README.md

@@ -1,28 +1,12 @@
 # rec_to_nwb
+[![PR Test](https://github.com/LorenFrankLab/rec_to_nwb/actions/workflows/PR-test.yml/badge.svg)](https://github.com/LorenFrankLab/rec_to_nwb/actions/workflows/PR-test.yml)
 # About
 rec_to_nwb is a python conda package for converting SpikeGadgets rec files to NWB files.<br>
 It converts experiment data from `/raw` or `/preprocessing` folder to `.nwb` file. It utilizes rec_to_binaries package for preprocessing phase.<br>
 <https://github.com/LorenFrankLab/rec_to_binaries><br>
 
-# Prerequisites
-## For users
-1. Install Spike Gadgets <br>
-   <https://bitbucket.org/mkarlsso/trodes/downloads/>
-2. Add SpikeGadgets to path. <br>
-   If Spike Gadgets is in default location: <br>
-   ```bash
-   export PATH="$HOME/SpikeGadgets/:$PATH"
-   ```
-3. Download miniconda from <br>
-   <https://docs.conda.io/en/latest/miniconda.html> <br>
-4. Install rec_to_nwb package:
-   ```bash
-   conda install -c conda-forge -c novelakrk rec_to_nwb
-   ```
-5. Install Jupyter notebook
-   ```bash
-   pip install jupyter notebook
-   ```
+# Instructions:
+Currently we suggest following the instructions to install https://github.com/LorenFrankLab/franklabnwb, as that includes additional files that are helpful, but you can install this using the instructions below.
 
 ## For developers
 1. Install Spike Gadgets <br>
@@ -32,21 +16,21 @@ It converts experiment data from `/raw` or `/preprocessing` folder to `.nwb` fil
    ```bash
    export PATH="$HOME/SpikeGadgets/:$PATH"
    ```
-3. Download miniconda from<br>
-   <https://docs.conda.io/en/latest/miniconda.html><br>
+3. Install anaconda or miniconda if you haven't already.
+
 4. clone repository
    ```bash
-   git clone https://github.com/NovelaNeuro/rec_to_nwb.git
+   git clone https://github.com/LorenFrankLab/rec_to_nwb.git
 
-   cd rec_to_nwb/rec_to_nwb
+   cd rec_to_nwb
    ```
 5. Create conda environment.
    ```bash
    conda env create -f environment.yml
    ```
-6. jupyter notebook installation
+6. Install rec_to_nwb
    ```bash
-   pip install jupyter notebook
+   pip install -e .
    ```
 7. Documentation can be viewed at <br>
     <https://novelaneuro.github.io/rec_to_nwb-docs/>
@@ -63,27 +47,31 @@ It converts experiment data from `/raw` or `/preprocessing` folder to `.nwb` fil
    jupyter notebook nwb_generation.ipynb
    ```
 4. Metadata.yml description:
-   ```
-    # general information about the experiment 
-    experimenter name: Alison Comrie
+
+   Important note: right now the code assumes that the electrode groups listed below (each of which corresponds to one or more NTrode in the file) are in ascending order by NTrode number. If this is not the case the data could be scrambled.  Thus, the first listed electrode group should correspond to, for example, NTrode 1 (or perhaps NTrodes 1-4) while the second would correspond to NTrode 2 (or 5-8), etc.  
+
+
+   ```yaml
+    # general information about the experiment
+    experimenter_name: Alison Comrie
     lab: Loren Frank
     institution: University of California, San Francisco
-    experiment description: Reinforcement learning
-    session description: Reinforcement leaarning
+    experiment_description: Reinforcement learning
+    session_description: Reinforcement leaarning
     session_id: beans_01
     subject:
       description: Long Evans Rat
       genotype: Wild Type
       sex: Male
       species: Rat
-      subject id: Beans
+      subject_id: Beans
       weight: Unknown
    #Units of analog and behavioral_events
    units:
       analog: 'unspecified'
       behavioral_events: 'unspecified'  
-   #data acq device used in experiment   
-   data acq device:
+   #data_acq_device used in experiment   
+   data_acq_device:
       - name: acq_0
         system: sample_system
         amplifier: sample_amplifier
@@ -129,19 +117,18 @@ It converts experiment data from `/raw` or `/preprocessing` folder to `.nwb` fil
        associated_video_files:
       - name: 20190718_beans_01_s1.1.h264
         camera_id : 0
-    # Times period multiplier is used in pos/mda invalid/valid times, to multiply the period when detecting gaps,
-        to avoid creating invalid times in case of only small deviations. (optional parameter, default 1.5)
+    # Times period multiplier is used in pos/mda invalid/valid times, to multiply the period when detecting gaps, to avoid creating invalid times in case of only small deviations. (optional parameter, default 1.5)
        times_period_multiplier: 1.5      
     # Din/Dout events which filter out files from DIO data in data directory. Each name has to be unique. Stored in behavioral_events section in output nwb file.
-    behavioral_events: 
+    behavioral_events:
       - name: Poke2
         description: Din2
     # Device name. Stored in output nwb file.
-    device: 
+    device:
       name:
         - Trodes
-    # Electrode Groups list used in experiment. Each Id has to be unique, device_type has to refer to existing device_type in probe.yml. Target_x,y,z fields describe the specified location where this group should be. Possible value of units: 'um' or 'mm' 
-    electrode groups:
+    # Electrode Groups list used in experiment. Each Id has to be unique, device_type has to refer to existing device_type in probe.yml. Target_x,y,z fields describe the specified location where this group should be. Possible value of units: 'um' or 'mm'
+    electrode_groups:
       - id: 0
         location: mPFC
         device_type: 128c-4s8mm6cm-20um-40um-sl
@@ -163,8 +150,8 @@ It converts experiment data from `/raw` or `/preprocessing` folder to `.nwb` fil
     # Ntrodes list which refer 1:1 to <SpikeNTrode> elements from xml header existing in rec binary file.
     # ntrode_id has to match to SpikeNTrode id, electrode_group_id refers to electrode group,
     # bad_channels is a list of broken channels in the map, where map corresponds to the electrode channels
-      - ntrode_id: 1 
-        electrode_group_id: 0 
+      - ntrode_id: 1
+        electrode_group_id: 0
         bad_channels: [0,2]
         map:  
           0: 0
@@ -179,9 +166,10 @@ It converts experiment data from `/raw` or `/preprocessing` folder to `.nwb` fil
           1: 5
           2: 6
           3: 7
-    ```
-5. Probe.yml description:
    ```
+
+5. Probe.yml description:
+   ```yaml
     probe_type: tetrode_12.5 # Type of the probe that refers to device_type in electrode_group in metadata.yml
     units: 'um' # possible value for unit is um or mm
     probe_description: 'four wire electrode'
@@ -212,8 +200,8 @@ It converts experiment data from `/raw` or `/preprocessing` folder to `.nwb` fil
 7. Input files `metadata.yml` as well as `probe[1-N].yml` are validated against rec files headers.
 
 8. We provide two class to generate the NWB file. <br>
-* `RawToNWBBuilder` - To generate NWB file from raw data. <br>
-* `NWBFileBuilder` - To generate NWB file from preprocessed data. <br>
+   * `RawToNWBBuilder` - To generate NWB file from raw data. <br>
+   * `NWBFileBuilder` - To generate NWB file from preprocessed data. <br>
 
 ##### Raw data
 Initialize RawToNWBBuilder, which requires `animal_name`, `data_path` and `dates` which exist in your experiment folder. Next build the NWB using `build_nwb()`. <br>
@@ -238,9 +226,9 @@ If you don't want mda or pos invalid/valid times in your nwb, set accordingly fl
       **dates** = `list of strings` names of folders that contain experiment data <br>
 
       **nwb_metadata** = `MetadataManager` object with metadata.yml and probes.yml <br>
-      
+
       **output_path** = `string` path specifying location and name of result file (dafault 'output.nwb') <br>
-      
+
       **video_path** = `string` path specifying location of video files .h264 where those are copied <b4>
 
       **extract_analog** = `boolean` flag specifying if analog data should be extracted from raw (default True) <br>
@@ -254,22 +242,22 @@ If you don't want mda or pos invalid/valid times in your nwb, set accordingly fl
       **extract_mda** = `boolean` flag specifying if mda data should be extracted from raw (default True) <br>
 
       **parallel_instances** = `int` number of threads, optimal value highly depends on hardware (default 4) <br>
-      
+
       **overwrite** = `boolean`  If true, will overwrite existing files. (default True) <br>
-      
+
       **trodes_rec_export_args** = `tuple of strings` path to rec header file which overrides all headers existing in rec binary files e.g `_DEFAULT_TRODES_REC_EXPORT_ARGS = ('-reconfig', str(path) + '/test/processing/res/reconfig_header.xml')` <br>  
 
    build_nwb arguments:
 
      **process_mda_valid_time** = 'boolean' True if the mda valid times should be build and append to nwb.
                 Need the mda data inside the nwb. (default True) <br>
-     
+
      **process_mda_invalid_time** = 'boolean' True if the mda invalid times should be build and append to nwb.
                 Need the mda data inside the nwb. (default True) <br>
-     
+
      **process_pos_valid_time** = 'boolean' True if the pos valid times should be build and append to nwb.
                 Need the pos data inside the nwb. (default True) <br>
-     
+
      **process_pos_invalid_time** = 'boolean' True if the pos invalid times should be build and append to nwb.
                 Need the pos data inside the nwb. (default True) <br>
 
@@ -294,38 +282,38 @@ After that, you can add mda or pos invalid/valid data to your NWB, using 'build_
    NWBFileBuilder arguments
 
      **data_path** = `string` path to directory containing all experiments data <br>
-     
+
      **animal_name** = `string` directory name which represents animal subject of experiment <br>
-     
+
      **date** = `string` date of experiment <br>
-     
+
      **nwb_metadata** = `MetadataManager` object contains metadata about experiment <br>
-    
+
      **process_dio** = `boolean` flag if dio data should be processed <br>
-     
+
      **process_mda** = `boolean` flag if mda data should be processed <br>
-     
+
      **process_analog** = `boolean` flag if analog data should be processed <br>
-     
+
      **video_path** = `string` path specifying location of video files .h264 where those are copied <b4>
-     
+
      **output_file** = `string` path and name specifying where .nwb file gonna be written <br>
 
    build_and_append_to_nwb arguments:
 
      **process_mda_valid_time** = 'boolean' True if the mda valid times should be build and append to nwb.
                 Need the mda data inside the nwb. (default True) <br>
-     
+
      **process_mda_invalid_time** = 'boolean' True if the mda invalid times should be build and append to nwb.
                 Need the mda data inside the nwb. (default True) <br>
-     
+
      **process_pos_valid_time** = 'boolean' True if the pos valid times should be build and append to nwb.
                 Need the pos data inside the nwb. (default True) <br>
-     
+
      **process_pos_invalid_time** = 'boolean' True if the pos invalid times should be build and append to nwb.
                 Need the pos data inside the nwb. (default True) <br>
 
-9. Make sure that the data structure in given directory (in that case `test_data`) looks similar to following example:
+9. Make sure that the file and directory structure (in that case `test_data`) looks similar to following example. Please NOTE that rec_to_nwb will fail if the data is not in this format:
    ```bash
     --test_data
       |
@@ -560,4 +548,3 @@ After that, you can add mda or pos invalid/valid data to your NWB, using 'build_
    |-- README.md
    ```
 When processing completes, a nwb file is created in the output_path directory
-