Pecking Test Computer

We are transitioning to running all pecking test boxes in 125 on one computer (Dell Optiplex 5080 Small form factor). Most things will be similar to the old setup. We will be using the Behringer UMC404HD audio interface for audio in/out.

Logging in

• hostname: plump
• user: fet
• pw: finchfinchfin
• static ip on 123E network: 192.168.0.110

Connecting remotely

ssh [email protected] -p 65456

Add this to ~/.ssh/config for easier ssh-ing

Host plump
    HostName 169.229.251.200
    Port 65456
    User fet

The router's static ip is 169.229.251.200 and maps its port 65456 to port 22 of plump for ssh)

With plump in your ssh config, you can run this to access the notebook

ssh -fN -L 8888:localhost:8888 plump

Important programs

pecking-test and peckd

pecking-test and peckd are the main ways to operate the pecking test boxes. Type pecking-test and peckd in the command line for more help. Common commands you may want to use are

pecking-test diagnostics          # Run diagnostic checks on all the boxes
pecking-test webcam               # Launch all VLC for all 4 webcams
pecking-test edit-config          # Open all config files in atom for editing
pecking-test shape -b BOX -r 1.0  # Run the pecking test on box BOX but override the reward probability to be 1.0
pecking-test run -b BOX           # Run the pecking test on box BOX
pecking-test run -b BOX -p        # Run the pecking test on box BOX with preference tests on

peckd provides a simplified interface to pecking-test run that launches pecking-test instances in background "screen" sessions. The common commands you may want to use are

peckd start BOX                   # Start the pecking test on box BOX
peckd start BOX --preference      # Start the pecking test on box BOX with preference tests on
peckd status                      # Show a list of the running screen sessions and their recent logs
peckd log                         # Show a live running log on all running screen sessions
peckd stop BOX                    # Stop the pecking test on box BOX by sending a kill command to the screen session
peckd stop all                    # Stop the pecking test on all boxes with one command

multiple_mics_test

multiple_mics_test is our own program for vocalization recording to replace that functionality of sound analysis pro that we used to use on the old Windows computer.

To start:

There is a desktop shortcut called Multiple Mics Test, but it can also be run by opening a terminal, cd code/multipe_mics_test, source env/bin/activate, and python code/main.py.

To record a subject:

First you should get the subject set up in a box, connect a microphone and make sure it is connected to one of the four input ports (1-4) on the front of the Behringer UMC404HD audio interface. These four inputs are defined as virtual devices on the computer named umc_ch1, umc_ch2, umc_ch3, and umc_ch4.

In the multiple mics test window, select your input channel under Device (or umc_all to use all channels simultaneously), set your save location to be a logical place under /data/vocalization_recordings_synced, and edit the channel name to be the subject name. Adjust the gain and threshold as needed. Finally, switch from "Monitor only" mode to "Triggered recording" or "Continuous recording" to start saving files.

In order to record from multiple subjects at once with separate triggers and save locations, open a new instance of multiple_mics_test.

RavenLite2

Raven Lite 2 is program for viewing audio files built by Cornell Lab of Ornithology. It is nice for opening many spectrograms at once, and clipping sections of audio you have recorded with multiple_mics_test.

File management

Pecking test data is synced to the spa_fetlab google drive account. Experiment data is written to the local folder /data/pecking_test/behavior/, which is automatically uploaded to the drive folder pecking_test_data > plump_synced > behavior by the program Insync.

At the end of an experiment, data should be copied on Google Drive from pecking_test_data > plump_synced > behavior to pecking_test_data > behavior_archive. This will remove that data from the computer's hard drive and will no longer be synced.

on Google Drive (spa_fetlab account)

pecking_test_data/
  stimuli/                 <- sync to /data/drive/pecking_test_data/stimuli
    ladder/
    shaping/
    categories/
  plump_synced/            <- sync to /data/drive/pecking_test_data/plump_synced
    configs/
    behavior/
  behavior_archive/        <- don't sync this but copy data from plump_synced to here when experiment is finished

Local data files

The local data files include data files needed for the pecking test in /data/pecking_test, and general vocalization recordings stored in /data/vocalization_recordings_synced. As the name implies, the vocalization recordings directory is synced directly to the spa_fetlab google drive.

The * denote a symlink

/data/
  drive/ 
    pecking_test_data/
      stimuli/                     # (3) Synced to drive (pecking_test_data/stimuli)
      plump_synced/
        behavior/                  # (1) Synced to drive (pecking_test_data/plump_synced/behavior)
        configs/                   # (2) Synced to drive (pecking_test_data/plump_synced/configs)
  vocalization_recordings_synced/  # Synced to SPA_FETLAB:Vocalization Recording Database/Synced Folders/Plump Recordings
  pecking_test/
    *behavior                      # Link to (1)
    *configs                       # Link to (2)
    *stimuli                       # Link to (3)
/home/
  fet/
    *data_Box2                      # These are links to /data/pecking_test/behavior/SUBJECT_NAME/TODAYS_DATE
    *data_Box3
    *data_Box5
    *data_Box6

Local configuration files

These are all the files we have intentionally added or modified to make our system run smoothly. More in the configuration section below.

/etc/
  pulse/default.pa             # Edited 1 line to prevent Behringer audio interface from becoming computer's main audio
  fstab                        # Added 1 line so that the 4TB HD is automatically mounted to /data
  udev/
    rules.d/
      arduino.rules            # Tells udev to map Arduinos by serial number to /dev/ttyArduino_box2, etc
      webcam.rules             # Tells udev to map webcams by serial number to /dev/video_box2, etc
    systemd/
      system/
        notebook.service       # Defines a background service to run the jupyter notebook. Use 'service notebook start|status`
/home/
  fet/
    .asoundrc                  # Defines virtual audio devices to split the audio interface's input and output channels up 
    .bashrc                    # Sets the default PATH variable to include ~/scripts

Local code files

The commands pecking-test and peckd are the main controls for the pecking boxes.

/home/
  fet/
    code/
      pyoperant/               # Main pecking test code
                               #   (https://github.com/theunissenlab/pyoperant)
      pecking_analysis/        # Used on this computer for jupyter notebook data visualization 
                               #   (https://github.com/theunissenlab/pecking_analysis)
      multiple_mics_test/      # Recording simultaneously on multiple microphones
                               #   (https://github.com/kevinyu/multiple_mics_test)
    scripts/
      pecking-test             # Main script for launching pecking test
      peckd                    # Wrapper script around pecking-test for lauching pecking test in background sessions

Devices

The computer communicates with several devices to run the pecking test. These are the webcams, the arduinos, and the audio interface (which connects to all the speakers and microphones). The program "udev" on ubuntu is responsibly for detecting these devices and reliably mapping them to fixed names, often in /dev.

Interrogate udev attributes of a device in /dev with udevadm info -q all -a /dev/$DEVICE_NAME

List audio cards with cat /proc/asound/cards, and list all audio devices with aplay -L

/dev/
  ttyArduino_box2               # Symlinks to arduino devices
  ttyArduino_box3
  ttyArduino_box5
  ttyArduino_box6
  video_box2                    # Symlinks to webcam devices
  video_box3
  video_box5
  video_box6
/proc/
  asound/
    cards                       # Lists all soundcards

Running pecking test and operating boxes

Manually operating boxes (shaping)

To manually operate the boxes, you want to open a IPython shell, import the box or boxes, and then call the specific functions for raising and lowering the feeder. You can do this two ways:

1 Manually launch python and import boxes

cd /home/fet/code/pyoperant
source env/bin/activate
ipython

[1]: from pyoperant.tlab.local_tlab import PANELS
[2]: box2 = PANELS["2"]()

2 Use pecking-test utility to launch shell

pecking-test shell -b 2

Set up notes

We bought the computer from Dell with Ubuntu 18.04 pre-installed. We upgraded it to 20.04 and installed an extra usb expansion card for 4 extra usb ports. This gives us 10 USB ports on back and 3 in front (not counting the 1 USB-C port on front).

Installed software

• python3.9 sudo apt install python3.9 python3.9-dev python3.9-venv
• git sudo apt install git
• portaudio sudo apt install portaudio19-dev (needed for pecking test pyaudio)
• nodejs and npm sudo apt install nodejs npm (needed only for jupyterlab extensions)
• vlc (installed from .deb file)
• slack
• insync (for syncing to google drive)
• atom
• nettools sudo apt install nettools
• libxcb-xinerama0 (this was needed for multiple mics test gui to work)
• screen, htop, memtest
• RavenLite2
• Audacity

Setting up webcams

When devices udev gives them a file descriptor in /dev. For webcam devices, they are mapped to /dev/video0, /dev/video1, etc. The order is not consistent, so we want to recognize which webcam corresponds to which box and assign it to a static path /dev/video_box2, etc.

To do this, you can interrogate the device attributes with udevadm /dev/video{idx}. For the webcams, we can see that each device has a serial number. You can figure out which serial number then corresponds to each box, and add a udev rule that identifies the device by serial number (ATTRS{serial}==), and then assigns that device to a symlink (SYMLINK+=video_box{box}).

Here are the udev rules

# /etc/udev/rules.d/webcam.rules
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="2F611FC0", ATTR{index}=="0", ATTR{name}=="HD Webcam C525", SYMLINK+="video_box2"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="62C94890", ATTR{name}=="UVC Camera (046d:081b)", ATTR{index}=="0", SYMLINK+="video_box3"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="E8322CA0", ATTR{name}=="HD Webcam C525", ATTR{index}=="0", SYMLINK+="video_box5"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="0E622C20", ATTR{name}=="HD Webcam C525", ATTR{index}=="0"SYMLINK+="video_box6"

I wrote a script to launch streams of all 4 webcams over http with VLC. Command is pecking-test webcam [-b BOX]. If you don't specify a box it will try to launch all 4.

Setting up audio interface

The UMC404HD interface has 4 playback outputs. In surround sound they correspond to front left, front right, rear left, rear right, and are numbered 1, 2, 3, and 4 respectively. To output to these channels we typically would output 4-channel surround sound, but we want individual control over each output.

I modified ~/.asoundrc to create a virtual device "split", locating the audio interface device with hw surround40:u192K. This seems to identify the audio interface reliably, so a udev rule to find the audio device doesn't seem to be necessary at the moment. Then it creates virtual devices speaker2, speaker3, speaker5 and speaker6 for each channel.

pcm.split {
  type dmix
  ipc_key 2048
  slave {
    pcm "surround40:U192k"
    channels 4
  }
}

pcm.speaker2 {
  type plug
  slave.pcm "split"
  ttable.0.0 1
}

pcm.speaker3 {
  type plug
  slave.pcm "split"
  ttable.0.1 1
}

pcm.speaker5 {
  type plug
  slave.pcm "split"
  ttable.0.2 1
}

pcm.speaker6 {
  type plug
  slave.pcm "split"
  ttable.0.3 1
}

pcm_slave.umc_in {
  pcm "hw:U192k"
  rate 48000
  format S32_LE
  buffer_size 4096
  period_size 1024
  channels 4
}

pcm.mic2 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 0
}

pcm.mic3 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 1
}

pcm.mic5 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 2
}

pcm.mic6 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 3
}

The second half of this is my first attempt to split the microphone channels. When using the multiple mics test recording there is a weird lag... I don't know if that is reflected in the recorded audio data yet.

• We also want to prevent this interface from ever becoming the default audio output of the system (we dont want the birds hearing random error sounds or notifications).

To do this pactl list short sinks

Edit /etc/pulse/default.pa, comment out all lines with switch in them.

#load-module module-switch-on-port-available
#load-module module-switch-on-connect

We also added a Startup Application called "Set default audio". Check it to see what the command does.

Setting up arduinos

Like the webcams, the arduino devices are by default mapped to /dev/ttyACM0, /dev/ttyACM1, etc but the order is not always consistent. We create rules in /etc/udev/rules.d/arduino.rules to detect Arduino's with the known serial numbers to their boxes using symlinks /dev/ttyArduino_Box5, etc.

# /etc/udev/rules.d/arduino.rules
### Box 2:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351909201", SYMLINK+="ttyArduino_box2"

### Box 3:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351403182", SYMLINK+="ttyArduino_box3"

# Box 5:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="9523234373335140A011", SYMLINK+="ttyArduino_box5"

# Box 6:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="952323437333518041A1", SYMLINK+="ttyArduino_box6"

Setting up 4TB storage drive

We ordered the computer to come with a 4TB HDD. I formatted as exFAT. However, the drive must be mounted to read/write from it. To set it up to automatically mount the drive (to /data) when you start the computer, you can make the mountpoint with sudo mkdir /data, give the fet user ownership of it with sudo chown -R fet:fet /data, find the UUID of the drive with sudo blkid, and then edit /etc/fstab with a line that tells fstab to automount the drive.

Relevant line in /etc/fstab

UUID=305ddbf7-485f-4aef-81aa-c196a600a785 /data ext4 auto,nosuid,nodev,x-gvfs-show 0 0

Set up remote access

The computer is connected to our wireless network 123E.

To give the computer a fixed ip address on the network, go to the router admin page at 192.168.0.1, Setup > Network Settings > Add DHCP reservation and map this computer plump -> 192.168.0.100 (this could be anything).

Setting up ssh

sudo apt install ssh && sudo service sshd start

Go to router config at 192.168.0.1, go to advanced > virtual server, and map private port 22 to public port 65456.

Setting up Google Drive Syncing

Installed Insync (required license) and synced the /data/vocalization_recordings_synced and folders in /data/pecking_test

Downloading code

• Cloned theunissenlab/pyoperant to /home/fet/code/pyoperant

• Cloned theunissenlab/pecking_analysis to /home/fet/code/pecking_analysis

• Created separate environments for each with python3.9 -m venv env

• Copied scripts/pecking_test from chubbyninja to /home/fet/scripts/pecking_test

• Modifying scripts/pecking_test to have a debug_audio command

• Add ~/scripts to PATH and ~/code to PYTHONPATH in ~/.bashrc

Setting up jupyter notebook

In the pecking_analysis environment (cd ~/code/pecking_analysis && source env/bin/activate), install dependencies (pip install -r requirements.txt) and then run a notebook with jupyter notebook --ip 0.0.0.0 --port 8888. Or jupyter lab --ip 0.0.0.0 --port 8888.

I set up a systemctl service to run the notebook in the background so you don't need to pop up a terminal window to launch the notebook. Useful commands:

# View status and log output of notebook
systemctl status notebook
journalctl -u notebook

# Command to stop/start/restart notebook
systemctl stop notebook
systemctl start notebook
systemctl restart notebook

The service is defined in sudo vim /etc/systemd/system/notebook.service.

TODOs

2. KEVIN Transfer vocalizatoin data from windows computer to google drive in some sane folder / organization -> with longer term goal of central place to put call recordings

• move computer up to 2nd floor lks

3. LOGAN Remote mounting zdrive (how to put our 123E network under VPN) > periodic syncing jobs

4. KEVIN Return old pci card

Troubleshooting

Freezing

The system has frozen quite a few times since we got it. The logs are saved in /home/fet/Desktop/crash_logs. If you have a crash, find the syslog file at /var/log/syslog (or /var/log/syslog.1 for the previous day's) and save those files. They might have some info about the crash.

The only common thing I've seen was a error that had something to do with a slack icon. I uninstalled slack in case this was the issue...

After another freeze that occured during a series of PCI errors

I added this to the file /etc/default/grub (found info here https://askubuntu.com/questions/863150/pcie-bus-error-severity-corrected-type-physical-layer-id-00e5receiver-id) as maybe there is an issue with the usb expansion card?

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=nomsi"

No clue if this will help