Pecking Test Computer

We are transitioning to running all pecking test boxes in 125 on one computer (Dell Optiplex 5080 Small form factor). Most things will be similar to the old setup. We will be using the Behringer UMC404HD audio interface for audio in/out.

TODOs

1. LOGAN Clean up Pecking_test google drive

• Move Projects, experiment_{host} up a level
• Renaming Projects to something sane? something_Archive? BehaviorArchive? Something that would tell us this is where data we're done collecting in 125 should go.
• Archive behavioral data from chubby and pumpkin and make sure we aren't losing anything
• Folders to sync: behavior output data, stimuli, config yaml files?
• Renaming synced folders to have "_Synced"
• In person, modify Insync (might require reinstalling) to sync this folder to /data/drive_synced or something

2. KEVIN Transfer vocalizatoin data from windows computer to google drive in some sane folder / organization -> with longer term goal of central place to put call recordings

• move computer up to 2nd floor lks

3. LOGAN Remote mounting zdrive (how to put our 123E network under VPN) > periodic syncing jobs

4. LOGAN Get a new USB pci (low profile) card that is compatible with Ubuntu 20.04 (linux kernel 5.4)

5. KEVIN Return old pci card

6. KEVIN + LOGAN cable cleanup and labeling

7. LOGAN Buy Displayport -> VGA hookup

8. LOGAN Gluing mic holders

9. KEVIN Testing phantom power on mics and recordings on preference tests

File management

Planned local file structure

/data/
  drive/                 <- SYNCED Google Drive folders thru Insync
    Pecking_test/        <- (OLD) outer folder for folders that used to be synced (PythonConfig/experiment_{hostname})
    pecking_test_data/
      plump_synced/
        behavior/          <- base experiment dir
          GreBla0101M/
            101020/
            ...
        configs/
          Box2.yaml
          ...
      stimuli/        
        shaping/
        ladder/
        ...
  pecking_test/              
    behavior/ <- symlink to /data/drive/pecking_test_data/plump_synced/behavior
    configs/  <- symlink to /data/drive/pecking_test_data/plump_synced/configs
    stimuli/  <- symlink to /data/drive/pecking_test_data/stimuli
  vocalization_recordings_synced/  <- copy over files from windows computer, don't sync?
    ???

Planned Google Drive file organization

Archive the current Pecking_test folder to Pecking_test_old

# Somewhere in spa_fetlab Google Drive

pecking_test_data/
  stimuli/                 <- sync to /data/drive/pecking_test_data/stimuli
    ladder/
    shaping/
    categories/
  plump_synced/            <- sync to /data/drive/pecking_test_data/plump_synced
    configs/
    behavior/
  behavior_archive/        <- don't sync this but copy data from plump_synced to here when experiment is finished

Logging in

• hostname: plump
• user: fet
• pw: finchfinchfin
• static ip on 123E network: 192.168.0.110

Connecting remotely

ssh [email protected] -p 65456

(The router's static ip is 169.229.251.200 and maps its port 65456 to port 22 for ssh on plump)

Important files and locations

System Configuration Files

/etc/
  pulse/default.pa              # Prevent UMC404 from becoming computer's main audio
  fstab                         # Auto-mounting of 4TB HD
  udev/
    rules.d/
      arduino.rules             # Identify and symlink Arduinos by serial num
      webcam.rules              # Identify and symlink webcams by serial num
  systemd/
    system/
      notebook.service          # Run the jupyter notebook
/home/
  fet/
    .asoundrc                   # Split ins and outs on UMC404 to separate virtual devices
    .bashrc                     # Set PATH and PYTHONPATH

Devices

Interrogate udev attributes of a device in /dev with udevadm info -q all -a /dev/$DEVICE_NAME

List audio cards with cat /proc/asound/cards, and list all audio devices with aplay -L

/dev/
  ttyArduino_box2               # Symlinks to arduino devices
  ttyArduino_box3
  ttyArduino_box5
  ttyArduino_box6
  video_box2                    # Symlinks to webcam devices
  video_box3
  video_box5
  video_box6
/proc/
  asound/
    cards                       # Lists all soundcards

Data Locations

/data/
  drive/
    pecking_test_data/               # Synced to fet_lab Google Drive (Insync)
  pecking_test/
    stimuli/                    # Stimulus wav files (symlink to /data/drive/pecking_test_data/stimuli/
    plump_synced/
      behavior/                 # Base experiment directory (symlink to /data/drive/pecking_test_data/plump_synced/behavior)
  vocalization_recordings/      # Base folder for all vocalization recordings (not necessarily part of pecking test)
/home/
  fet/
    data                        # Symlink to /data
    data_Box2                   # Symlink to /data/pecking_test/behavior/$SUBJECT/$DATE folder
    data_Box3                   #   based on configuration files in /home/fet/configs
    data_Box5
    data_Box6

Code Locations

/home/
  fet/
    configs/                    # Configuration files per subject
      Box2.yaml
      Box3.yaml
      Box5.yaml
      Box6.yaml
    code/
      pyoperant/                # Main pecking test code
      pecking_analysis/         # Data notebook code
      multiple_mics_test/       # Mic recording software
    scripts/
      pecking-test              # Entrypoint to running pecking test commands (recently rewritten from pecking_test)
      peckd                     # Bash functions for launching "pecking-test run -b BOX --preference" in screen sessions

Running pecking test and operating boxes

pecking-test

This is the normal command line interface for operating the boxes. Most commands require a -b BOXNUMBER option. shell is useful during shaping for easy access to python methods to operate box components, and run is used when running the pecking experiments. edit-config and read-config are helper functions for accessing the config files for one or more boxes.

Usage: pecking-test [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  diagnostics  Run diagnostics for debugging audio/hardware on box
  edit-config  Opens config files in Atom for editing
  read-config  Read config parameters
  run          Run the pecking test on specific box
  shell        Launch an IPython shell with boxes imported
  test-audio   Test box's audio output
  test-mic     Test box's microphone recording
  webcam       Launch webcam for box

Examples

pecking-test run -b 2 --preference
pecking-test test-audio -b 2
pecking-test read-config -b 2
pecking-test edit-config
pecking-test diagnostics
pecking-test test-mic -b 2

peckd

peckd (peck daemon) is a wrapper around the pecking-test command line interface above to manage running the pecking test with preference tests in screen sessions (so that the boxes will continue to run if disconnected, and the status can be checked remotely). Typically you would use peckd status to see if the pecking tests are currently running, peckd start BOX to start a box in a new screen session, and peckd stop BOX to stop it. peckd log will show a running log output for all boxes at the same time.

peckd status will only show the status of sessions started with the peckd tool, and won't see anything that you start with pecking-test directly.

Usage: peckd start|stop|log BOX [BOX [BOX ..]]
or
Usage: peckd status

Manage pecking test operation in screen sessions
BOX may be a list of integer box names (e.g. 2 or 3) or 'all' for 2 3 5 6 (see examples below)

Subcommands:
 start: start pecking test for one or more boxes in screen sessions
 stop: stop all pecking test screens or per box
 log: show log output for one or more boxes
 status: show status of screen sessions and latest log output
 run: alias for start

Examples:
 > peckd start 2 3 5 6
 > peckd log all
 > peckd status
 > peckd stop 2 3
 > peckd stop all

Manually operating boxes (shaping)

To manually operate the boxes, you want to open a IPython shell, import the box or boxes, and then call the specific functions for raising and lowering the feeder. You can do this two ways:

1 Manually launch python and import boxes

cd /home/fet/code/pyoperant
source env/bin/activate
ipython

[1]: from pyoperant.tlab.local_tlab import PANELS
[2]: box2 = PANELS["2"]

2 Use pecking-test utility to launch shell

pecking-test shell

Upgraded from Ubuntu 18.04 to 20.04

USBs

The Optiplex 5080 has 9 usb ports built in (4 USB 3.2 on back, 2 USB2.0 on back, 3 USB3.2 on front, and 1 USB-C on front). However, we require at least 10 USB ports: 4 webcams, 4 arduinos, 1 audio interface, and 1 usb hub with connected keyboard and mouse. In theory, we could connect arduinos to usb hubs as well, but I would keep the webcams and audio interface on their own usbs, since hubs typically split the bandwidth of the port.

I got a startech usb pci expansion card but unfortunately it seems to only be compatible with older versions of linux. So we need to replace it. Until we do, I'll use a usb hub for the keyboard, mouse, and 2 arduinos.

Installed software

• python3.9 sudo apt install python3.9 python3.9-dev python3.9-venv
• git sudo apt install git
• portaudio sudo apt install portaudio19-dev (needed for pecking test pyaudio)
• nodejs and npm sudo apt install nodejs npm (needed only for jupyterlab extensions)
• vlc (installed from .deb file)
• slack
• insync (for syncing to google drive)
• atom
• nettools sudo apt install nettools
• libxcb-xinerama0 (this was needed for multiple mics test gui to work)
• screen, htop, memtest

Setting up webcams

When devices udev gives them a file descriptor in /dev. For webcam devices, they are mapped to /dev/video0, /dev/video1, etc. The order is not consistent, so we want to recognize which webcam corresponds to which box and assign it to a static path /dev/video_box2, etc.

To do this, you can interrogate the device attributes with udevadm /dev/video{idx}. For the webcams, we can see that each device has a serial number. You can figure out which serial number then corresponds to each box, and add a udev rule that identifies the device by serial number (ATTRS{serial}==), and then assigns that device to a symlink (SYMLINK+=video_box{box}).

Here are the udev rules

# /etc/udev/rules.d/webcam.rules
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="2F611FC0", ATTR{index}=="0", ATTR{name}=="HD Webcam C525", SYMLINK+="video_box2"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="62C94890", ATTR{name}=="UVC Camera (046d:081b)", ATTR{index}=="0", SYMLINK+="video_box3"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="E8322CA0", ATTR{name}=="HD Webcam C525", ATTR{index}=="0", SYMLINK+="video_box5"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="0E622C20", ATTR{name}=="HD Webcam C525", ATTR{index}=="0"SYMLINK+="video_box6"

I wrote a script to launch streams of all 4 webcams over http with VLC. Command is pecking-test webcam [-b BOX]. If you don't specify a box it will try to launch all 4.

Setting up audio interface

The UMC404HD interface has 4 playback outputs. In surround sound they correspond to front left, front right, rear left, rear right, and are numbered 1, 2, 3, and 4 respectively. To output to these channels we typically would output 4-channel surround sound, but we want individual control over each output.

I modified ~/.asoundrc to create a virtual device "split", locating the audio interface device with hw surround40:u192K. This seems to identify the audio interface reliably, so a udev rule to find the audio device doesn't seem to be necessary at the moment. Then it creates virtual devices speaker2, speaker3, speaker5 and speaker6 for each channel.

pcm.split {
  type dmix
  ipc_key 2048
  slave {
    pcm "surround40:U192k"
    channels 4
  }
}

pcm.speaker2 {
  type plug
  slave.pcm "split"
  ttable.0.0 1
}

pcm.speaker3 {
  type plug
  slave.pcm "split"
  ttable.0.1 1
}

pcm.speaker5 {
  type plug
  slave.pcm "split"
  ttable.0.2 1
}

pcm.speaker6 {
  type plug
  slave.pcm "split"
  ttable.0.3 1
}

pcm_slave.umc_in {
  pcm "hw:U192k"
  rate 48000
  format S32_LE
  buffer_size 4096
  period_size 1024
  channels 4
}

pcm.mic2 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 0
}

pcm.mic3 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 1
}

pcm.mic5 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 2
}

pcm.mic6 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 3
}

The second half of this is my first attempt to split the microphone channels. When using the multiple mics test recording there is a weird lag... I don't know if that is reflected in the recorded audio data yet.

• We also want to prevent this interface from ever becoming the default audio output of the system (we dont want the birds hearing random error sounds or notifications).

To do this pactl list short sinks

Edit /etc/pulse/default.pa, comment out all lines with switch in them.

#load-module module-switch-on-port-available
#load-module module-switch-on-connect

We also added a Startup Application called "Set default audio". Check it to see what the command does.

Setting up arduinos

Like the webcams, the arduino devices are by default mapped to /dev/ttyACM0, /dev/ttyACM1, etc but the order is not always consistent. We create rules in /etc/udev/rules.d/arduino.rules to detect Arduino's with the known serial numbers to their boxes using symlinks /dev/ttyArduino_Box5, etc.

# /etc/udev/rules.d/arduino.rules
### Box 2:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351909201", SYMLINK+="ttyArduino_box2"

### Box 3:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351403182", SYMLINK+="ttyArduino_box3"

# Box 5:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="9523234373335140A011", SYMLINK+="ttyArduino_box5"

# Box 6:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="952323437333518041A1", SYMLINK+="ttyArduino_box6"

Setting up 4TB storage drive

We ordered the computer to come with a 4TB HDD. I formatted as exFAT. However, the drive must be mounted to read/write from it. To set it up to automatically mount the drive (to /data) when you start the computer, you can make the mountpoint with sudo mkdir /data, give the fet user ownership of it with sudo chown -R fet:fet /data, find the UUID of the drive with sudo blkid, and then edit /etc/fstab with a line that tells fstab to automount the drive.

Relevant line in /etc/fstab

UUID=305ddbf7-485f-4aef-81aa-c196a600a785 /data ext4 auto,nosuid,nodev,x-gvfs-show 0 0

Set up remote access

The computer is connected to our wireless network 123E.

To give the computer a fixed ip address on the network, go to the router admin page at 192.168.0.1, Setup > Network Settings > Add DHCP reservation and map this computer plump -> 192.168.0.100 (this could be anything).

Setting up ssh

sudo apt install ssh && sudo service sshd start

Go to router config at 192.168.0.1, go to advanced > virtual server, and map private port 22 to public port 65456.

Setting up Google Drive Syncing

TODO: incloud

Downloading code

• Cloned theunissenlab/pyoperant to /home/fet/code/pyoperant

• Cloned theunissenlab/pecking_analysis to /home/fet/code/pecking_analysis

• Created separate environments for each with python3.9 -m venv env

• Copied scripts/pecking_test from chubbyninja to /home/fet/scripts/pecking_test

• Modifying scripts/pecking_test to have a debug_audio command

• Add ~/scripts to PATH and ~/code to PYTHONPATH in ~/.bashrc

Setting up jupyter notebook

In the pecking_analysis environment (cd ~/code/pecking_analysis && source env/bin/activate), install dependencies (pip install -r requirements.txt) and then run a notebook with jupyter notebook --ip 0.0.0.0 --port 8888. Or jupyter lab --ip 0.0.0.0 --port 8888.

I set up a systemctl service to run the notebook in the background so you don't need to pop up a terminal window to launch the notebook. Useful commands:

# View status and log output of notebook
systemctl status notebook
journalctl -u notebook

# Command to stop/start/restart notebook
systemctl stop notebook
systemctl start notebook
systemctl restart notebook

The service is defined in sudo vim /etc/systemd/system/notebook.service.

TODO: connect to notebook remotely through ssh? dont want to have a notebook publicly accessible

Troubleshooting

Freezing

The system has frozen quite a few times since we got it. The logs are saved in /home/fet/Desktop/crash_logs. If you have a crash, find the syslog file at /var/log/syslog (or /var/log/syslog.1 for the previous day's) and save those files. They might have some info about the crash.

The only common thing I've seen was a error that had something to do with a slack icon. I uninstalled slack in case this was the issue...