Pecking Test Computer

We are transitioning to running all pecking test boxes in 125 on one computer (Dell Optiplex 5080 Small form factor). Most things will be similar to the old setup. We will be using the Behringer UMC404HD audio interface for audio in/out.

TODOs

/data/
  drive/   <-   synced Google Drive folders
    Pecking_test/    <- a folder on gdrive
    pecking_test_data/
      behavior/       <- base exp dir   -> 
        GreBla0101M/
          101020/
          ...
      stimuli/        
        SovsSo_8v8_d2/

  pecking_test -> /data/drive/pecking_test_data

  vocalization_recordings/  <- copy over files from win
    ???

#Currently
data -> /home/fet/drive/Pecking_test/PythonConfig/experiment_chubbyninja/data

Planned

On drive
pecking_test/
  stimuli/
    ladder/
    shaping/
    categories/
  plump_synced/
    configs/
    behavior/
  behavior_archive/   # move data from *_synced here when experiment finished

Archive by renaming the Pecking_test folder to Pecking_test_old

1. LOGAN Clean up Pecking_test google drive

• Move Projects, experiment_{host} up a level
• Renaming Projects to something sane? something_Archive? BehaviorArchive? Something that would tell us this is where data we're done collecting in 125 should go.
• Archive behavioral data from chubby and pumpkin and make sure we aren't losing anything
• Folders to sync: behavior output data, stimuli, config yaml files?
• Renaming synced folders to have "_Synced"
• In person, modify Insync (might require reinstalling) to sync this folder to /data/drive_synced or something

2. KEVIN Transfer vocalizatoin data from windows computer to google drive in some sane folder / organization -> with longer term goal of central place to put call recordings

• move computer up to 2nd floor lks

3. LOGAN Remote mounting zdrive (how to put our 123E network under VPN) > periodic syncing jobs

4. LOGAN Get a new USB pci (low profile) card that is compatible with Ubuntu 20.04 (linux kernel 5.4)

5. KEVIN Return old pci card

6. KEVIN + LOGAN cable cleanup and labeling

7. LOGAN Buy Displayport -> VGA hookup

8. LOGAN Gluing mic holders

9. KEVIN Testing phantom power on mics and recordings on preference tests

For later

10. Running in screen sessions helper?

11. Diagnostics script

Logging in

• hostname: plump
• user: fet
• pw: finchfinchfin
• static ip on 123E network: 192.168.0.110

Connecting remotely

ssh [email protected] -p 65456

(The router's static ip is 169.229.251.200 and maps its port 65456 to port 22 for ssh on plump)

plump important files and locations

System Configuration Files

/etc/
  pulse/default.pa              # Prevent UMC404 from becoming computer's main audio
  fstab                         # Auto-mounting of 4TB HD
  udev/
    rules.d/
      arduino.rules             # Identify and symlink Arduinos by serial num
      webcam.rules              # Identify and symlink webcams by serial num
  systemd/
    system/
      notebook.service          # Run the jupyter notebook
/home/
  fet/
    .asoundrc                   # Split ins and outs on UMC404 to separate virtual devices
    .bashrc                     # Set PATH and PYTHONPATH

Devices

Interrogate udev attributes of a device in /dev with udevadm info -q all -a /dev/$DEVICE_NAME

List audio cards with cat /proc/asound/cards, and list all audio devices with aplay -L

/dev/
  ttyArduino_box2               # Symlinks to arduino devices
  ttyArduino_box3
  ttyArduino_box5
  ttyArduino_box6
  video_box2                    # Symlinks to webcam devices
  video_box3
  video_box5
  video_box6
/proc/
  asound/
    cards                       # Lists all soundcards

Data Locations

/data/
  drive/
    Pecking_test/               # Synced to fet_lab Google Drive (Insync)
  pecking_test/ 
    stimuli/                    # Stimulus wav files
    behavior/                   # Base experiment directory with folders by subject
  vocalization_recordings/      # Base folder for all vocalization recordings (not necessarily part of pecking test)
/home/
  fet/
    data                        # Symlink to /data
    data_Box2                   # Symlink to /data/$SUBJECT/$DATE folder
    data_Box3                   #   based on configuration files in /home/fet/configs
    data_Box5
    data_Box6

Code Locations

/home/
  fet/
    configs/                    # Configuration files per subject
      Box2.yaml
      Box3.yaml
      Box5.yaml
      Box6.yaml
    code/
      pyoperant/                # Main pecking test code
      pecking_analysis/         # Data notebook code
      multiple_mics_test/       # Mic recording software
      webcam_viewer/            # Launch webcams under one process
    scripts/
      pecking_test              # Entrypoint to running pecking_test commands

Running code

Upgraded from Ubuntu 18.04 to 20.04

USBs

The Optiplex 5080 has 9 usb ports built in (4 USB 3.2 on back, 2 USB2.0 on back, 3 USB3.2 on front, and 1 USB-C on front). However, we require at least 10 USB ports: 4 webcams, 4 arduinos, 1 audio interface, and 1 usb hub with connected keyboard and mouse. In theory, we could connect arduinos to usb hubs as well, but I would keep the webcams and audio interface on their own usbs, since hubs typically split the bandwidth of the port.

I got a startech usb pci expansion card but unfortunately it seems to only be compatible with older versions of linux. So we need to replace it. Until we do, I'll use a usb hub for the keyboard, mouse, and 2 arduinos.

Installed software

• python3.9 sudo apt install python3.9 python3.9-dev python3.9-venv
• git sudo apt install git
• portaudio sudo apt install portaudio19-dev (needed for pecking test pyaudio)
• nodejs and npm sudo apt install nodejs npm (needed only for jupyterlab extensions)
• vlc (installed from .deb file)
• slack
• insync (for syncing to google drive)
• atom
• nettools sudo apt install nettools
• libxcb-xinerama0 (this was needed for multiple mics test gui to work)
• screen, htop, memtest

Setting up webcams

When devices udev gives them a file descriptor in /dev. For webcam devices, they are mapped to /dev/video0, /dev/video1, etc. The order is not consistent, so we want to recognize which webcam corresponds to which box and assign it to a static path /dev/video_box2, etc.

To do this, you can interrogate the device attributes with udevadm /dev/video{idx}. For the webcams, we can see that each device has a serial number. You can figure out which serial number then corresponds to each box, and add a udev rule that identifies the device by serial number (ATTRS{serial}==), and then assigns that device to a symlink (SYMLINK+=video_box{box}).

Here are the udev rules

# /etc/udev/rules.d/webcam.rules
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="2F611FC0", ATTR{index}=="0", ATTR{name}=="HD Webcam C525", SYMLINK+="video_box2"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="62C94890", ATTR{name}=="UVC Camera (046d:081b)", ATTR{index}=="0", SYMLINK+="video_box3"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="E8322CA0", ATTR{name}=="HD Webcam C525", ATTR{index}=="0", SYMLINK+="video_box5"
SUBSYSTEM=="video4linux", SUBSYSTEMS=="usb", ATTRS{serial}=="0E622C20", ATTR{name}=="HD Webcam C525", ATTR{index}=="0"SYMLINK+="video_box6"

I wrote a script to launch streams of all 4 webcams over http with VLC. The script in ~/code/webcam_viewer/run.py launches 4 instances of VLC streaming to Ports 8082, 8083, 8085, and 8086. They are accessible on localhost:8082/box2.ogg etc. Opening the file ~/code/webcam_viewer/index.html in the browser shows all 4 in a grid. But there is some latency and some problems with autoplaying and keeping the stream live in the browser, so maybe there's a better way. Worst case we just use 4 instances of the VLC viewer, but it would be nice to get it into a grid view.

Setting up audio interface

The UMC404HD interface has 4 playback outputs. In surround sound they correspond to front left, front right, rear left, rear right, and are numbered 1, 2, 3, and 4 respectively. To output to these channels we typically would output 4-channel surround sound, but we want individual control over each output.

I modified ~/.asoundrc to create a virtual device "split", locating the audio interface device with hw surround40:u192K. This seems to identify the audio interface reliably, so a udev rule to find the audio device doesn't seem to be necessary at the moment. Then it creates virtual devices speaker2, speaker3, speaker5 and speaker6 for each channel.

pcm.split {
  type dmix
  ipc_key 2048
  slave {
    pcm "surround40:U192k"
    channels 4
  }
}

pcm.speaker2 {
  type plug
  slave.pcm "split"
  ttable.0.0 1
}

pcm.speaker3 {
  type plug
  slave.pcm "split"
  ttable.0.1 1
}

pcm.speaker5 {
  type plug
  slave.pcm "split"
  ttable.0.2 1
}

pcm.speaker6 {
  type plug
  slave.pcm "split"
  ttable.0.3 1
}

pcm_slave.umc_in {
  pcm "hw:U192k"
  rate 48000
  format S32_LE
  buffer_size 4096
  period_size 1024
  channels 4
}

pcm.mic2 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 0
}

pcm.mic3 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 1
}

pcm.mic5 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 2
}

pcm.mic6 {
  type dsnoop
  ipc_key 1232
  slave umc_in
  bindings.0 3
}

The second half of this is my first attempt to split the microphone channels. When using the multiple mics test recording there is a weird lag... I don't know if that is reflected in the recorded audio data yet.

• We also want to prevent this interface from ever becoming the default audio output of the system (we dont want the birds hearing random error sounds or notifications).

To do this pactl list short sinks

Edit /etc/pulse/default.pa, comment out all lines with switch in them.

#load-module module-switch-on-port-available
#load-module module-switch-on-connect

We also added a Startup Application called "Set default audio". Check it to see what the command does.

Setting up arduinos

Like the webcams, the arduino devices are by default mapped to /dev/ttyACM0, /dev/ttyACM1, etc but the order is not always consistent. We create rules in /etc/udev/rules.d/arduino.rules to detect Arduino's with the known serial numbers to their boxes using symlinks /dev/ttyArduino_Box5, etc.

# /etc/udev/rules.d/arduino.rules
### Box 2:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351909201", SYMLINK+="ttyArduino_box2"

### Box 3:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="95232343733351403182", SYMLINK+="ttyArduino_box3"

# Box 5:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="9523234373335140A011", SYMLINK+="ttyArduino_box5"

# Box 6:
SUBSYSTEM=="tty", SUBSYSTEMS=="usb", ATTRS{manufacturer}=="Arduino (www.arduino.cc)", ATTRS{serial}=="952323437333518041A1", SYMLINK+="ttyArduino_box6"

Setting up 4TB storage drive

We ordered the computer to come with a 4TB HDD. I formatted as exFAT. However, the drive must be mounted to read/write from it. To set it up to automatically mount the drive (to /data) when you start the computer, you can make the mountpoint with sudo mkdir /data, give the fet user ownership of it with sudo chown -R fet:fet /data, find the UUID of the drive with sudo blkid, and then edit /etc/fstab with a line that tells fstab to automount the drive.

Relevant line in /etc/fstab

UUID=305ddbf7-485f-4aef-81aa-c196a600a785 /data ext4 auto,nosuid,nodev,x-gvfs-show 0 0

Set up remote access

The computer is connected to our wireless network 123E.

To give the computer a fixed ip address on the network, go to the router admin page at 192.168.0.1, Setup > Network Settings > Add DHCP reservation and map this computer plump -> 192.168.0.100 (this could be anything).

Setting up ssh

sudo apt install ssh && sudo service sshd start

Go to router config at 192.168.0.1, go to advanced > virtual server, and map private port 22 to public port 65456.

Setting up Google Drive Syncing

TODO: incloud

Downloading code

• Cloned theunissenlab/pyoperant to /home/fet/code/pyoperant

• Cloned theunissenlab/pecking_analysis to /home/fet/code/pecking_analysis

• Created separate environments for each with python3.9 -m venv env

• Copied scripts/pecking_test from chubbyninja to /home/fet/scripts/pecking_test

• Modifying scripts/pecking_test to have a debug_audio command

• Add ~/scripts to PATH and ~/code to PYTHONPATH in ~/.bashrc

Setting up jupyter notebook

In the pecking_analysis environment (cd ~/code/pecking_analysis && source env/bin/activate), install dependencies (pip install -r requirements.txt) and then run a notebook with jupyter notebook --ip 0.0.0.0 --port 8888. Or jupyter lab --ip 0.0.0.0 --port 8888.

I set up a systemctl service to run the notebook in the background so you don't need to pop up a terminal window to launch the notebook. To see if the notebook is running, systemctl status notebook. To see the log output of the notebook, journalctl -u notebook. To stop/start/restart the service, systemctl stop/start/restart notebook.

The service is defined in sudo vim /etc/systemd/system/notebook.service.

TODO: connect to notebook remotely through ssh? dont want to have a notebook publicly accessible

Troubleshooting

Freezing

The system has frozen quite a few times since we got it. The logs are saved in /home/fet/Desktop/crash_logs. If you have a crash, find the syslog file at /var/log/syslog (or /var/log/syslog.1 for the previous day's) and save those files. They might have some info about the crash.

The only common thing I've seen was a error that had something to do with a slack icon. I uninstalled slack in case this was the issue...