Network Admin

Lab Network and IT

The following sections are instructions for managing computer and network issues in the lab. They are organized roughly in descending order of how frequently the tasks are needed.

For more detailed documentation about the lab network architecture, see the Network Architecture page.

Adding a new user account

New users are added by modifying the ldap entries on our ldap server zebra as ubuntu (super user).

1. Connect to zebra

   ssh zebra -l ubuntu

2. Create the user account

   Run the following command, replacing NEWUSER with the desired username.

   sudo ldapadduser NEWUSER theunissen

3. Modify the user ldap entry

   sudo ldapmodifyuser newuser
       changetype: modify
       replace: homeDirectory
       homeDirectory: /auto/fhome/NEWUSER
   

   This sets the correct path for the user's home directory, remember to replace NEWUSER with the username set in step 2.

4. Take note of the user's uid and gid

   In the output of the previous step, find the uid and gid numbers (the uid should be unique to the user, and the gid is 702 for theunissen). These will be used in the next step.

5. Create the new user's home directory in nashome

   Login into nashome as root to make the new home directory and change its owner and group. Remember to replace NEWUSER with the new username.

   ssh root@nashome (password is fff)

   cd /raid/data/_NAS_NFS_Exports_/homes/fethome

   mkdir NEWUSER

   chown UID NEWUSER (uid is a number you got from ldap in 3 above)

   chgrp GID NEWUSER (gid is a number you got from ldap in 3 above - probably 702)

6. Set the user's password

   Logout from nashome and log in as ubuntu user on zebra. Run the command and create a new password.

   sudo ldapsetpasswd newuser

7. Login from one of the desktops to see if it all works. You will also want to check that .bashrc file from one of the older users (e.g. /auto/fhome/fet/.bashrc) and take what you like from there and edit to make it your own.

Setting up a user's VPN credentials

1. Connect to zebra

   ssh zebra -l ubuntu

2. Go to the easy-rsa directory and run the makeuser script

   cd easy-rsa
   ./makeuser KEYNAME
   

   This generates a new private key and certificate for a new VPN user under the name KEYNAME. For convenience, KEYNAME can be the user's normal username they use to log into the lab computers, but it doesn't need to be. (In fact, you can generate multiple VPN keys for someone to use. This can be important if a key ever needs to be revoked (see below)).

   The script will first ask you for passwords for the VPN user as well as a password to encrypt the new user's private key with. The final password prompt is for you, dear reader, to unlock zebra's private key that is used to sign the new user's certificate.

3. Copy the keys to your local computer

   The keys are created on zebra at /home/ubuntu/keys/KEYNAME_keys.tar.gz. Copy these from zebra to your local computer (e.g. with ssh):

   # From your own computer
   ssh ubuntu@zebra:/home/ubuntu/keys/KEYNAME_keys.tar.gz .
   

4. Send the keys to the user

   The user will use these keys as well as their two passwords (VPN password and private key password) to connect to the VPN. (link to instructions).

Revoking VPN keys

If someone's laptop or device used to connect to the VPN is lost, stolen, or hacked, any VPN keys on that device should be revoked, and new ones generated. Lists of revoked keys appear on zebra in the file ~/easy-rsa/pki/index.txt. Do not edit this file yourself unless you know what you are doing.

To revoke a key

1. Connect to zebra

   ssh zebra -l ubuntu

2. Go to the easy-rsa directory and run the revoke command

   cd easy-rsa
   ./easyrsa revoke KEYNAME
   

   Here, KEYNAME should correspond to the KEYNAME used above when generating the keys in the first place.

3. Verify the keys have been revoked

   Use the command cat ~/easy-rsa/pki/index.txt to verify that the key you have revoked is marked as revoked (R).

Set up a new workstation

Do this when you build/buy a new computer and want to connect it to the lab network. These instructions apply to Ubuntu only - for Mac and/or Windows, you will need to solve (and document) the setup yourself, or simply use the VPN.

1. Install Ubuntu; our workstations are all on 16.04 as of Nov 2020, but should be upgraded to 20.04 in the near future.

2. Assign a desired hostname

3. Plug the computer into an ethernet port on our internal network (LKS)

4. Copy the script activate-netlogin.sh from zebra (/home/ubuntu/activate-netlogin.sh) and run it with default responses

5. Set up SSH: sudo apt-get install ssh and start it with sudo service sshd start

6. Edit the lighdm config: Write the following to the file /etc/lightdm/lightdm.conf.d/50-manual-login.conf

   [Seat:*]
   greeter-show-manual-login=true
   allow-guest=false
   

7. Install vital software: ssh, screen, htop, git

Connecting by SSH to zebra

Zebra is in charge of authentication on our lab computers and network. This is where you'll most likely want to be debugging if things are going wrong.

A SSH connection to zebra requires the client's public key added to zebra's authorized keys. (TODO: add a separate instructions on how to do this). Connect to zebra as user ubuntu, e.g. ssh ubuntu@zebra.

Generate SSH Keys

1. On the computer you will be connecting from (e.g. your laptop, lab workstation), create a new set of ssh keys with ssh-keygen -t ed25519 -C "[email protected]".

2. You will be prompted to enter a file to save the key. By default it will be something like ~/.ssh/id_ed25519, but you can name it something else (if you want to have multiple SSH credentials, for example).

3. When prompted, secure your SSH keys with a password.

4. This will have created a priavte key at ~/.ssh/id_ed25519 and a public key at ~/.ssh/id_ed25519.pub. The private key should never be shared or leave your computer. If the private key is exposed, anyone with it can connect as you. The public key, however, can be freely shared with any server or service that you would like to be able to verify your identity.

5. (Optional) If you add your key to the ssh agent, you won't need to enter your key's password every time you use it. On ubuntu you can do this with ssh-add ~/.ssh/id_ed25519 (replace with the name of the key you generated in the previous steps).

Adding a SSH key to a host's authorized keys

6. If you add your key to the host's "authorized keys", you can connect without entering your password for that server every time. Some hosts will not allow any SSH connections that are not explicitly included in their authorized keys. To do this, connect to the host and open the file ~/.ssh/authorized_keys. Copy your public key, not your private key, and paste it in a new line in the authorized keys file.

Accessing the physical zebra, tdrive and zdrive servers

The tdrive and zdrive servers are located in the colocation datacenter (or "Colo") which is in Earl Warren Hall.

To physically access these drives, you need to get to the 3rd floor of the Colo which is restricted access. You need to call them to let you up. There is a phone on the wall of the first floor; dial the last 5 digits of the phone number posted by the phone, state your business, and someone will answer and let you up. If its your first time, you may need to have them add you to their system when you get there. The relevant contact people there (as of Nov 2019) are Debbie Meads and Agustin.

Our servers can be found at the bottom of Rack F10, tdrive in slots 3-4 and zdrive in slots 5-6. They are connected by LAN 1 to port 29 of the Gallant lab switch in rack F11. Note zebra is a virtual machine that exists on Gallant lab hardware. It is in the same rack space but to my knowledge (as of Nov 2020) is inaccessible - though I am not 100% sure on this point.

To connect to the NAS software on them, you will need to connect by direct ethernet connection (note that normally you can connect to the zdrive Synology Assistant on the lab VPN by connecting to https://zdrive.fet.berkeley.edu:5001, and don't need to be there in person).

Troubleshooting

I can't access one of the lab's shared data drives (tdrive, zdrive, fhome, ...)

The remote mounting is configured on zebra. When connected to zebra, you can see a list of the drives configured for automount with the command ldapsearch.

To add a new entry, modify this template and create a new ldif file. Add the ldif entry using ldapadd. To do this, bind as root and use -W for a password prompt. The password is finchfinchfin. Use ldapsearch again to verify that the new entry is there.

I can't log in on my Ubuntu workstation / login loop

One common problem you may encounter on Ubuntu is that you put in your password on the login screen and it looks like its logging you in but you come back to the login screen again. This is typically caused when the user's home folder is inaccessible for some reason.

In all cases, the first thing to do is press Ctrl-Alt-F1 to enter the command shell (You can return to the GUI with Ctrl-Alt-F7), then follow the troubleshooting for I can't see the shared data drives.

I can't see the shared data drives

Check if network or nashome is down

Check to see if computer is connected to the network by ping zebra. Try ping nashome (the system with our home directories on it) to see if it is accessible. If not, see if it is accessible from any of the other workstations.

Retry mounting

There is sometimes a timing bug where the computer tries to mount fhome before the network is accessible. Running sudo service autofs restart should retry the automounting. Check ls /auto/fhome and ls /auto/tdrive and ls /auto/zdrive to see if they are visible. If they are, hit Ctrl-Alt-F7 and attempt to log in again.

Still doesnt work

If /auto/fhome is still not visible, see if you can locate the activate-netlogin.sh script on the system (might be located in the root directory / from when the computer was first set up). If you find it, re-run that script (in particular the imporant line of the script is pam-auth-update). Ctrl-Alt-F7 and attempt to log in again.

Just manually mount the drives

If none of the above worked (or you didn't find the actiavte-netlogin.sh script), you can manually run the commands to mount the remote drives. If they don't exist already, you might need to mkdir /auto/zdrive, mkdir /auto/tdrive, and mkdir /auto/fhome before running these commands:

sudo mount -t nfs zdrive:/volume1/zdrive /auto/zdrive
sudo mount -t nfs tdrive:/mnt/NAS/tdrive/tdrive /auto/tdrive
sudo mount -t nfs nashome:/raid/data/_NAS_NFS_Exports_/homes/fethome /auto/fhome

I can't SSH to zebra and/or finch

• First, can you ping zebra/ping finch? If you can't there is probably an issue with the network itself.

• You can't SSH to zebra (or finch) normally with your username and password; your SSH public key must be added to the host's list of authorized keys before it will accept your SSH connection.

• While you can connect to zebra as any user, finch only has two users: root and ubuntu. Make sure you are connecting with ssh ubuntu@finch, not just ssh finch.

• Try connecting to zebra with its external IP address. Same for finch. If you can't, the server you are connecting to might be down. If you can connect with the external IP but not the internal IP / hostname, then the problem is probably in the network. In particular, the link between zebra and finch may be down (see next section).

I can't connect to zebra from finch (or vice versa)

Zebra and finch are connected by something called IPsec. If this connection is broken, users will not be able to log in (since zebra is in charge of authentication, and the workstations are connected to finch).

To check this, connect to finch as user ubuntu. (Note: as of Oct 2020, I'm not sure how to ssh into finch since my connections get rejected - Kevin). While connected, check the ipsec connection with sudo ipsec status.

You should see something like this

Security Associations (1 up, 0 connecting):
   tlab-lab[1]: ESTABLISHED 2 minutes ago, 169.229.146.178[169.229.146.178]...169.229.219.169[169.229.219.169]  
   tlab-lab{1}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c2eb3265_i c82afb2f_o  
   tlab-lab{1}:   10.0.2.0/24 === 10.0.1.0/24

If you instead see

Security Associations (0 up, 0 connecting):    
   none

you should restart the tunnel using the command sudo ipsec restart. The other end should automatically reconnect.

I can't read or save to files (or directories) on zdrive or tdrive that I should be able to

If there are issues with reading or writing files that other users (or yourself) have created on the shared data drives (in /auto/zdrive, /auto/tdrive, or any subdirectories), it is possible that permissions on the NAS are not configured correctly.

A temporary fix is to add the correct permissions with chmod (for example chmod 777 filename.txt to give all permissions to everyone on filename.txt).

However, a permanent fix may involve fixing the configuration on the NAS system. Connect to zdrive's Synology Assistant software in the browser. The configuration settings we have changed are documented in this wiki - if you solve this or change any other configuration, make sure to make a note of it there.