From f09e02fc63bee9b25868bef61427883348e333e7 Mon Sep 17 00:00:00 2001 From: lukemartinlogan Date: Fri, 13 Dec 2024 00:26:43 -0600 Subject: [PATCH 1/2] Update readme --- .../04-cpp-introduction/02-cpp-build-manually.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/02-hpc-tutorials/04-cpp-introduction/02-cpp-build-manually.md b/docs/02-hpc-tutorials/04-cpp-introduction/02-cpp-build-manually.md index cd49472c..a62860be 100644 --- a/docs/02-hpc-tutorials/04-cpp-introduction/02-cpp-build-manually.md +++ b/docs/02-hpc-tutorials/04-cpp-introduction/02-cpp-build-manually.md @@ -24,9 +24,9 @@ cases, one could develop a single generic database technology, and then build the movie and grocery databases using that single technology. In C++, this can be done using a shared library. In our example: -1. [src/database_lib.cc](https://github.com/scs-lab/scs-tutorial/blob/main/3.2.building_cpp/src/database_lib.cc) implements the CRUD operations. -2. [src/grocery_db.cc](https://github.com/scs-lab/scs-tutorial/blob/main/3.2.building_cpp/src/grocery_db.cc) implements the grocery database on top of CRUD -3. [src/movies_db.cc](https://github.com/scs-lab/scs-tutorial/blob/main/3.2.building_cpp/src/movies_db.cc) implements the movie database on top of CRUD +1. [src/database_lib.cc](https://github.com/grc-iit/grc-tutorial/blob/main/cpp/01-cpp-build-manually/src/database_lib.cc) implements the Create, Read, Update, Delete (CRUD) operations. +2. [src/grocery_db.cc](https://github.com/grc-iit/grc-tutorial/blob/main/cpp/01-cpp-build-manually/src/grocery_db.cc) implements the grocery database using CRUD operations +3. [src/movies_db.cc](https://github.com/grc-iit/grc-tutorial/blob/main/cpp/01-cpp-build-manually/src/movies_db.cc) implements the movie database using CRUD operations ## Setup + Repo Structure From dfd23717a7219a16b80dcc4069432bcd914378b5 Mon Sep 17 00:00:00 2001 From: lukemartinlogan Date: Fri, 13 Dec 2024 00:34:59 -0600 Subject: [PATCH 2/2] Docker cluster basics --- .../05-docker/01-docker-basics.md | 144 ++++++++++++ .../05-docker/02-docker-cluster.md | 214 ++++++++++++++++++ 2 files changed, 358 insertions(+) create mode 100644 docs/02-hpc-tutorials/05-docker/01-docker-basics.md create mode 100644 docs/02-hpc-tutorials/05-docker/02-docker-cluster.md diff --git a/docs/02-hpc-tutorials/05-docker/01-docker-basics.md b/docs/02-hpc-tutorials/05-docker/01-docker-basics.md new file mode 100644 index 00000000..193c2db3 --- /dev/null +++ b/docs/02-hpc-tutorials/05-docker/01-docker-basics.md @@ -0,0 +1,144 @@ +# Docker Guide +This is a brief guide on how to use Docker. Docker is a containerization +framework. Containers are used run multiple OSes at once without +having to reboot or log out. There are many uses for containers: +1. Portability: For example, code programmed on for a Linux system +can now execute on a Mac or Windows machine. +2. Testing: You can test your code on multiple OSes just by spawning +different containers for each OS and running your unit tests there. +3. Reliability: Containers can be migrated if a machine is expected +to go down. This is frequently done by cloud providers + +## Installation + +The official install guide is [here](https://docs.docker.com/engine/install/). +You do need root priviliges to install and use docker. + +## Docker Basics + +In this section, we go through an example of a Dockerfile and how to create a +container. + +### Setup + +First, cd into the tutorial directory. +```bash +cd ${SCS_TUTORIAL}/5.1.docker_basics +``` + +This directory contains a single file: Dockerfile + +## Create a Dockerfile +Below is an example [Dockerfile](https://github.com/grc-iit/grc-tutorial/blob/main/docker/01-docker-basics/Dockerfile) which creates a basic Ubuntu20.04 container. +```docker +# Install ubuntu 20.04 +FROM ubuntu:20.04 +LABEL maintainer="llogan@hawk.iit.edu" +LABEL version="0.0" +LABEL description="An example docker image" + +# Disable Prompt During Packages Installation +ARG DEBIAN_FRONTEND=noninteractive + +# Update ubuntu +RUN apt update && apt install + +# Install some basic packages +RUN apt install -y \ + openssh-server \ + sudo + +# Set an environment variable +ENV MY_VAR=hi + +# Print environment variable +RUN echo ${MY_VAR} +``` + +1. FROM ubuntu:20.04 indicates the OS version that docker should install. +There are other OSes, such as fedora:latest, ubuntu:latest, centos:centos8. +This can be useful for testing portability. +2. LABEL parameters are just metadata +3. RUN executes a command as if in a terminal +4. ENV sets an environment variable + +## Build the container image + +First, the container image must be built. This will parse the Dockerfile, install the OS, and run all commands in the Dockerfile. +The syntax is as follows: +```bash +sudo docker build -t [IMAGE_NAME] [DOCKERFILE_DIR, can be a github link] -f [DOCKERFILE_NAME] +``` +1. IMAGE_NAME: a semantic name for the image being built. NOTE: the name must be in snake case (i.e., no caps). +2. DOCKERFILE_DIR: the directory containing the Dockerfile. +3. DOCKERFILE_NAME: the name of the dockerfile in that directory. This is optional. Default: Dockerfile. + +Let's say that our Dockerfile is located at ${HOME}/MyDockerfiles/Dockerfile. +We could build the image two ways: +``` +# Option 1: a single command +sudo docker build -t myimage ${HOME}/MyDockerfiles + +# Option 2: cd into the directory +cd ${HOME}/MyDockerfiles +sudo docker build -t myimage . +``` + +## Run the container + +Next, we must run the container. This will create a container from the container image. There can be multiple containers made from the same image. +The syntax is as follows: +```bash +sudo docker run [OPTIONS] [IMAGE_NAME] [COMMAND (optional)] +``` +1. OPTIONS: There are many settings which docker provides. We'll go over some of them below. +2. IMAGE_NAME: The semantic name of the image to build the container from +3. COMMAND: An optional command to run within the container. + +This command will create a container CONTAINER_ID from IMAGE_NAME which uses the host network to connect to the internet and download packages. + +In our case, we want to make the container interactive (i.e., have a shell): +``` +sudo docker run -it --name mycontainer --network host myimage +``` +We use the option "-it" to specify this is an interactive session. + +## Interacting with the container + +You can reconnect to an interactive container's shell using docker exec. The syntax is as follows: +```bash +sudo docker exec [CONTAINER_ID] /bin/bash +``` + +You can now run commands within the image. For us, this would be: +```bash +sudo docker exec mycontainer /bin/bash +``` + +## Useful Commands +```bash +# Run a container with a shared directory between guest and host +sudo docker run -it --name [CONTAINER_ID] --mount src=[HOST_PATH],target=[CONTAINER_PATH],type=bind --network host [IMAGE_NAME] + +# List all running containers +sudo docker container ls + +# List all container IDs +sudo docker container ls --all + +# Get interactive shell for container +sudo docker exec [CONTAINER_ID] /bin/bash + +# Execute command in container +docker exec [CONTAINER_ID] [COMMAND] + +# Kill a running container +sudo docker stop [CONTAINER_ID] + +# Delete a container +sudo docker rm [CONTAINER_ID] + +# Commit the state of a container CONTAINER_ID into a new container +# COPY_CONTAINER_ID +sudo docker commit [CONTAINER_ID] [COPY_CONTAINER_ID] +``` diff --git a/docs/02-hpc-tutorials/05-docker/02-docker-cluster.md b/docs/02-hpc-tutorials/05-docker/02-docker-cluster.md new file mode 100644 index 00000000..d739eb30 --- /dev/null +++ b/docs/02-hpc-tutorials/05-docker/02-docker-cluster.md @@ -0,0 +1,214 @@ +# Docker Clusters + +It can be useful to create clusters of Docker images for purposes of continuous +integration. In this section, we provide an example of spawning a cluster of two +nodes and executing commands in them. + +## Setup + +First, cd into the correct tutorial directory. +```bash +cd ${SCS_TUTORIAL}/5.2.docker_clusters +``` + +This directory contains two files: +1. Dockerfile +2. docker-compose.yml + +### Create SSH keys + +Next we need to create SSH keys. We will place the SSH keys +in the current working directory, **NOT** in ~/.ssh. Data cannot +be copied to a Docker container at build time unless that data +is a subdirectory of the current working directory. + +```bash +ssh-keygen -t rsa -f ${PWD}/id_rsa -N "" -q +``` +**-t rsa** uses RSA for the algorithm. +**-f ${PWD}/id_rsa** defines the output for the private key to be in this directory. +**-N ""** indicates no password should be generated. +**-q** disables interactive prompts. + +## OpenSSH-Server Dockerfile + +We have a sample [Dockerfile](https://github.com/scs-lab/scs-tutorial/blob/main/5.2.docker_clusters/Dockerfile) which provides passwordless openssh +daemon in ubuntu 20.04. We describe the sections of the Dockerfile below. + +### Install OpenSSH + +First, we install openssh, sudo, some text editors, and git. +Technically, git and the text editors aren't required, but they +almost always come in useful in real projects. + +```dockerfile +# Install ubuntu 20.04 +FROM ubuntu:20.04 +LABEL maintainer="llogan@hawk.iit.edu" +LABEL version="0.0" +LABEL description="An example docker image" + +# Disable Prompt During Packages Installation +ARG DEBIAN_FRONTEND=noninteractive + +# Update ubuntu +RUN apt update && apt install + +# Install some basic packages +RUN apt install -y \ + openssh-server \ + sudo git nano vim +``` + +### Create a user + +Next, we create a new user called "sshuser". Many tools complain about +using root mode for everything. While technically safe to do in a container, +we make a custom user anyway. + +```dockerfile +# Create a new user +# -m makes the home directory +RUN useradd -m sshuser + +# Make the user an admin +RUN usermod -aG sudo sshuser + +# Disable password for this user +RUN passwd -d sshuser +``` + +### Copy SSH keys + +We now copy the SSH keys from the host machine to the client machine and give +them the proper permissions. The SSH keys we created in section 5.2.2 should be +located in the same directory as this Dockerfile. + +```dockerfile +# Copy the host's SSH keys +# Docker requires COPY be relative to the current working +# directory. We cannot pass ~/.ssh/id_rsa unfortunately... +RUN sudo -u sshuser mkdir ${SSHDIR} +COPY id_rsa ${SSHDIR}/id_rsa +COPY id_rsa.pub ${SSHDIR}/id_rsa.pub + +# Authorize host SSH keys +RUN sudo -u sshuser touch ${SSHDIR}/authorized_keys +RUN cat ${SSHDIR}/id_rsa.pub >> ${SSHDIR}/authorized_keys + +# Set SSH permissions +RUN chmod 700 ${SSHDIR} +RUN chmod 644 ${SSHDIR}/id_rsa.pub +RUN chmod 600 ${SSHDIR}/id_rsa +RUN chmod 600 ${SSHDIR}/authorized_keys +``` + +### Start SSH server + +Lastly, we configure the openssh server to allow for empty passwords and +then start it. +```dockerfile +# Enable passwordless SSH +# Replaces #PermitEmptyPasswords no with PermitEmptyPasswords yes +RUN sed -i 's/#PermitEmptyPasswords no/PermitEmptyPasswords yes/' /etc/ssh/sshd_config + +# Create this directory, because sshd doesn't automatically +RUN mkdir /run/sshd + +# Start SSHD +CMD ["/usr/sbin/sshd", "-D"] +``` + +## Docker Compose File + +Docker compose is used to spawn multiple docker containers. This has +a separate configuration. + +Below is our example [docker-compose.yaml](https://github.com/scs-lab/scs-tutorial/blob/main/5.2.docker_clusters/docker-compose.yaml) +```yaml +version: "3" + +services: + node1: + build: . + links: + - node2 + networks: + - net + hostname: node1 + stdin_open: true + tty: true + + node2: + build: . + networks: + - net + hostname: node2 + stdin_open: true + tty: true + +networks: + net: + driver: bridge +``` + +Here we create two nodes: node1 and node2. The "services" section represents the +set of nodes that will be spawned. +1. node1 and node2 are the names of the containers that will be spawned. +2. build: where docker-compose will search for the Dockerfile. In our case, +its the local directory. We used the default names for the Dockerfile and +docker-compose.yaml. +3. networks: label the network the containers are apart of. +"net" is not special; it is just a name, it can be anything. +4. hostname: the name of the host on the network. We force the containers +hostname to be equivalent to the name of the container. +5. links: enable communication between two nodes. Note, node2 doesn't specify +a link to node1. This is because links are already two-way, so it will +result in a cyclic dependency error. + +## Build the cluster + +First we have to build the container images for the cluster. This will +parse docker-compose.yaml (which is the default name used by docker-compose) +```bash +sudo HOST_USER=${USER} docker-compose build +``` + +## Spawn the cluster + +To spawn the cluster, run the following command +```bash +sudo HOST_USER=${USER} docker-compose up -d +``` + +## Execute commands + +First, we will verify node1 and node2 can be accessed: +```bash +sudo docker-compose exec -u sshuser node1 hostname +sudo docker-compose exec -u sshuser node2 hostname +``` + +These commands should print "node1" and "node2". +![docker-compose exec hostname results](images/5/5.2.7.docker-exec-hostname.png) + +Next, we will try performing ssh from one node into the other. +```bash +sudo docker-compose exec -u sshuser node1 ssh node2 hostname +``` + +The above command will execute "ssh node2 hostname" in node1. Its +result should be: +![docker-compose exec ssh results](images/5/5.2.7.ssh-test.png) + +## Interactive shell with cluster nodes + +To get an interactive shell of a node in the cluster, do the following +```bash +sudo docker-compose exec -u sshuser node1 /bin/bash +``` + +## Shutdown the cluster +```bash +sudo docker-compose down +```