Skip to content

Commit

Permalink
Change analogy for consistency between dependencies and containers
Browse files Browse the repository at this point in the history
  • Loading branch information
tpfau committed Aug 30, 2024
1 parent c84b445 commit 46b2957
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 24 deletions.
14 changes: 7 additions & 7 deletions content/dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ From [xkcd - dependency](https://xkcd.com/2347/). Another image that might be fa
````{discussion} Kitchen analogy
- Software <-> recipe
- Data <-> ingredients
- Libraries <-> cooking books/blogs
- Libraries <-> Pots/Tools
```{figure} img/kitchen/recipe.png
:alt: Cooking recipe in an unfamiliar language
Expand All @@ -38,7 +38,7 @@ Cooking recipe in an unfamiliar language [Midjourney, CC-BY-NC 4.0]
:alt: Kitchen with few open cooking books
:width: 50%
When we create recipes, we often use existing recipes written by others (libraries) [Midjourney, CC-BY-NC 4.0]
When we create recipes, we often use tools created by others (libraries) [Midjourney, CC-BY-NC 4.0]
```
````

Expand All @@ -48,6 +48,7 @@ When we create recipes, we often use existing recipes written by others (librari

**Conda, Anaconda, pip, virtualenv, Pipenv, pyenv, Poetry, requirements.txt,
environment.yml, renv**, ..., these tools try to solve the following problems:

- **Defining a specific set of dependencies**, possibly with well defined versions
- **Installing those dependencies** mostly automatically
- **Recording the versions** for all dependencies
Expand All @@ -61,7 +62,7 @@ Isolated environments are also useful because they help you make sure
that you know your dependencies!

**If things go wrong, you can delete and re-create** - much better
than debugging. The more often you re-create your environment, the
than debugging. The more often you re-create your environment, the
more reproducible it is.

---
Expand Down Expand Up @@ -244,12 +245,12 @@ Answer in the collaborative document:
become very difficult for to create the software environment required to
run the software. But at least we know the list of libraries. But we don't
know the versions.
**C**: Having a standard file listing dependencies is definitely better
than nothing. However, if the versions are not specified, you or someone
else might run into problems with dependencies, deprecated features,
changes in package APIs, etc.
**D** and **E**: In both these cases exact versions of all dependencies are
specified and one can recreate the software environment required for the
project. One problem with the dependencies that come from GitHub is that
Expand Down Expand Up @@ -304,7 +305,7 @@ information?
Have a look at the generated file and discuss what you see.
In the future — or on a different computer — we can re-create this environment with:
```console
$ conda env create -f environment.yml
```
Expand Down Expand Up @@ -352,7 +353,6 @@ information?
`````
``````


```{keypoints}
- Recording dependencies with versions can make it easier for the next person to execute your code
- There are many tools to record dependencies
Expand Down
33 changes: 16 additions & 17 deletions content/environments.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,11 @@
- 10 min demo
```


## What is a container?

Imagine if you didn't have to install things yourself, but instead you could
get a computer with the exact software for a task pre-installed? Containers
effectively do that, with various advantages and disadvantages. They are
get a computer with the exact software for a task pre-installed? Containers
effectively do that, with various advantages and disadvantages. They are
**like an entire operating system with software installed, all in one file**.

```{figure} img/docker_meme.jpg
Expand All @@ -31,16 +30,16 @@ From [reddit](https://www.reddit.com/r/ProgrammerHumor/comments/cw58z7/it_works_
- Our codes/scripts <-> cooking recipes
- Container definition files <-> like a blueprint to build a kitchen with all
utensils in which the recipe can be prepared.
- Container images <-> example kitchens
- Containers <-> identical factory-built mobile food truck kitchens
- Container images <-> showroom kitchens
- Containers <-> A real connected kitchen
Just for fun: which operating systems do the following example kitchens represent?
`````{tabs}
````{tab} 1
```{figure} img/kitchen/macos.png
:alt: Generated image of a kitchen
:width: 50%
[Midjourney, CC-BY-NC 4.0]
```
````
Expand All @@ -49,7 +48,7 @@ Just for fun: which operating systems do the following example kitchens represen
```{figure} img/kitchen/windows.png
:alt: Generated image of a kitchen
:width: 50%
[Midjourney, CC-BY-NC 4.0]
```
````
Expand All @@ -58,17 +57,16 @@ Just for fun: which operating systems do the following example kitchens represen
```{figure} img/kitchen/linux.png
:alt: Generated image of a kitchen
:width: 50%
[Midjourney, CC-BY-NC 4.0]
```
````
`````
``````


## From definition files to container images to containers

- Containers can be built to bundle *all the necessary ingredients* (data, code, environment, operating system).
- Containers can be built to bundle _all the necessary ingredients_ (data, code, environment, operating system).
- A container image is like a piece of paper with all the operating system on it. When you run it,
a transparent sheet is placed on top to form a container. The container runs and writes only on
that transparent sheet (and what other mounts have been layered on top). When you are done,
Expand All @@ -85,6 +83,7 @@ Just for fun: which operating systems do the following example kitchens represen
## The container recipe

Here is an example of a Singularity definition file ([reference](https://apptainer.org/docs/user/main/build_a_container.html#building-containers-from-apptainer-definition-files)):

```
Bootstrap: docker
From: ubuntu:20.04
Expand All @@ -102,12 +101,14 @@ From: ubuntu:20.04
```

Popular container implementations:

- [Docker](https://www.docker.com/)
- [Singularity](https://sylabs.io/docs/) (popular on high-performance computing systems)
- [Apptainer](https://apptainer.org) (popular on high-performance computing systems, fork of Singularity)
- [podman](https://podman.io/)

They are to some extent interoperable:

- podman is very close to Docker
- Docker images can be converted to Singularity/Apptainer images
- [Singularity Python](https://singularityhub.github.io/singularity-cli/) can convert Dockerfiles to Singularity definition files
Expand All @@ -118,6 +119,7 @@ They are to some extent interoperable:

Containers are popular for a reason - they solve a number of
important problems:

- Allow for seamlessly **moving workflows across different platforms**.
- Can solve the **"works on my machine"** situation.
- For software with many dependencies, in turn with its own dependencies,
Expand All @@ -129,10 +131,11 @@ important problems:
installation)

However, containers may also have some drawbacks:

- Can be used to hide away software installation problems and thereby
**discourage good software development practices**.
- Instead of "works on my machine" problem: **"works only in this container"** problem?
- They can be **difficult to modify**
- They can be **difficult to modify**
- Container **images can become large**

```{danger}
Expand Down Expand Up @@ -246,10 +249,9 @@ package repositories.
`````
``````

````{exercise} (optional) Containers-2: Installing the impossible.
````{exercise} (optional) Containers-2: Installing the impossible.
When you are missing privileges for installing certain software tools, containers can come handy.
When you are missing privileges for installing certain software tools, containers can come handy.
Here we build a Singularity/Apptainer container for installing `cowsay` and `lolcat` Linux programs.
1. Make sure you have apptainer installed:
Expand Down Expand Up @@ -282,8 +284,6 @@ Here we build a Singularity/Apptainer container for installing `cowsay` and `lol
````



````{exercise} (optional) Containers-3: Explore two really useful Docker images
You can try the below if you have Docker installed. If you have
Singularity/Apptainer and not Docker, the goal of the exercise can be to run
Expand Down Expand Up @@ -317,7 +317,6 @@ the Docker containers through Singularity/Apptainer.
- [Carpentries incubator lesson on Docker](https://carpentries-incubator.github.io/docker-introduction/)
- [Carpentries incubator lesson on Singularity/Apptainer](https://carpentries-incubator.github.io/singularity-introduction/)


```{keypoints}
- Containers can be helpful if complex setups are needed to running a specific software
- They can also be helpful for prototyping without "messing up" your own computing environment, or for running software that requires a different operating system than your own
Expand Down

0 comments on commit 46b2957

Please sign in to comment.