Skip to content

Commit

Permalink
Add mount to the target container without restarting it
Browse files Browse the repository at this point in the history
  • Loading branch information
felipecruz91 committed Oct 25, 2022
1 parent ef85896 commit 42cca84
Show file tree
Hide file tree
Showing 2 changed files with 208 additions and 104 deletions.
48 changes: 34 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,10 @@

A commandline tool for interactive troubleshooting when a container has crashed or a container image doesn't include debugging utilities, such as distroless images. Heavily inspired by `kubectl debug`, but for containers instead of Pods.

## Debugging with a temporary debug container
## Option 1: Debugging adding a mount

Sometimes a container configuration options make it difficult to troubleshoot in certain situations. For example, you can't run `docker exec` to troubleshoot your container if your container image does not include a shell or if your application crashes on startup. In these situations you can use `debug-ctr debug` to create a "copy" of the container with configuration values changed to aid debugging.

### How does it work?
`ctr-debug debug` runs a new container (a "copy" a.k.a the debugger container) that can be useful when your application is running but not behaving as you expect, and you'd like to add additional troubleshooting utilities to the container. This new container is simply a "copy" of the container you want to debug which now includes the utilities tools that you need to debug it.

The tools are first downloaded into a Docker volume from the image you specify with the `--image` flag from the `/bin` directory. When the debugger container is created, the volume is mounted at `/.debugger` and thus the tools in `/bin` from the image are available in the debugger container filesystem (e.g. `ls` will be available at `/.debugger/ls`) and added to the `PATH` automatically for you.
This approach uses [justincormack/addmount](https://github.com/justincormack/addmount) to mount the tools from a running container (e.g. `busybox`) into a target container **without** having to restart it.
The benefit of this approach is that you wouldn't lose the running state of the container and the tools are available in the target container.

For example, you can run the following container from a distroless image that doesn't have a shell:

Expand All @@ -26,21 +22,45 @@ docker exec -it my-distroless /bin/sh
OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown
```

You can bring the `sh` tool from `busybox:1.28` and simply run the following command to create a debugger container and use the `docker exec` command suggested in the output to access it:
You can bring the tools from `busybox:1.28` that are available in `/bin` into the target container (**without** having to restart it) by simply running:

```shell
debug-ctr debug --image=busybox:1.28 --target=my-distroless

...
2022/10/22 20:09:26 Starting debug container d3270296d96e77481399d130492d2b1389790e410c4996ab8eaa1b8d1a2c2f23
2022/10/25 09:32:40 -------------------------------
2022/10/25 09:32:40 Debug your container:
2022/10/25 09:32:40 $ docker exec -it my-distroless /bin/sh
2022/10/25 09:32:40 -------------------------------
```

## Option 2: Debugging using a "copy" of the container

Sometimes a container configuration options make it difficult to troubleshoot in certain situations. For example, you can't run `docker exec` to troubleshoot your container if your container image does not include a shell or if your application crashes on startup. In these situations you can use `debug-ctr debug` to create a "copy" of the container with configuration values changed to aid debugging.

### How does it work?

`debug-ctr debug` uses the `--copy-to` flag to run a new container (a "copy" a.k.a the debugger container) that can be useful when your application is running but not behaving as you expect, and you'd like to add additional troubleshooting utilities to the container. This new container is simply a "copy" of the container you want to debug which now includes the utilities tools that you need to debug it.

The tools are first downloaded into a Docker volume from the image you specify with the `--image` flag from the `/bin` directory. When the debugger container is created, the volume is mounted at `/.debugger` and thus the tools in `/bin` from the image are available in the debugger container filesystem (e.g. `ls` will be available at `/.debugger/ls`) and added to the `PATH` automatically for you.

You can bring the `sh` tool from `busybox:1.28` and simply run the following command to **create a new debugger container** and use the `docker exec` command suggested in the output to access it:

```shell
debug-ctr debug --image=busybox:1.28 --target=my-distroless --copy-to=my-distroless-copy

...
2022/10/22 20:09:26 Starting debug container my-distroless-copy
2022/10/22 20:09:26 -------------------------------
2022/10/22 20:09:26 Debug your container:
2022/10/22 20:09:26 $ docker exec -it d3270296d96e77481399d130492d2b1389790e410c4996ab8eaa1b8d1a2c2f23 /.debugger/sh -c "PATH=\$PATH:/.debugger /.debugger/sh"
2022/10/22 20:09:26 $ docker exec -it my-distroless-copy /.debugger/sh -c "PATH=\$PATH:/.debugger /.debugger/sh"
2022/10/22 20:09:26 -------------------------------
```

Note that the `docker exec` command from the output is used to **exec into the debugger container, not into the original one**.
Note that with this approach the `docker exec` command from the output is used to **exec into the debugger container, not into the original one**.

### Changing its entrypoint and/or command

Sometimes it's useful to change the entrypoint and/or command for a container, for example to add a debugging flag or because the application is crashing.

To simulate a crashing application, use docker run to create a container that immediately exits:
Expand All @@ -49,14 +69,14 @@ To simulate a crashing application, use docker run to create a container that im
docker run busybox:1.28 /bin/sh -c "false"
```

You can use `ctr-debug debug` with `--entrypoint` and/or `--cmd` to create a copy of this container with the command changed to an interactive shell:
You can use `debug-ctr debug` with `--entrypoint` and/or `--cmd` to create a copy of this container with the command changed to an interactive shell:

```shell
ctr-debug debug --image=docker.io/alpine:latest --target=my-distroless --entrypoint="/.debugger/sleep" --cmd="365d"
debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --copy-to=my-distroless-copy --entrypoint="/.debugger/sleep" --cmd="365d"
```

Now you have an interactive shell that you can use to perform tasks like checking filesystem paths or running a container command manually.

## Acknowledgements

- https://iximiuz.com/en/posts/docker-debug-slim-containers/
- https://iximiuz.com/en/posts/docker-debug-slim-containers/
Loading

0 comments on commit 42cca84

Please sign in to comment.