diff --git a/README.md b/README.md index 82a31ef..06d5ad0 100644 --- a/README.md +++ b/README.md @@ -2,14 +2,10 @@ A commandline tool for interactive troubleshooting when a container has crashed or a container image doesn't include debugging utilities, such as distroless images. Heavily inspired by `kubectl debug`, but for containers instead of Pods. -## Debugging with a temporary debug container +## Option 1: Debugging adding a mount -Sometimes a container configuration options make it difficult to troubleshoot in certain situations. For example, you can't run `docker exec` to troubleshoot your container if your container image does not include a shell or if your application crashes on startup. In these situations you can use `debug-ctr debug` to create a "copy" of the container with configuration values changed to aid debugging. - -### How does it work? -`ctr-debug debug` runs a new container (a "copy" a.k.a the debugger container) that can be useful when your application is running but not behaving as you expect, and you'd like to add additional troubleshooting utilities to the container. This new container is simply a "copy" of the container you want to debug which now includes the utilities tools that you need to debug it. - -The tools are first downloaded into a Docker volume from the image you specify with the `--image` flag from the `/bin` directory. When the debugger container is created, the volume is mounted at `/.debugger` and thus the tools in `/bin` from the image are available in the debugger container filesystem (e.g. `ls` will be available at `/.debugger/ls`) and added to the `PATH` automatically for you. +This approach uses [justincormack/addmount](https://github.com/justincormack/addmount) to mount the tools from a running container (e.g. `busybox`) into a target container **without** having to restart it. +The benefit of this approach is that you wouldn't lose the running state of the container and the tools are available in the target container. For example, you can run the following container from a distroless image that doesn't have a shell: @@ -26,21 +22,45 @@ docker exec -it my-distroless /bin/sh OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown ``` -You can bring the `sh` tool from `busybox:1.28` and simply run the following command to create a debugger container and use the `docker exec` command suggested in the output to access it: +You can bring the tools from `busybox:1.28` that are available in `/bin` into the target container (**without** having to restart it) by simply running: ```shell debug-ctr debug --image=busybox:1.28 --target=my-distroless ... -2022/10/22 20:09:26 Starting debug container d3270296d96e77481399d130492d2b1389790e410c4996ab8eaa1b8d1a2c2f23 +2022/10/25 09:32:40 ------------------------------- +2022/10/25 09:32:40 Debug your container: +2022/10/25 09:32:40 $ docker exec -it my-distroless /bin/sh +2022/10/25 09:32:40 ------------------------------- +``` + +## Option 2: Debugging using a "copy" of the container + +Sometimes a container configuration options make it difficult to troubleshoot in certain situations. For example, you can't run `docker exec` to troubleshoot your container if your container image does not include a shell or if your application crashes on startup. In these situations you can use `debug-ctr debug` to create a "copy" of the container with configuration values changed to aid debugging. + +### How does it work? + +`debug-ctr debug` uses the `--copy-to` flag to run a new container (a "copy" a.k.a the debugger container) that can be useful when your application is running but not behaving as you expect, and you'd like to add additional troubleshooting utilities to the container. This new container is simply a "copy" of the container you want to debug which now includes the utilities tools that you need to debug it. + +The tools are first downloaded into a Docker volume from the image you specify with the `--image` flag from the `/bin` directory. When the debugger container is created, the volume is mounted at `/.debugger` and thus the tools in `/bin` from the image are available in the debugger container filesystem (e.g. `ls` will be available at `/.debugger/ls`) and added to the `PATH` automatically for you. + +You can bring the `sh` tool from `busybox:1.28` and simply run the following command to **create a new debugger container** and use the `docker exec` command suggested in the output to access it: + +```shell +debug-ctr debug --image=busybox:1.28 --target=my-distroless --copy-to=my-distroless-copy + +... +2022/10/22 20:09:26 Starting debug container my-distroless-copy 2022/10/22 20:09:26 ------------------------------- 2022/10/22 20:09:26 Debug your container: -2022/10/22 20:09:26 $ docker exec -it d3270296d96e77481399d130492d2b1389790e410c4996ab8eaa1b8d1a2c2f23 /.debugger/sh -c "PATH=\$PATH:/.debugger /.debugger/sh" +2022/10/22 20:09:26 $ docker exec -it my-distroless-copy /.debugger/sh -c "PATH=\$PATH:/.debugger /.debugger/sh" 2022/10/22 20:09:26 ------------------------------- ``` -Note that the `docker exec` command from the output is used to **exec into the debugger container, not into the original one**. +Note that with this approach the `docker exec` command from the output is used to **exec into the debugger container, not into the original one**. + ### Changing its entrypoint and/or command + Sometimes it's useful to change the entrypoint and/or command for a container, for example to add a debugging flag or because the application is crashing. To simulate a crashing application, use docker run to create a container that immediately exits: @@ -49,14 +69,14 @@ To simulate a crashing application, use docker run to create a container that im docker run busybox:1.28 /bin/sh -c "false" ``` -You can use `ctr-debug debug` with `--entrypoint` and/or `--cmd` to create a copy of this container with the command changed to an interactive shell: +You can use `debug-ctr debug` with `--entrypoint` and/or `--cmd` to create a copy of this container with the command changed to an interactive shell: ```shell -ctr-debug debug --image=docker.io/alpine:latest --target=my-distroless --entrypoint="/.debugger/sleep" --cmd="365d" +debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --copy-to=my-distroless-copy --entrypoint="/.debugger/sleep" --cmd="365d" ``` Now you have an interactive shell that you can use to perform tasks like checking filesystem paths or running a container command manually. ## Acknowledgements -- https://iximiuz.com/en/posts/docker-debug-slim-containers/ \ No newline at end of file +- https://iximiuz.com/en/posts/docker-debug-slim-containers/ diff --git a/cmd/debug.go b/cmd/debug.go index 99ecaae..56a4022 100644 --- a/cmd/debug.go +++ b/cmd/debug.go @@ -3,10 +3,6 @@ package cmd import ( "context" "fmt" - "github.com/docker/docker/api/types" - "github.com/docker/docker/api/types/container" - "github.com/docker/docker/api/types/strslice" - "github.com/docker/docker/client" "io" "log" "os" @@ -14,9 +10,16 @@ import ( "runtime" "strings" + "github.com/docker/docker/api/types" + "github.com/docker/docker/api/types/container" + "github.com/docker/docker/api/types/strslice" + "github.com/docker/docker/client" + "github.com/spf13/cobra" ) +const addMountImage = "justincormack/addmount:latest" + var ( cli *client.Client @@ -24,13 +27,15 @@ var ( cmdFlag []string ) -// debugCmd represents the debug command var debugCmd = &cobra.Command{ Use: "debug", Short: "Debug a container using a image", Long: `A way to interactively inspect a container filesystem with the utilities you need.`, - Example: `debug-ctr debug --image=busybox:1.28 --target=my-distroless -debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --entrypoint="/.debugger/sleep" --cmd="365d" + Example: ` +debug-ctr debug --target=my-distroless +debug-ctr debug --image=busybox:1.28 --target=my-distroless +debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --copy-to=my-distroless-copy +debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --copy-to=my-distroless-copy --entrypoint="/.debugger/sleep" --cmd="365d" `, PersistentPreRunE: func(cmd *cobra.Command, args []string) error { var err error @@ -40,103 +45,44 @@ debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --entrypo RunE: func(cmd *cobra.Command, args []string) error { debugImage, _ := cmd.PersistentFlags().GetString("image") targetContainer, _ := cmd.PersistentFlags().GetString("target") + copyContainerName, _ := cmd.PersistentFlags().GetString("copy-to") entryPointOverride := entrypointFlag cmdOverride := cmdFlag ctx := context.Background() - // Get the bin folder of the image fs into a volume - reader, err := cli.ImagePull(ctx, debugImage, types.ImagePullOptions{ - Platform: "linux/" + runtime.GOARCH, - }) - if err != nil { - return err - } - _, err = io.Copy(os.Stdout, reader) - if err != nil { - return err - } - - // Create one volume per container to debug to avoid overwriting binaries - volumeName := strings.Replace(strings.Replace(debugImage, ":", "_", 1), "/", "_", 1) - volume := fmt.Sprintf("debug-ctr-%s", volumeName) - resp, err := cli.ContainerCreate(ctx, &container.Config{ - Image: debugImage, - }, &container.HostConfig{ - AutoRemove: true, - Binds: []string{ - volume + ":" + "/bin", - }, - }, nil, nil, "") + // Check target container exists + _, err := cli.ContainerInspect(ctx, targetContainer) if err != nil { return err } - if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil { - return err - } - - // Create debug container - - inspect, err := cli.ContainerInspect(ctx, targetContainer) - if err != nil { + if err := pullImage(ctx, debugImage); err != nil { return err } - // For example, you can't run docker exec to troubleshoot your container if your container image does not include a shell or if your application crashes on startup. - // In these situations you can use debug-ctr debug to create a copy of the container with configuration values changed to aid debugging. - var containerEntrypoint = inspect.Config.Entrypoint - if len(entryPointOverride) > 0 { - x := strslice.StrSlice{} - for _, y := range entryPointOverride { - x = append(x, y) + debugContainer := targetContainer + dockerExecCmd := "" + if copyContainerName == "" { + if err := addMountToTargetContainer(ctx, debugImage, targetContainer); err != nil { + return err } - containerEntrypoint = x - } - log.Printf("entrypoint: %+v", containerEntrypoint) - - var containerCmd = inspect.Config.Cmd - //var containerCmd = []string{"sh", "-c", `export PATH=$PATH:/.debugger /.debugger/sh`} //TODO: last part can be just "sh"? - if len(cmdOverride) > 0 { - x := strslice.StrSlice{} - for _, y := range cmdOverride { - x = append(x, y) - } - containerCmd = x - } - log.Printf("containerCmd: %+v", containerCmd) - - targetContainerCreate, err := cli.ContainerCreate(ctx, &container.Config{ - Image: inspect.Image, - User: inspect.Config.User, - Env: inspect.Config.Env, - Entrypoint: containerEntrypoint, - Cmd: containerCmd, - WorkingDir: inspect.Config.WorkingDir, - Labels: inspect.Config.Labels, - }, &container.HostConfig{ - Binds: []string{ - //TODO: provide support for nixery images - volume + ":" + "/.debugger", - }, - }, nil, nil, "") - if err != nil { - return err - } + dockerExecCmd = fmt.Sprintf("docker exec -it %s /bin/sh", debugContainer) + } else { - log.Printf("Starting debug container %s", targetContainerCreate.ID) - if err := cli.ContainerStart(ctx, targetContainerCreate.ID, types.ContainerStartOptions{}); err != nil { - return err + if err := createCopyContainer(ctx, debugImage, targetContainer, copyContainerName, entryPointOverride, cmdOverride); err != nil { + return err + } + dockerExecCmd = fmt.Sprintf(`docker exec -it %s /.debugger/sh -c "PATH=\$PATH:/.debugger /.debugger/sh"`, copyContainerName) } log.Println("-------------------------------") log.Println("Debug your container:") - log.Printf(`$ docker exec -it %s /.debugger/sh -c "PATH=\$PATH:/.debugger /.debugger/sh"`, targetContainerCreate.ID) + log.Printf("$ %s", dockerExecCmd) log.Println("-------------------------------") - //TODO: if "--open" flag switch runtime.GOOS { - // TODO: windows + //TODO: windows //TODO: linux case "darwin": @@ -145,10 +91,10 @@ debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --entrypo tell current window create tab with default profile tell current session - write text "docker exec -it %s /.debugger/sh -c \"PATH=\\$PATH:/.debugger /.debugger/sh\"" + write text "%s" end tell end tell - end tell`, targetContainerCreate.ID) + end tell`, strings.ReplaceAll(strings.ReplaceAll(dockerExecCmd, `\`, `\\`), `"`, `\"`)) err := exec.Command("/usr/bin/osascript", "-e", "tell application \"iTerm\"", "-e", args).Run() if err != nil { @@ -163,11 +109,149 @@ debug-ctr debug --image=docker.io/alpine:latest --target=my-distroless --entrypo func init() { rootCmd.AddCommand(debugCmd) - debugCmd.PersistentFlags().String("image", "", "(required) The image to use for debugging purposes") - debugCmd.PersistentFlags().String("target", "", "(required) The container to debug") - debugCmd.PersistentFlags().StringArrayVar(&entrypointFlag, "entrypoint", nil, "(optional) The entrypoint to run when starting the debug container") - debugCmd.PersistentFlags().StringArrayVar(&cmdFlag, "cmd", nil, "(optional) The command to run when starting the debug container") + debugCmd.PersistentFlags().String("image", "docker.io/library/busybox:latest", "(optional) The image to use for debugging purposes") + debugCmd.PersistentFlags().String("target", "", "(required) The target container to debug") + debugCmd.PersistentFlags().String("copy-to", "", "(optional) The name of the copy container") + debugCmd.PersistentFlags().StringArrayVar(&entrypointFlag, "entrypoint", nil, "(optional) The entrypoint to run when starting the debug container (if --copy-to is specified)") + debugCmd.PersistentFlags().StringArrayVar(&cmdFlag, "cmd", nil, "(optional) The command to run when starting the debug container (if --copy-to is specified)") - _ = debugCmd.MarkPersistentFlagRequired("image") _ = debugCmd.MarkPersistentFlagRequired("target") } + +func pullImage(ctx context.Context, image string) error { + reader, err := cli.ImagePull(ctx, image, types.ImagePullOptions{ + Platform: "linux/" + runtime.GOARCH, + }) + if err != nil { + return err + } + _, err = io.Copy(os.Stdout, reader) + return err +} + +// addMountToTargetContainer mounts the tools from a running container (e.g. `busybox`) into the target container **without** having to restart it. +// The benefit of this approach is that you wouldn't lose the running state of the container and the tools are available in the target container. +func addMountToTargetContainer(ctx context.Context, debugImage, targetContainer string) error { + // Run toolkit image + toolkitContainerResp, err := cli.ContainerCreate(ctx, &container.Config{ + Image: debugImage, + Entrypoint: []string{"/bin/sh", "-c", "tail -f /dev/null"}, // keep container running in the background + }, nil, nil, nil, "") + if err != nil { + return err + } + if err := cli.ContainerStart(ctx, toolkitContainerResp.ID, types.ContainerStartOptions{}); err != nil { + return err + } + + // Add mount to the original container + if err := pullImage(ctx, addMountImage); err != nil { + return err + } + addMountContainerResp, err := cli.ContainerCreate(ctx, &container.Config{ + Image: addMountImage, + Cmd: []string{toolkitContainerResp.ID, "/bin", targetContainer, "/bin"}, + }, &container.HostConfig{ + AutoRemove: true, + Privileged: true, + PidMode: "host", + Binds: []string{ + "/var/run/docker.sock:/var/run/docker.sock", + }, + }, nil, nil, "") + if err != nil { + return err + } + if err := cli.ContainerStart(ctx, addMountContainerResp.ID, types.ContainerStartOptions{}); err != nil { + return err + } + statusCh, errCh := cli.ContainerWait(ctx, addMountContainerResp.ID, container.WaitConditionRemoved) + select { + case err := <-errCh: + if err != nil { + panic(err) + } + case <-statusCh: + } + + // Remove the toolkit container + if err := cli.ContainerRemove(ctx, toolkitContainerResp.ID, types.ContainerRemoveOptions{ + Force: true, + }); err != nil { + return err + } + return nil +} + +// createCopyContainer creates a new container (a "copy") that is used to debug. +// For example, you can't run docker exec to troubleshoot your container if your container image does not include a shell or if your application crashes on startup. +// In these situations you can use debug-ctr debug with "--copy-to" to create a copy of the container with configuration values changed to aid debugging. +func createCopyContainer(ctx context.Context, debugImage, targetContainer, copyContainerName string, entryPointOverride, cmdOverride []string) error { + // Create one volume per container to debug to avoid overwriting binaries + volumeName := strings.Replace(strings.Replace(debugImage, ":", "_", 1), "/", "_", -1) + volume := fmt.Sprintf("debug-ctr-%s", volumeName) + resp, err := cli.ContainerCreate(ctx, &container.Config{ + Image: debugImage, + }, &container.HostConfig{ + AutoRemove: true, + Binds: []string{ + volume + ":" + "/bin", + }, + }, nil, nil, "") + if err != nil { + return err + } + + if err := cli.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil { + return err + } + + // Create the "copy" container + inspect, err := cli.ContainerInspect(ctx, targetContainer) + if err != nil { + return err + } + + var containerEntrypoint = inspect.Config.Entrypoint + if len(entryPointOverride) > 0 { + x := strslice.StrSlice{} + for _, y := range entryPointOverride { + x = append(x, y) + } + containerEntrypoint = x + } + log.Printf("entrypoint: %+v", containerEntrypoint) + + var containerCmd = inspect.Config.Cmd + if len(cmdOverride) > 0 { + x := strslice.StrSlice{} + for _, y := range cmdOverride { + x = append(x, y) + } + containerCmd = x + } + log.Printf("containerCmd: %+v", containerCmd) + + copyContainerCreateResp, err := cli.ContainerCreate(ctx, &container.Config{ + Image: inspect.Image, + User: inspect.Config.User, + Env: inspect.Config.Env, + Entrypoint: containerEntrypoint, + Cmd: containerCmd, + WorkingDir: inspect.Config.WorkingDir, + Labels: inspect.Config.Labels, + }, &container.HostConfig{ + Binds: []string{ + volume + ":" + "/.debugger", + }, + }, nil, nil, copyContainerName) + if err != nil { + return err + } + + log.Printf("Starting debug container %s", copyContainerCreateResp.ID) + if err := cli.ContainerStart(ctx, copyContainerCreateResp.ID, types.ContainerStartOptions{}); err != nil { + return err + } + return nil +}