Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add timeouts for docker save and copy. #313

Merged
merged 1 commit into from
Nov 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/goal/apply.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ Runs kubectl to apply manifests.
| `timeout` | `k3s.timeout` | Timeout in seconds to wait for resources getting ready. | 300 |
| `skipApply` | `k3s.skipApply` | Skip applying kubectl manifests. | false |
| `debug` | `k3s.debug` | Stream logs of docker and kubectl. | false |
| `taskTimeout` | `k3s.taskTimeout` | Default timeout for docker tasks in seconds. | 30 |
7 changes: 5 additions & 2 deletions docs/goal/image.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,14 @@
Import images into k3s containerd.

| Name | User Property | Description | Default |
| -----| ------------- | ----------- | ------- |
| ---- | ---------------------- | ----------- | ------- |
| `ctrImages` | `k3s.ctrImages` | Download given images via `ctr image pull` inside k3s container. | [] |
| `tarFiles` | `k3s.tarFiles` | Import given tar files via `ctr image import` inside k3s container. | [] |
| `dockerImages` | `k3s.dockerImages` | Copy given images from docker deamon via `ctr image import` inside k3s container. | [] |
| `dockerPullAlways` | `k3s.dockerPullAlways` | Always pull docker images or only if not present. | false |
| `pullTimeout` | `k3s.pullTimeout` | Timout for `ctr image pull` or `docker pull` in seconds. | 1200 |
| `pullTimeout` | `k3s.pullTimeout` | Timout for `ctr image pull` or `docker pull` in seconds. | 1200 |
| `copyTimeout` | `k3s.copyToContainerTimeout` | Timout for `docker cp` in seconds. | 120 |
| `saveTimeout` | `k3s.saveImageTimeout` | Timout for `docker save` in seconds. | 120 |
| `skipImage` | `k3s.skipImage` | Skip image handling. | false |
| `debug` | `k3s.debug` | Stream logs of docker and kubectl. | false |
| `taskTimeout` | `k3s.taskTimeout` | Default timeout for docker tasks in seconds. | 30 |
1 change: 1 addition & 0 deletions docs/goal/restart.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ Restart selected resources. Usefull for local development and restarting service
| `timeout` | `k3s.timeout` | Timeout in seconds to wait for resources getting ready. | 300 |
| `skipRestart` | `k3s.skipImage` | Skip image handling. | false |
| `debug` | `k3s.debug` | Stream logs of docker and kubectl. | false |
| `taskTimeout` | `k3s.taskTimeout` | Default timeout for docker tasks in seconds. | 30 |
1 change: 1 addition & 0 deletions docs/goal/rm.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ Stop and remove k3s container.
| `includeCache` | `k3s.includeCache` | Include cache directory with downloaded images. | false |
| `skipRm` | `k3s.skipRm` | Skip removing k3s container. | false |
| `debug` | `k3s.debug` | Stream logs of docker and kubectl. | false |
| `taskTimeout` | `k3s.taskTimeout` | Default timeout for docker tasks in seconds. | 30 |
1 change: 1 addition & 0 deletions docs/goal/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,4 @@ Start and run k3s container.
| `registries` | `k3s.registries` | Path to "registry.yaml" to mount to "/etc/rancher/k3s/registries.yaml". | `null` |
| `skipRun` | `skipRun` | Skip running of k3s. | false |
| `debug` | `k3s.debug` | Stream logs of docker and kubectl. | false |
| `taskTimeout` | `k3s.taskTimeout` | Default timeout for docker tasks in seconds. | 30 |
2 changes: 1 addition & 1 deletion src/main/java/io/kokuwa/maven/k3s/mojo/ApplyMojo.java
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ public void execute() throws MojoExecutionException {
if (getDocker().getContainer().isEmpty()) {
throw new MojoExecutionException("No k3s container found");
}
getDocker().copyToContainer(manifests, toLinuxPath(path));
getDocker().copyToContainer(manifests, toLinuxPath(path), timeout);

// wait for service account, see https://github.com/kubernetes/kubernetes/issues/66689

Expand Down
30 changes: 27 additions & 3 deletions src/main/java/io/kokuwa/maven/k3s/mojo/ImageMojo.java
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,22 @@ public class ImageMojo extends K3sMojo {
@Parameter(property = "k3s.pullTimeout", defaultValue = "1200")
private Duration pullTimeout;

/**
* Timeout for "docker cp" in seconds.
*
* @since 1.5.0
*/
@Parameter(property = "k3s.copyToContainerTimeout", defaultValue = "120")
private Duration copyTimeout;

/**
* Timeout for "docker save" in seconds.
*
* @since 1.5.0
*/
@Parameter(property = "k3s.saveTimeout", defaultValue = "120")
private Duration saveTimeout;

/**
* Skip starting of k3s container.
*
Expand Down Expand Up @@ -154,7 +170,7 @@ private boolean tar(Map<String, Map<String, ?>> existingImages, Path tarFile) {
var destination = "/tmp/" + tarFile.getFileName() + "_" + System.nanoTime();
var outputPattern = Pattern.compile("^unpacking (?<image>.*) \\(sha256:[0-9a-f]{64}\\).*$");

getDocker().copyToContainer(tarFile, destination);
getDocker().copyToContainer(tarFile, destination, copyTimeout);
for (var output : getDocker().exec(pullTimeout, "ctr", "image", "import", destination.toString())) {
var matcher = outputPattern.matcher(output);
if (matcher.matches()) {
Expand Down Expand Up @@ -232,8 +248,8 @@ private boolean docker(Map<String, Map<String, ?>> existingImages, String image)
var source = Paths.get(System.getProperty("java.io.tmpdir")).resolve(filename);
var destination = "/tmp/" + filename;
try {
getDocker().saveImage(image, source);
getDocker().copyToContainer(source, destination);
getDocker().saveImage(image, source, saveTimeout);
getDocker().copyToContainer(source, destination, copyTimeout);
getDocker().exec(pullTimeout, "ctr", "image", "import", destination.toString());
getDocker().exec("ctr", "image", "label", normalizedImage, label + "=" + digest);
} catch (MojoExecutionException e) {
Expand Down Expand Up @@ -290,6 +306,14 @@ public void setPullTimeout(int pullTimeout) {
this.pullTimeout = Duration.ofSeconds(pullTimeout);
}

public void setCopyTimeout(int copyTimeout) {
this.copyTimeout = Duration.ofSeconds(copyTimeout);
}

public void setSaveTimeout(int saveTimeout) {
this.saveTimeout = Duration.ofSeconds(saveTimeout);
}

public void setSkipImage(boolean skipImage) {
this.skipImage = skipImage;
}
Expand Down
18 changes: 16 additions & 2 deletions src/main/java/io/kokuwa/maven/k3s/mojo/K3sMojo.java
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import java.io.File;
import java.nio.file.Path;
import java.time.Duration;

import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.logging.Log;
Expand All @@ -21,13 +22,21 @@
public abstract class K3sMojo extends AbstractMojo {

/**
* Enable debuging of docker and k3s logs.
* Enable debugging of docker and k3s logs.
*
* @since 1.0.0
*/
@Parameter(property = "k3s.debug", defaultValue = "false")
private boolean debug;

/**
* Default timeout for docker tasks in seconds.
*
* @since 1.5.0
*/
@Parameter(property = "k3s.taskTimeout", defaultValue = "30")
private Duration taskTimeout;

/**
* Skip plugin.
*
Expand Down Expand Up @@ -68,7 +77,8 @@ public Log getLog() {
}

public Docker getDocker() {
return docker == null ? docker = new Docker(containerName, volumeName, log) : docker;
if (docker == null && taskTimeout == null) throw new NullPointerException();
return docker == null ? docker = new Docker(containerName, volumeName, log, taskTimeout) : docker;
}

public String toLinuxPath(Path path) {
Expand All @@ -86,6 +96,10 @@ public void setDebug(boolean debug) {
this.debug = debug;
}

public void setTaskTimeout(int taskTimeout) {
this.taskTimeout = Duration.ofSeconds(taskTimeout);
}

public void setSkip(boolean skip) {
this.skip = skip;
}
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/io/kokuwa/maven/k3s/mojo/RestartMojo.java
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,8 @@ private Callable<Boolean> restart(String resoure) {
try {
getDocker().exec("kubectl", "rollout", "restart", kind, name, "--namespace=" + namespace);
log.info("{} {}/{} restarted", kind, namespace, name);
getDocker().exec("kubectl", "rollout", "status", kind, name, "--namespace=" + namespace,
"--timeout=" + timeout.getSeconds() + "s");
getDocker().exec(timeout.plusSeconds(5), "kubectl", "rollout", "status", kind, name,
"--namespace=" + namespace, "--timeout=" + timeout.getSeconds() + "s");
log.info("{} {}/{} restart finished", kind, namespace, name);
return true;
} catch (MojoExecutionException e) {
Expand Down
59 changes: 29 additions & 30 deletions src/main/java/io/kokuwa/maven/k3s/util/Docker.java
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,19 @@ public class Docker {
private final String containerName;
private final String volumeName;
private final Logger log;
private final Duration timeout;

public Docker(String containerName, String volumeName, Logger log) {
public Docker(String containerName, String volumeName, Logger log, Duration timeout) {
this.containerName = containerName;
this.volumeName = volumeName;
this.log = log;
this.timeout = timeout;
}

public Optional<Container> getContainer() throws MojoExecutionException {
return Task
.of(log, "docker", "container", "ls", "--all", "--filter=name=" + containerName, "--format={{json .}}")
.of(log, timeout, "docker", "container", "ls", "--all", "--filter=name=" + containerName,
"--format={{json .}}")
.run().stream()
.map(output -> readValue(Container.class, output))
.filter(container -> containerName.equals(container.name))
Expand All @@ -62,78 +65,74 @@ public void createContainer(String image, List<String> ports, List<String> k3s,
ports.stream().map(port -> "--publish=" + port).forEach(command::add);
command.add(image);
command.addAll(k3s);
Task.of(log, command).run();
Task.of(log, timeout, command).run();
}

public void startContainer() throws MojoExecutionException {
Task.of(log, "docker", "start", containerName).run();
Task.of(log, timeout, "docker", "start", containerName).run();
}

public void removeContainer() throws MojoExecutionException {
Task.of(log, "docker", "rm", containerName, "--force", "--volumes").run();
Task.of(log, timeout, "docker", "rm", containerName, "--force", "--volumes").run();
}

public void copyFromContainer(String source, Path destination) throws MojoExecutionException {
Task.of(log, "docker", "cp", containerName + ":" + source, destination.toString()).run();
Task.of(log, timeout, "docker", "cp", containerName + ":" + source, destination.toString()).run();
}

public void copyToContainer(Path source, String destination) throws MojoExecutionException {
public void copyToContainer(Path source, String destination, Duration copyTimeout) throws MojoExecutionException {
// suffix directories with '/.', see https://docs.docker.com/engine/reference/commandline/cp/#description
var sourceString = Files.isDirectory(source) ? source + File.separator + "." : source.toString();
Task.of(log, "docker", "cp", sourceString, containerName + ":" + destination).run();
Task.of(log, copyTimeout, "docker", "cp", sourceString, containerName + ":" + destination).run();
}

public void waitForLog(Await await, Function<List<String>, Boolean> checker) throws MojoExecutionException {
var process = Task.of(log, "docker", "logs", containerName, "--follow").timeout(Duration.ofHours(1)).start();
var process = Task.of(log, Duration.ofHours(1), "docker", "logs", containerName, "--follow").start();
await.onTimeout(() -> process.output().forEach(log::warn)).until(() -> process.output(), checker);
process.close();
}

public List<String> exec(String... commands) throws MojoExecutionException {
return exec(null, commands);
return exec(timeout, commands);
}

public List<String> exec(Duration timeout, String... commands) throws MojoExecutionException {
return exec(timeout, List.of(commands));
public List<String> exec(Duration execTimeout, String... commands) throws MojoExecutionException {
return exec(execTimeout, List.of(commands));
}

public List<String> exec(Duration timeout, List<String> commands) throws MojoExecutionException {
return execWithoutVerify(timeout, commands).verify().output();
public List<String> exec(Duration execTimeout, List<String> commands) throws MojoExecutionException {
return execWithoutVerify(execTimeout, commands).verify().output();
}

public Task execWithoutVerify(List<String> commands) throws MojoExecutionException {
return execWithoutVerify(null, commands);
return execWithoutVerify(timeout, commands);
}

public Task execWithoutVerify(Duration timeout, List<String> commands) throws MojoExecutionException {
public Task execWithoutVerify(Duration execTimeout, List<String> commands) throws MojoExecutionException {
var command = new ArrayList<String>();
command.add("docker");
command.add("exec");
command.add(containerName);
command.addAll(commands);
var task = Task.of(log, command);
if (timeout != null) {
task.timeout(timeout);
}
return task.start().waitFor();
return Task.of(log, execTimeout, command).start().waitFor();
}

// volume

public Optional<ContainerVolume> getVolume() throws MojoExecutionException {
return Task.of(log, "docker", "volume", "ls", "--filter=name=" + volumeName, "--format={{json .}}")
return Task.of(log, timeout, "docker", "volume", "ls", "--filter=name=" + volumeName, "--format={{json .}}")
.run().stream()
.map(output -> readValue(ContainerVolume.class, output))
.filter(volume -> volumeName.equals(volume.name))
.findAny();
}

public void createVolume() throws MojoExecutionException {
Task.of(log, "docker", "volume", "create", volumeName).run();
Task.of(log, timeout, "docker", "volume", "create", volumeName).run();
}

public void removeVolume() throws MojoExecutionException {
Task.of(log, "docker", "volume", "rm", volumeName, "--force").run();
Task.of(log, timeout, "docker", "volume", "rm", volumeName, "--force").run();
}

// images
Expand All @@ -157,22 +156,22 @@ public String normalizeImage(String image) {
}

public Optional<ContainerImage> getImage(String image) throws MojoExecutionException {
var task = Task.of(log, "docker", "image", "inspect", image, "--format={{json .}}").start().waitFor();
var task = Task.of(log, timeout, "docker", "image", "inspect", image, "--format={{json .}}").start().waitFor();
return task.exitCode() == 0
? task.output().stream().map(output -> readValue(ContainerImage.class, output)).findAny()
: Optional.empty();
}

public void pullImage(String image, Duration timeout) throws MojoExecutionException {
Task.of(log, "docker", "image", "pull", "--quiet", image).timeout(timeout).run();
public void pullImage(String image, Duration pullTimeout) throws MojoExecutionException {
Task.of(log, pullTimeout, "docker", "image", "pull", "--quiet", image).run();
}

public void saveImage(String image, Path path) throws MojoExecutionException {
Task.of(log, "docker", "image", "save", "--output=" + path, image).run();
public void saveImage(String image, Path path, Duration saveTimeout) throws MojoExecutionException {
Task.of(log, saveTimeout, "docker", "image", "save", "--output=" + path, image).run();
}

public void removeImage(String image) throws MojoExecutionException {
Task.of(log, "docker", "image", "rm", "--force", image).run();
Task.of(log, timeout, "docker", "image", "rm", "--force", image).run();
}

// internal
Expand Down
22 changes: 7 additions & 15 deletions src/main/java/io/kokuwa/maven/k3s/util/Task.java
Original file line number Diff line number Diff line change
Expand Up @@ -24,40 +24,32 @@ public class Task {

private final Logger log;
private final List<String> command;
private Duration timeout = Duration.ofSeconds(30);
private Duration timeout;

private Process process;
private final List<String> output = new ArrayList<>();
private final List<Thread> threads = new ArrayList<>();

private Task(Logger log, List<String> command) {
private Task(Logger log, Duration timeout, List<String> command) {
this.log = log;
this.command = command;
this.timeout = timeout;
}

// config

public static Task of(Logger log, String... command) {
return new Task(log, List.of(command));
public static Task of(Logger log, Duration timeout, String... command) {
return new Task(log, timeout, List.of(command));
}

public static Task of(Logger log, List<String> command) {
return new Task(log, command);
public static Task of(Logger log, Duration timeout, List<String> command) {
return new Task(log, timeout, command);
}

public List<String> command() {
return command;
}

public Duration timeout() {
return timeout;
}

public Task timeout(Duration newTimeout) {
this.timeout = newTimeout;
return this;
}

@Override
public String toString() {
return command.stream().collect(Collectors.joining(" ")) + " (timeout: " + timeout + ")";
Expand Down
7 changes: 4 additions & 3 deletions src/test/java/io/kokuwa/maven/k3s/mojo/RunMojoTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import static org.junit.jupiter.api.Assertions.assertTrue;

import java.io.File;
import java.time.Duration;
import java.util.List;

import org.apache.maven.plugin.MojoExecutionException;
Expand Down Expand Up @@ -67,7 +68,7 @@ void withFailIfExistsStopped(RunMojo runMojo, Logger log) throws MojoExecutionEx
runMojo.setFailIfExists(true);
assertDoesNotThrow(runMojo::execute);
assertTrue(runMojo.getMarker().consumeStarted(), "started marker expected");
Task.of(log, "docker", "stop", "k3s-maven-plugin").run();
Task.of(log, Duration.ofSeconds(30), "docker", "stop", "k3s-maven-plugin").run();
var expectedMessage = "Container with id '" + docker.getContainer().get().id
+ "' found. Please remove that container or set 'k3s.failIfExists' to false.";
var actualMessage = assertThrows(MojoExecutionException.class, runMojo::execute).getMessage();
Expand Down Expand Up @@ -97,7 +98,7 @@ void withReplaceIfExistsStopped(RunMojo runMojo, Logger log) throws MojoExecutio
assertDoesNotThrow(runMojo::execute);
assertTrue(runMojo.getMarker().consumeStarted(), "started marker expected");
var containerBefore = docker.getContainer().orElseThrow();
Task.of(log, "docker", "stop", "k3s-maven-plugin").run();
Task.of(log, Duration.ofSeconds(30), "docker", "stop", "k3s-maven-plugin").run();
assertDoesNotThrow(runMojo::execute);
var containerAfter = docker.getContainer().orElseThrow();
assertNotEquals(containerBefore.id, containerAfter.id, "container was not replaced");
Expand Down Expand Up @@ -126,7 +127,7 @@ void withoutFailIfExistsStopped(RunMojo runMojo, Logger log, ApplyMojo applyMojo
assertDoesNotThrow(runMojo::execute);
assertTrue(runMojo.getMarker().consumeStarted(), "started marker expected");
var containerBefore = docker.getContainer().orElseThrow();
Task.of(log, "docker", "stop", "k3s-maven-plugin").run();
Task.of(log, Duration.ofSeconds(30), "docker", "stop", "k3s-maven-plugin").run();
assertDoesNotThrow(runMojo::execute);
assertTrue(runMojo.getMarker().consumeStarted(), "started marker expected");
assertDoesNotThrow(applyMojo::execute);
Expand Down
Loading