Make various UX improvements to the CLI (#187)

* Enable `--parallel` by default & consolidate its definition

* Add `plt add-repo` subcmd, and alias `require-repo` to `add-repo`

* Make `plt clone` cache required repos afterwards

* Cache all required repos after `[dev] plt add-repo` and `plt pull`

* Add `stage set-next-result` subcommand

CHANGELOG.md

@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+### Added
+
+- (cli) The `dev plt add-repo` subcommand now has additional aliases with clearer names: `require-repo` and `require-repositories`.
+- (cli) Added a `plt add-repo` subcommand which is just like `dev plt add-repo` (including its new `require-repo` and `require-repositories` aliases).
+- (cli) By default, now the `[dev] plt add-repo` subcommand will also cache all repos required by the pallet after adding/updating a repo requirement (or multiple repo requirements). This added behavior can be disabled with a new `--no-cache-req` flag.
+- (cli) By default, now the `plt clone` and `plt pull` subcommands will also cache all required repos after cloning/pulling the pallet. This added behavior can be disabled with a new `--no-cache-req` flag.
+- (cli) Added a `stage unset-next` subcommand which will update the stage store so that no staged pallet bundle will be applied next.
+- (cli) Now the `stage set-next` subcommand will accept an index of 0, which will update the stage store so that no staged pallet bundle will be applied next.
+- (cli) Added a `stage set-next-result` subcommand which can be used on non-Docker systems (where `forklift stage apply` doesn't work) to record whether the next staged pallet bundle to be applied has been successfully applied or has failed to be applied (or to reset its state from "failed" to "pending", representing that we don't know whether it has been applied successfully or unsuccessfully). This is intended to be used by systems which need to use the files exported by the next staged pallet bundle but might encounter unrecoverable errors.
+
+### Changed
+
+- (Breaking change; cli) The `--parallel` flag for various subcommands has now been consolidated and moved to the top level (e.g. `forklift --parallel plt cache-img` instead of `forklift plt cache-img --parallel`). Additionally, now the flag is enabled by default (because sequential downloading of images and bringup of Docker containers is so much slower than parallel downloading/bringup); to avoid parallel execution, use `--parallel=false` (e.g. `forklift --parallel=false plt cache-img`).
+- (Breaking change; cli) `plt clone` no longer deletes the `.git` directory after cloning a pallet, because the new pallet staging functionality makes it feasible to keep a long-running local pallet which can change independently of what is actually applied on a computer.
+
 ## 0.7.0-alpha.3 - 2024-04-13
 
 ### Fixed

README.md

@@ -74,7 +74,7 @@ If you aren't running Docker in rootless mode and your user isn't in a `docker`
 
   ```
   # Run now:
-  forklift pallet switch github.com/ethanjli/pallet-example-minimal@main
+  forklift pallet switch --no-cache-img github.com/ethanjli/pallet-example-minimal@main
   sudo -E forklift stage cache-img
   # Run when you want to apply the pallet:
   sudo -E forklift stage apply
@@ -109,7 +109,7 @@ cd /etc/
 /home/pi/forklift dev --cwd /home/pi/dev/pallet-example-minimal plt show
 ```
 
-You can also use the `forklift dev plt add-repo` command to add additional Forklift repositories to your development pallet, and/or to change the versions of Forklift repositories already added to your development pallet.
+You can also use the `forklift dev plt require-repo` command to require additional Forklift repositories for use in your development pallet, and/or to change the versions of Forklift repositories already required by your development pallet.
 
 You can also run commands like `forklift dev plt cache-all` and `forklift dev plt stage --no-cache-img` (with appropriate values in the `--cwd` flag if necessary) to download the Forklift repositories specified by your development pallet into your local cache and stage your development pallet to be applied with `sudo -E forklift stage apply`. This is useful if, for example, you want to make some experimental changes to your development pallet and test them on your local machine before committing and pushing those changes onto GitHub.
 
@@ -126,21 +126,21 @@ cd /home/pi/
 
 The following projects solve related problems with containers for application software, though they make different trade-offs compared to Forklift:
 
-- poco enables Git-based management of Docker Compose projects and collections (*catalogs*) of projects and repositories and provides some similar functionalities to forklift: <https://github.com/shiwaforce/poco>
-- Terraform (an inspiration for this project) has a Docker Provider which enables declarative management of Docker hosts and Docker Swarms from a Terraform configuration: <https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs>
-- swarm-pack (an inspiration for this project) uses collections of packages from user-specified Git repositories and enables templated configuration of Docker Compose files, with imperative deployments of packages to a Docker Swarm: <https://github.com/swarm-pack/swarm-pack>
-- SwarmManagement uses a single YAML file for declarative configuration of an entire Docker Swarm: <https://github.com/hansehe/SwarmManagement>
-- Podman Quadlets enable management of containers, volumes, and networks using declarative systemd units: <https://github.com/containers/podlet> & <https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html>
-- FetchIt enables Git-based management of containers in Podman: <https://github.com/containers/fetchit>
-- Projects developing GitOps tools such as ArgoCD, Flux, etc., store container environment configurations as Git repositories but are generally designed for Kubernetes: <https://www.gitops.tech/>
+- [poco](https://github.com/shiwaforce/poco) enables Git-based management of Docker Compose projects and collections (*catalogs*) of projects and repositories and provides some similar functionalities to forklift
+- [Terraform](https://registry.terraform.io/providers/kreuzwerker/docker/latest/docs) (an early inspiration for this project) has a Docker Provider which enables declarative management of Docker hosts and Docker Swarms from a Terraform configuration
+- [swarm-pack](https://github.com/swarm-pack/swarm-pack) (an early inspiration for this project) uses collections of packages from user-specified Git repositories and enables templated configuration of Docker Compose files, with imperative deployments of packages to a Docker Swarm
+- [SwarmManagement](https://github.com/hansehe/SwarmManagement) uses a single YAML file for declarative configuration of an entire Docker Swarm
+- Podman [Quadlets](https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html) enable management of containers, volumes, and networks using declarative systemd units
+- [FetchIt](https://github.com/containers/fetchit) enables Git-based management of containers in Podman
+- Projects developing [GitOps](https://www.gitops.tech/) tools such as ArgoCD, Flux, etc., store container environment configurations as Git repositories but are generally designed for Kubernetes
 
 The following projects solve related problems in the base OS, though they make different trade-offs compared to Forklift (especially because of the PlanktoScope project's legacy software):
 
-- systemd-sysext and systemd-confext provide a more structured/constrained way (compared to Forklift) to atomically overlay system files onto the base OS; however, Forklift can also be used as a way to deploy sysexts/confexts onto an OS (see [this demo](https://github.com/ethanjli/ublue-forklift-sysext-demo?tab=readme-ov-file#explanation)): <https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html>
-- systemd's Portable Services pattern and `portablectl` tool provide a more structured/constrained/sandboxed way (compared to Forklift) to atomically add system services: <https://systemd.io/PORTABLE_SERVICES/>
-- ostree enables atomic updates of the base OS, but [it is not supported by Raspberry Pi OS](https://github.com/ostreedev/ostree/issues/2223): <https://ostreedev.github.io/ostree/>
-- The bootc project enables the entire operating system to be delivered as a bootable OCI container image, but currently it relies on bootupd, which [currently only works on RPM-based distros](https://github.com/coreos/bootupd/issues/468): <https://containers.github.io/bootc/>
-- gokrazy enables atomic deployment of Go programs (and also of software containers!), but it has a very different architecture compared to traditional Linux distros: <https://gokrazy.org/>
+- [systemd-sysext and systemd-confext](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html) provide a more structured/constrained way (compared to Forklift) to atomically overlay system files onto the base OS; however, Forklift can also be used as a way to deploy sysexts/confexts onto an OS (see [this demo](https://github.com/ethanjli/ublue-forklift-sysext-demo?tab=readme-ov-file#explanation))
+- systemd's [Portable Services](https://systemd.io/PORTABLE_SERVICES/) pattern and `portablectl` tool provide a more structured/constrained/sandboxed way (compared to Forklift) to atomically add system services
+- [ostree](https://ostreedev.github.io/ostree/) enables atomic updates of the base OS, but [it is not supported by Raspberry Pi OS](https://github.com/ostreedev/ostree/issues/2223)
+- The [bootc](https://containers.github.io/bootc/) project enables the entire operating system to be delivered as a bootable OCI container image, but currently it relies on bootupd, which [currently only works on RPM-based distros](https://github.com/coreos/bootupd/issues/468)
+- [gokrazy](https://gokrazy.org/) enables atomic deployment of Go programs (and also of software containers!), but it has a very different architecture compared to traditional Linux distros
 
 Other related OS-level projects can be found at [github.com/castrojo/awesome-immutable](https://github.com/castrojo/awesome-immutable).
 

cmd/forklift/cache/git-repositories.go

@@ -94,7 +94,7 @@ func rmGitRepoAction[Cache remover](
 			return err
 		}
 
-		fmt.Printf("Clearing %s cache...", gitRepoType)
+		fmt.Printf("Clearing %s cache...\n", gitRepoType)
 		if err = cache.Remove(); err != nil {
 			return errors.Wrapf(err, "couldn't clear %s cache", gitRepoType)
 		}

cmd/forklift/dev/plt/cli.go

@@ -53,12 +53,6 @@ func makeUseSubcmds(versions Versions) []*cli.Command {
 			Usage: "Determines the changes needed to update the host to match the deployments " +
 				"specified by the local pallet",
 			Action: planAction(versions),
-			Flags: []cli.Flag{
-				&cli.BoolFlag{
-					Name:  "parallel",
-					Usage: "construct a plan for parallel updating of deployments",
-				},
-			},
 		},
 		&cli.Command{
 			Name:     "stage",
@@ -68,11 +62,7 @@ func makeUseSubcmds(versions Versions) []*cli.Command {
 			Flags: []cli.Flag{
 				&cli.BoolFlag{
 					Name:  "no-cache-img",
-					Usage: "don't download container images",
-				},
-				&cli.BoolFlag{
-					Name:  "parallel",
-					Usage: "parallelize downloading of container images",
+					Usage: "Don't download container images",
 				},
 			},
 		},
@@ -82,12 +72,6 @@ func makeUseSubcmds(versions Versions) []*cli.Command {
 			Usage: "Builds, stages, and immediately applies a bundle of the development pallet to " +
 				"update the host to match the deployments specified by the development pallet",
 			Action: applyAction(versions),
-			Flags: []cli.Flag{
-				&cli.BoolFlag{
-					Name:  "parallel",
-					Usage: "parallelize updating of package deployments",
-				},
-			},
 		},
 	)
 }
@@ -103,11 +87,7 @@ func makeUseCacheSubcmds(versions Versions) []*cli.Command {
 			Flags: []cli.Flag{
 				&cli.BoolFlag{
 					Name:  "include-disabled",
-					Usage: "also cache things needed for disabled package deployments",
-				},
-				&cli.BoolFlag{
-					Name:  "parallel",
-					Usage: "parallelize downloading of container images",
+					Usage: "Also cache things needed for disabled package deployments",
 				},
 			},
 		},
@@ -127,11 +107,7 @@ func makeUseCacheSubcmds(versions Versions) []*cli.Command {
 			Flags: []cli.Flag{
 				&cli.BoolFlag{
 					Name:  "include-disabled",
-					Usage: "also download images for disabled package deployments",
-				},
-				&cli.BoolFlag{
-					Name:  "parallel",
-					Usage: "parallelize downloading of container images",
+					Usage: "Also download images for disabled package deployments",
 				},
 			},
 		},
@@ -210,7 +186,7 @@ func makeQueryDeplSubcmds(category string) []*cli.Command {
 			Flags: []cli.Flag{
 				&cli.BoolFlag{
 					Name:  "allow-disabled",
-					Usage: "locates the package even if the specified deployment is disabled",
+					Usage: "Locates the package even if the specified deployment is disabled",
 				},
 			},
 		},
@@ -221,15 +197,26 @@ func makeModifySubcmds(versions Versions) []*cli.Command {
 	const category = "Modify the pallet"
 	return []*cli.Command{
 		{
-			Name:      "add-repo",
-			Aliases:   []string{"add-repositories"},
-			Category:  category,
-			Usage:     "Adds repos to the pallet, tracking specified versions or branches",
+			Name:     "add-repo",
+			Aliases:  []string{"add-repositories", "require-repo", "require-repositories"},
+			Category: category,
+			Usage: "Adds (or re-adds) repo requirements to the pallet, tracking specified versions " +
+				"or branches",
 			ArgsUsage: "[repo_path@version_query]...",
-			Action:    addRepoAction(versions),
+			Flags: []cli.Flag{
+				&cli.BoolFlag{
+					Name: "no-cache-req",
+					Usage: "Don't download repositories and pallets required by this pallet after adding " +
+						"the repo",
+				},
+			},
+			Action: addRepoAction(versions),
 		},
-		// TODO: add an rm-repo action
-		// TODO: add an add-depl action
+		// TODO: add an rm-repo action with alias "drop-repo"; it should ensure no depls depend on it
+		// or delete those depls if `--force` is set
+		// TODO: add an add-depl --features=... depl_path package_path action
 		// TODO: add an rm-depl action
+		// TODO: add an add-depl-feat depl_path [feature]... action
+		// TODO: add an rm-depl-feat depl_path [feature]... action
 	}
 }

cmd/forklift/dev/plt/images.go

@@ -30,7 +30,7 @@ func cacheImgAction(versions Versions) cli.ActionFunc {
 			return err
 		}
 		fmt.Println()
-		fmt.Println("Done! Next, you'll probably want to run `sudo -E forklift dev plt apply`.")
+		fmt.Println("Done!")
 		return nil
 	}
 }

cmd/forklift/dev/plt/pallets.go

@@ -134,7 +134,7 @@ func cacheAllAction(versions Versions) cli.ActionFunc {
 			fmt.Println("Done! No further actions are needed at this time.")
 			return nil
 		}
-		fmt.Println("Done! Next, you might want to run `forklift dev plt stage`.")
+		fmt.Println("Done!")
 		return nil
 	}
 }
@@ -225,7 +225,10 @@ func stageAction(versions Versions) cli.ActionFunc {
 		); err != nil {
 			return err
 		}
-		fmt.Println("Done! To apply the staged pallet immediately, run `sudo -E forklift stage apply`.")
+		fmt.Println(
+			"Done! To apply the staged pallet, you may need to reboot or run " +
+				"`forklift stage apply` (or `sudo -E forklift stage apply` if you need sudo for Docker).",
+		)
 		return nil
 	}
 }

cmd/forklift/dev/plt/repositories.go

@@ -2,15 +2,9 @@ package plt
 
 import (
 	"fmt"
-	"os"
-	"path"
-	"path/filepath"
 
-	"github.com/pkg/errors"
 	"github.com/urfave/cli/v2"
-	"gopkg.in/yaml.v3"
 
-	"github.com/PlanktoScope/forklift/internal/app/forklift"
 	fcli "github.com/PlanktoScope/forklift/internal/app/forklift/cli"
 )
 
@@ -41,7 +35,7 @@ func cacheRepoAction(versions Versions) cli.ActionFunc {
 
 		// TODO: warn if any downloaded repo doesn't appear to be an actual repo, or if any repo's
 		// forklift version is incompatible or ahead of the pallet version
-		fmt.Println("Done! Next, you might want to run `sudo -E forklift dev plt apply`.")
+		fmt.Println("Done!")
 		return nil
 	}
 }
@@ -64,60 +58,34 @@ func showRepoAction(c *cli.Context) error {
 		return err
 	}
 
-	repoPath := c.Args().First()
-	return fcli.PrintRepoInfo(0, pallet, cache, repoPath)
+	return fcli.PrintRepoInfo(0, pallet, cache, c.Args().First())
 }
 
 // add-repo
 
 func addRepoAction(versions Versions) cli.ActionFunc {
 	return func(c *cli.Context) error {
-		pallet, cache, err := processFullBaseArgs(c, false, false)
+		pallet, repoCache, err := processFullBaseArgs(c, false, false)
 		if err != nil {
 			return err
 		}
 		if err = fcli.CheckShallowCompatibility(
-			pallet, cache, versions.Tool, versions.MinSupportedRepo, versions.MinSupportedPallet,
+			pallet, repoCache, versions.Tool, versions.MinSupportedRepo, versions.MinSupportedPallet,
 			c.Bool("ignore-tool-version"),
 		); err != nil {
 			return err
 		}
 
-		repoQueries := c.Args().Slice()
-		if err = fcli.ValidateGitRepoQueries(repoQueries); err != nil {
-			return errors.Wrap(err, "one or more arguments is invalid")
-		}
-		resolved, err := fcli.ResolveQueriesUsingLocalMirrors(0, cache.Underlay.Path(), repoQueries)
-		if err != nil {
+		if err = fcli.AddRepoRequirements(
+			0, pallet, repoCache.Underlay.Path(), c.Args().Slice(),
+		); err != nil {
 			return err
 		}
-		fmt.Println()
-		fmt.Printf("Saving configurations to %s...\n", pallet.FS.Path())
-		for _, repoQuery := range repoQueries {
-			req, ok := resolved[repoQuery]
-			if !ok {
-				return errors.Errorf("couldn't find configuration for %s", repoQuery)
-			}
-			reqsReposFS, err := pallet.GetRepoReqsFS()
-			if err != nil {
+
+		if !c.Bool("no-cache-req") {
+			if _, err = fcli.CacheStagingRequirements(pallet, repoCache.Path()); err != nil {
 				return err
 			}
-			repoReqPath := path.Join(reqsReposFS.Path(), req.Path(), forklift.VersionLockDefFile)
-			marshaled, err := yaml.Marshal(req.VersionLock.Def)
-			if err != nil {
-				return errors.Wrapf(err, "couldn't marshal repo requirement from %s", repoReqPath)
-			}
-			if err := forklift.EnsureExists(filepath.FromSlash(path.Dir(repoReqPath))); err != nil {
-				return errors.Wrapf(
-					err, "couldn't make directory %s", filepath.FromSlash(path.Dir(repoReqPath)),
-				)
-			}
-			const perm = 0o644 // owner rw, group r, public r
-			if err := os.WriteFile(filepath.FromSlash(repoReqPath), marshaled, perm); err != nil {
-				return errors.Wrapf(
-					err, "couldn't save repo requirement to %s", filepath.FromSlash(repoReqPath),
-				)
-			}
 		}
 		fmt.Println("Done!")
 		return nil

cmd/forklift/main.go

@@ -73,6 +73,13 @@ var app = &cli.App{
 			Usage:   "Ignore the version of the forklift tool in version compatibility checks",
 			EnvVars: []string{"FORKLIFT_IGNORE_TOOL_VERSION"},
 		},
+		&cli.BoolFlag{
+			Name:  "parallel",
+			Value: true,
+			Usage: "Allow parallel execution of I/O-bound tasks, such as downloading container images " +
+				"or starting containers",
+			EnvVars: []string{"FORKLIFT_PARALLEL"},
+		},
 	},
 	Suggest: true,
 }