Introduce zarf tools registry prune (#1966)

## Description This introduces a new Zarf tools command to prune images in a registry that do not relate to any currently deployed Zarf components. ## Related Issue Fixes #1960 Fixes #1945 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds functionality) - [ ] Other (security config, docs update, etc) ## Checklist before merging - [x] Test, docs, adr added or updated as needed - [X] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow) followed
zarf-dev · Aug 11, 2023 · b5cf624 · b5cf624
1 parent 314e07c
commit b5cf624
Show file tree

Hide file tree

Showing 11 changed files with 327 additions and 16 deletions.
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry.md
@@ -18,7 +18,10 @@ Tools for working with container registries using go-containertools
 * [zarf tools](zarf_tools.md)	 - Collection of additional tools to make airgap easier
 * [zarf tools registry catalog](zarf_tools_registry_catalog.md)	 - List the repos in a registry
 * [zarf tools registry copy](zarf_tools_registry_copy.md)	 - Efficiently copy a remote image from src to dst while retaining the digest value
+* [zarf tools registry delete](zarf_tools_registry_delete.md)	 - Delete an image reference from its registry
+* [zarf tools registry digest](zarf_tools_registry_digest.md)	 - Get the digest of an image
 * [zarf tools registry login](zarf_tools_registry_login.md)	 - Log in to a registry
 * [zarf tools registry ls](zarf_tools_registry_ls.md)	 - List the tags in a repo
+* [zarf tools registry prune](zarf_tools_registry_prune.md)	 - Prunes images from the registry that are not currently being used by any Zarf packages.
 * [zarf tools registry pull](zarf_tools_registry_pull.md)	 - Pull remote images by reference and store their contents locally
 * [zarf tools registry push](zarf_tools_registry_push.md)	 - Push local image contents to a remote registry
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry_delete.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry_delete.md
@@ -0,0 +1,39 @@
+# zarf tools registry delete
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+Delete an image reference from its registry
+
+```
+zarf tools registry delete IMAGE [flags]
+```
+
+## Examples
+
+```
+
+# delete an image digest from an internal repo in Zarf
+$ zarf tools registry delete 127.0.0.1:31999/stefanprodan/podinfo@sha256:57a654ace69ec02ba8973093b6a786faa15640575fbf0dbb603db55aca2ccec8
+
+# delete an image digest from a repo hosted at reg.example.com
+$ zarf tools registry delete reg.example.com/stefanprodan/podinfo@sha256:57a654ace69ec02ba8973093b6a786faa15640575fbf0dbb603db55aca2ccec8
+
+```
+
+## Options
+
+```
+  -h, --help   help for delete
+```
+
+## Options inherited from parent commands
+
+```
+      --allow-nondistributable-artifacts   Allow pushing non-distributable (foreign) layers
+      --insecure                           Allow image references to be fetched without TLS
+      --platform string                    Specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). (default "all")
+  -v, --verbose                            Enable debug logs
+```
+
+## SEE ALSO
+
+* [zarf tools registry](zarf_tools_registry.md)	 - Tools for working with container registries using go-containertools
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry_digest.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry_digest.md
@@ -0,0 +1,41 @@
+# zarf tools registry digest
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+Get the digest of an image
+
+```
+zarf tools registry digest IMAGE [flags]
+```
+
+## Examples
+
+```
+
+# return an image digest for an internal repo in Zarf
+$ zarf tools registry digest 127.0.0.1:31999/stefanprodan/podinfo:6.4.0
+
+# return an image digest from a repo hosted at reg.example.com
+$ zarf tools registry digest reg.example.com/stefanprodan/podinfo:6.4.0
+
+```
+
+## Options
+
+```
+      --full-ref         (Optional) if true, print the full image reference by digest
+  -h, --help             help for digest
+      --tarball string   (Optional) path to tarball containing the image
+```
+
+## Options inherited from parent commands
+
+```
+      --allow-nondistributable-artifacts   Allow pushing non-distributable (foreign) layers
+      --insecure                           Allow image references to be fetched without TLS
+      --platform string                    Specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). (default "all")
+  -v, --verbose                            Enable debug logs
+```
+
+## SEE ALSO
+
+* [zarf tools registry](zarf_tools_registry.md)	 - Tools for working with container registries using go-containertools
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry_prune.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_tools_registry_prune.md
@@ -0,0 +1,28 @@
+# zarf tools registry prune
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+Prunes images from the registry that are not currently being used by any Zarf packages.
+
+```
+zarf tools registry prune [flags]
+```
+
+## Options
+
+```
+      --confirm   Confirm the image prune action to prevent accidental deletions
+  -h, --help      help for prune
+```
+
+## Options inherited from parent commands
+
+```
+      --allow-nondistributable-artifacts   Allow pushing non-distributable (foreign) layers
+      --insecure                           Allow image references to be fetched without TLS
+      --platform string                    Specifies the platform in the form os/arch[/variant][:osversion] (e.g. linux/amd64). (default "all")
+  -v, --verbose                            Enable debug logs
+```
+
+## SEE ALSO
+
+* [zarf tools registry](zarf_tools_registry.md)	 - Tools for working with container registries using go-containertools
diff --git a/packages/zarf-registry/chart/values.yaml b/packages/zarf-registry/chart/values.yaml
@@ -17,6 +17,7 @@ persistence:
   accessMode: "ReadWriteOnce"
   enabled: true
   size: 20Gi
+  deleteEnabled: true
 
 secrets:
   htpasswd: ""

diff --git a/src/cmd/tools/crane.go b/src/cmd/tools/crane.go
@@ -9,10 +9,12 @@ import (
 	"os"
 	"strings"
 
+	"github.com/AlecAivazis/survey/v2"
 	"github.com/defenseunicorns/zarf/src/config"
 	"github.com/defenseunicorns/zarf/src/config/lang"
 	"github.com/defenseunicorns/zarf/src/internal/cluster"
 	"github.com/defenseunicorns/zarf/src/pkg/message"
+	"github.com/defenseunicorns/zarf/src/pkg/transform"
 	"github.com/defenseunicorns/zarf/src/pkg/utils/exec"
 	craneCmd "github.com/google/go-containerregistry/cmd/crane/cmd"
 	"github.com/google/go-containerregistry/pkg/crane"
@@ -64,6 +66,16 @@ func init() {
 		},
 	}
 
+	pruneCmd := &cobra.Command{
+		Use:     "prune",
+		Aliases: []string{"p"},
+		Short:   lang.CmdToolsRegistryPruneShort,
+		RunE:    pruneImages,
+	}
+
+	// Always require confirm flag (no viper)
+	pruneCmd.Flags().BoolVar(&config.CommonOptions.Confirm, "confirm", false, lang.CmdToolsRegistryPruneFlagConfirm)
+
 	craneLogin := craneCmd.NewCmdAuthLogin()
 	craneLogin.Example = ""
 
@@ -76,6 +88,9 @@ func init() {
 	registryCmd.AddCommand(zarfCraneInternalWrapper(craneCmd.NewCmdList, &craneOptions, lang.CmdToolsRegistryListExample, 0))
 	registryCmd.AddCommand(zarfCraneInternalWrapper(craneCmd.NewCmdPush, &craneOptions, lang.CmdToolsRegistryPushExample, 1))
 	registryCmd.AddCommand(zarfCraneInternalWrapper(craneCmd.NewCmdPull, &craneOptions, lang.CmdToolsRegistryPullExample, 0))
+	registryCmd.AddCommand(zarfCraneInternalWrapper(craneCmd.NewCmdDelete, &craneOptions, lang.CmdToolsRegistryDeleteExample, 0))
+	registryCmd.AddCommand(zarfCraneInternalWrapper(craneCmd.NewCmdDigest, &craneOptions, lang.CmdToolsRegistryDigestExample, 0))
+	registryCmd.AddCommand(pruneCmd)
 
 	registryCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, lang.CmdToolsRegistryFlagVerbose)
 	registryCmd.PersistentFlags().BoolVar(&insecure, "insecure", false, lang.CmdToolsRegistryFlagInsecure)
@@ -175,11 +190,144 @@ func zarfCraneInternalWrapper(commandToWrap func(*[]crane.Option) *cobra.Command
 		}
 
 		// Add the correct authentication to the crane command options
-		authOption := config.GetCraneAuthOption(zarfState.RegistryInfo.PullUsername, zarfState.RegistryInfo.PullPassword)
+		authOption := config.GetCraneAuthOption(zarfState.RegistryInfo.PushUsername, zarfState.RegistryInfo.PushPassword)
 		*cranePlatformOptions = append(*cranePlatformOptions, authOption)
 
 		return originalListFn(cmd, args)
 	}
 
 	return wrappedCommand
 }
+
+func pruneImages(_ *cobra.Command, _ []string) error {
+	// Try to connect to a Zarf initialized cluster
+	zarfCluster, err := cluster.NewCluster()
+	if err != nil {
+		return err
+	}
+
+	// Load the state
+	zarfState, err := zarfCluster.LoadZarfState()
+	if err != nil {
+		return err
+	}
+
+	// Load the currently deployed packages
+	zarfPackages, errs := zarfCluster.GetDeployedZarfPackages()
+	if len(errs) > 0 {
+		return lang.ErrUnableToGetPackages
+	}
+
+	// Set up a tunnel to the registry if applicable
+	registryAddress := zarfState.RegistryInfo.Address
+	if zarfState.RegistryInfo.InternalRegistry {
+		// Open a tunnel to the Zarf registry
+		tunnelReg, err := cluster.NewZarfTunnel()
+		if err != nil {
+			return err
+		}
+		err = tunnelReg.Connect(cluster.ZarfRegistry, false)
+		if err != nil {
+			return err
+		}
+		registryAddress = tunnelReg.Endpoint()
+	}
+
+	authOption := config.GetCraneAuthOption(zarfState.RegistryInfo.PushUsername, zarfState.RegistryInfo.PushPassword)
+
+	// Determine which image digests are currently used by Zarf packages
+	pkgImages := map[string]bool{}
+	for _, pkg := range zarfPackages {
+		deployedComponents := map[string]bool{}
+		for _, depComponent := range pkg.DeployedComponents {
+			deployedComponents[depComponent.Name] = true
+		}
+
+		for _, component := range pkg.Data.Components {
+			if _, ok := deployedComponents[component.Name]; ok {
+				for _, image := range component.Images {
+					// We use the no checksum image since it will always exist and will share the same digest with other tags
+					transformedImageNoCheck, err := transform.ImageTransformHostWithoutChecksum(registryAddress, image)
+					if err != nil {
+						return err
+					}
+
+					digest, err := crane.Digest(transformedImageNoCheck, authOption)
+					if err != nil {
+						return err
+					}
+					pkgImages[digest] = true
+				}
+			}
+		}
+	}
+
+	// Find which images and tags are in the registry currently
+	imageCatalog, err := crane.Catalog(registryAddress, authOption)
+	if err != nil {
+		return err
+	}
+	imageRefToDigest := map[string]string{}
+	for _, image := range imageCatalog {
+		imageRef := fmt.Sprintf("%s/%s", registryAddress, image)
+		tags, err := crane.ListTags(imageRef, authOption)
+		if err != nil {
+			return err
+		}
+		for _, tag := range tags {
+			taggedImageRef := fmt.Sprintf("%s:%s", imageRef, tag)
+			digest, err := crane.Digest(taggedImageRef, authOption)
+			if err != nil {
+				return err
+			}
+			imageRefToDigest[taggedImageRef] = digest
+		}
+	}
+
+	// Figure out which images are in the registry but not needed by packages
+	imageDigestsToPrune := map[string]bool{}
+	for imageRef, digest := range imageRefToDigest {
+		if _, ok := pkgImages[digest]; !ok {
+			ref, err := transform.ParseImageRef(imageRef)
+			if err != nil {
+				return err
+			}
+			imageRef = fmt.Sprintf("%s@%s", ref.Name, digest)
+			imageDigestsToPrune[imageRef] = true
+		}
+	}
+
+	if len(imageDigestsToPrune) > 0 {
+		message.Note(lang.CmdToolsRegistryPruneImageList)
+
+		for imageRef := range imageDigestsToPrune {
+			message.Info(imageRef)
+		}
+
+		confirm := config.CommonOptions.Confirm
+
+		if confirm {
+			message.Note(lang.CmdConfirmProvided)
+		} else {
+			prompt := &survey.Confirm{
+				Message: lang.CmdConfirmContinue,
+			}
+			if err := survey.AskOne(prompt, &confirm); err != nil {
+				message.Fatalf(nil, lang.ErrConfirmCancel, err)
+			}
+		}
+		if confirm {
+			// Delete the image references that are to be pruned
+			for imageRef := range imageDigestsToPrune {
+				err = crane.Delete(imageRef, authOption)
+				if err != nil {
+					return err
+				}
+			}
+		}
+	} else {
+		message.Note(lang.CmdToolsRegistryPruneNoImages)
+	}
+
+	return nil
+}
diff --git a/src/config/lang/english.go b/src/config/lang/english.go
@@ -27,10 +27,15 @@ const (
 	ErrCreatingDir         = "failed to create directory %s: %s"
 	ErrRemoveFile          = "failed to remove file %s: %s"
 	ErrUnarchive           = "failed to unarchive %s: %s"
+	ErrConfirmCancel       = "confirm selection canceled: %s"
 )
 
 // Zarf CLI commands.
 const (
+	// common command language
+	CmdConfirmProvided = "Confirm flag specified, continuing without prompting."
+	CmdConfirmContinue = "Continue with these changes?"
+
 	// root zarf command
 	RootCmdShort = "DevSecOps for Airgap"
 	RootCmdLong  = "Zarf eliminates the complexity of air gap software delivery for Kubernetes clusters and cloud native workloads\n" +
@@ -358,6 +363,27 @@ const (
 	$ zarf tools registry pull reg.example.com/stefanprodan/podinfo:6.4.0 image.tar
 `
 
+	CmdToolsRegistryDeleteExample = `
+# delete an image digest from an internal repo in Zarf
+$ zarf tools registry delete 127.0.0.1:31999/stefanprodan/podinfo@sha256:57a654ace69ec02ba8973093b6a786faa15640575fbf0dbb603db55aca2ccec8
+
+# delete an image digest from a repo hosted at reg.example.com
+$ zarf tools registry delete reg.example.com/stefanprodan/podinfo@sha256:57a654ace69ec02ba8973093b6a786faa15640575fbf0dbb603db55aca2ccec8
+`
+
+	CmdToolsRegistryDigestExample = `
+# return an image digest for an internal repo in Zarf
+$ zarf tools registry digest 127.0.0.1:31999/stefanprodan/podinfo:6.4.0
+
+# return an image digest from a repo hosted at reg.example.com
+$ zarf tools registry digest reg.example.com/stefanprodan/podinfo:6.4.0
+`
+
+	CmdToolsRegistryPruneShort       = "Prunes images from the registry that are not currently being used by any Zarf packages."
+	CmdToolsRegistryPruneFlagConfirm = "Confirm the image prune action to prevent accidental deletions"
+	CmdToolsRegistryPruneImageList   = "The following image digests will be pruned from the registry:"
+	CmdToolsRegistryPruneNoImages    = "There are no images to prune"
+
 	CmdToolsRegistryInvalidPlatformErr = "Invalid platform '%s': %s"
 	CmdToolsRegistryFlagVerbose        = "Enable debug logs"
 	CmdToolsRegistryFlagInsecure       = "Allow image references to be fetched without TLS"
@@ -514,9 +540,10 @@ const (
 
 // Collection of reusable error messages.
 var (
-	ErrInitNotFound      = errors.New("this command requires a zarf-init package, but one was not found on the local system. Re-run the last command again without '--confirm' to download the package")
-	ErrUnableToCheckArch = errors.New("unable to get the configured cluster's architecture")
-	ErrInterrupt         = errors.New("execution cancelled due to an interrupt")
+	ErrInitNotFound        = errors.New("this command requires a zarf-init package, but one was not found on the local system. Re-run the last command again without '--confirm' to download the package")
+	ErrUnableToCheckArch   = errors.New("unable to get the configured cluster's architecture")
+	ErrInterrupt           = errors.New("execution cancelled due to an interrupt")
+	ErrUnableToGetPackages = errors.New("unable to load the Zarf Package data from the cluster")
 )
 
 // Collection of reusable warn messages.