feat: add zarf prepare lint to perform schema validation (#2075)

## Description Intent of this PR is to introduce the command zarf prepare lint, with the ability to validate the zarf schema ## Related Issue Relates to #2064 #1667 ## Type of change - [X] New feature (non-breaking change which adds functionality) ## Checklist before merging - [ ] Test, docs, adr added or updated as needed - [ ] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow) followed --------- Co-authored-by: Barry Waldbaum <[email protected]> Co-authored-by: Wayne Starr <[email protected]> Co-authored-by: Lucas Rodriguez <[email protected]> Co-authored-by: razzle <[email protected]>
zarf-dev · Nov 29, 2023 · 587c695 · 587c695
1 parent 9cf4482
commit 587c695
Show file tree

Hide file tree

Showing 25 changed files with 492 additions and 46 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -47,3 +47,7 @@ repos:
         files: "zarf.yaml"
         types: [yaml]
         args: ["--schemafile", "zarf.schema.json"]
+        exclude: |
+          (?x)^(
+            src/test/packages/12-lint/.*
+          )$
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_prepare.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_prepare.md
@@ -27,6 +27,7 @@ Tools to help prepare assets for packaging
 * [zarf](zarf.md)	 - DevSecOps for Airgap
 * [zarf prepare find-images](zarf_prepare_find-images.md)	 - Evaluates components in a zarf file to identify images specified in their helm charts and manifests
 * [zarf prepare generate-config](zarf_prepare_generate-config.md)	 - Generates a config file for Zarf
+* [zarf prepare lint](zarf_prepare_lint.md)	 - Verifies the package schema
 * [zarf prepare patch-git](zarf_prepare_patch-git.md)	 - Converts all .git URLs to the specified Zarf HOST and with the Zarf URL pattern in a given FILE.  NOTE:
 This should only be used for manifests that are not mutated by the Zarf Agent Mutating Webhook.
 * [zarf prepare sha256sum](zarf_prepare_sha256sum.md)	 - Generates a SHA256SUM for the given file
diff --git a/docs/2-the-zarf-cli/100-cli-commands/zarf_prepare_lint.md b/docs/2-the-zarf-cli/100-cli-commands/zarf_prepare_lint.md
@@ -0,0 +1,35 @@
+# zarf prepare lint
+<!-- Auto-generated by hack/gen-cli-docs.sh -->
+
+Verifies the package schema
+
+## Synopsis
+
+Verifies the package schema and warns the user if they have variables that won't be evaluated
+
+```
+zarf prepare lint [ DIRECTORY ] [flags]
+```
+
+## Options
+
+```
+  -h, --help   help for lint
+```
+
+## Options inherited from parent commands
+
+```
+  -a, --architecture string   Architecture for OCI images and Zarf packages
+      --insecure              Allow access to insecure registries and disable other recommended security enforcements such as package checksum and signature validation. This flag should only be used if you have a specific reason and accept the reduced security posture.
+  -l, --log-level string      Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info")
+      --no-color              Disable colors in output
+      --no-log-file           Disable log file creation
+      --no-progress           Disable fancy UI progress bars, spinners, logos, etc
+      --tmpdir string         Specify the temporary directory to use for intermediate files
+      --zarf-cache string     Specify the location of the Zarf cache directory (default "~/.zarf-cache")
+```
+
+## SEE ALSO
+
+* [zarf prepare](zarf_prepare.md)	 - Tools to help prepare assets for packaging
diff --git a/docs/3-create-a-zarf-package/4-zarf-schema.md b/docs/3-create-a-zarf-package/4-zarf-schema.md
@@ -783,10 +783,6 @@ Must be one of:
 | -------- | -------- |
 | **Type** | `string` |
 
-| Restrictions                      |                                                                                                                         |
-| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
-| **Must match regular expression** | ```^(?!.*###ZARF_PKG_TMPL_).*$``` [Test](https://regex101.com/?regex=%5E%28%3F%21.%2A%23%23%23ZARF_PKG_TMPL_%29.%2A%24) |
-
 </blockquote>
 </details>
 
@@ -803,9 +799,9 @@ Must be one of:
 | -------- | -------- |
 | **Type** | `string` |
 
-| Restrictions                      |                                                                                                                                           |
-| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
-| **Must match regular expression** | ```^oci://(?!.*###ZARF_PKG_TMPL_).*$``` [Test](https://regex101.com/?regex=%5Eoci%3A%2F%2F%28%3F%21.%2A%23%23%23ZARF_PKG_TMPL_%29.%2A%24) |
+| Restrictions                      |                                                                             |
+| --------------------------------- | --------------------------------------------------------------------------- |
+| **Must match regular expression** | ```^oci://.*$``` [Test](https://regex101.com/?regex=%5Eoci%3A%2F%2F.%2A%24) |
 
 </blockquote>
 </details>

diff --git a/docs/3-create-a-zarf-package/index.md b/docs/3-create-a-zarf-package/index.md
@@ -21,6 +21,9 @@ To learn more about creating a Zarf package, you can check out the following res
 The general flow of a Zarf package deployment on an existing initialized cluster is as follows:
 
 ```shell
+# Before creating your package you can lint your zarf.yaml
+$ zarf prepare lint <directory>
+
 # To create a package run the following:
 $ zarf package create <directory>
 # - Enter any package templates that have not yet been defined

diff --git a/docs/5-zarf-tutorials/0-creating-a-zarf-package.md b/docs/5-zarf-tutorials/0-creating-a-zarf-package.md
@@ -37,6 +37,7 @@ metadata:
 :::tip
 
 If you are using an Integrated Development Environment (such as [VS Code](../3-create-a-zarf-package/8-vscode.md)) to create and edit the `zarf.yaml` file, you can install or reference the [`zarf.schema.json`](https://github.com/defenseunicorns/zarf/blob/main/zarf.schema.json) file to get error checking and autocomplete.
+Additionally, you can run `zarf prepare lint <directory>` to validate aginst the [`zarf.schema.json`](https://github.com/defenseunicorns/zarf/blob/main/zarf.schema.json)
 
 :::
 

diff --git a/go.mod b/go.mod
@@ -1,8 +1,10 @@
 module github.com/defenseunicorns/zarf
 
-go 1.21.0
+go 1.21.1
 
-toolchain go1.21.3
+toolchain go1.21.4
+
+replace github.com/xeipuuv/gojsonschema => github.com/defenseunicorns/gojsonschema v0.0.0-20231116163348-e00f069122d6
 
 require (
 	cuelang.org/go v0.6.0
@@ -41,6 +43,7 @@ require (
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.17.0
 	github.com/stretchr/testify v1.8.4
+	github.com/xeipuuv/gojsonschema v1.2.0
 	golang.org/x/crypto v0.14.0
 	golang.org/x/sync v0.5.0
 	golang.org/x/term v0.13.0
@@ -413,7 +416,6 @@ require (
 	github.com/xanzy/ssh-agent v0.3.3 // indirect
 	github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
 	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
-	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
 	github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 // indirect
 	github.com/xlab/treeprint v1.2.0 // indirect
 	github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect

diff --git a/go.sum b/go.sum
@@ -445,6 +445,8 @@ github.com/daviddengcn/go-colortext v1.0.0/go.mod h1:zDqEI5NVUop5QPpVJUxE9UO10hR
 github.com/decred/dcrd/crypto/blake256 v1.0.1/go.mod h1:2OfgNZ5wDpcsFmHmCK5gZTPcCXqlm2ArzUIkw9czNJo=
 github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 h1:8UrgZ3GkP4i/CLijOJx79Yu+etlyjdBU4sfcs2WYQMs=
 github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0/go.mod h1:v57UDF4pDQJcEfFUCRop3lJL149eHGSe9Jvczhzjo/0=
+github.com/defenseunicorns/gojsonschema v0.0.0-20231116163348-e00f069122d6 h1:gwevOZ0fxT2nzM9hrtdPbsiOHjFqDRIYMzJHba3/G6Q=
+github.com/defenseunicorns/gojsonschema v0.0.0-20231116163348-e00f069122d6/go.mod h1:StKLYMmPj1R5yIs6CK49EkcW1TvUYuw5Vri+LRk7Dy8=
 github.com/deitch/magic v0.0.0-20230404182410-1ff89d7342da h1:ZOjWpVsFZ06eIhnh4mkaceTiVoktdU67+M7KDHJ268M=
 github.com/deitch/magic v0.0.0-20230404182410-1ff89d7342da/go.mod h1:B3tI9iGHi4imdLi4Asdha1Sc6feLMTfPLXh9IUYmysk=
 github.com/depcheck-test/depcheck-test v0.0.0-20220607135614-199033aaa936 h1:foGzavPWwtoyBvjWyKJYDYsyzy+23iBV7NKTwdk+LRY=
@@ -1526,13 +1528,10 @@ github.com/xdg-go/scram v1.1.2/go.mod h1:RT/sEzTbU5y00aCK8UOx6R7YryM0iF1N2MOmC3k
 github.com/xdg-go/stringprep v1.0.2/go.mod h1:8F9zXuvzgwmyT5DUm4GUfZGDdT3W+LCvS6+da4O5kxM=
 github.com/xdg-go/stringprep v1.0.3/go.mod h1:W3f5j4i+9rC0kuIEJL0ky1VpHXQU3ocBgklLGvcBnW8=
 github.com/xdg-go/stringprep v1.0.4/go.mod h1:mPGuuIYwz7CmR2bT9j4GbQqutWS1zV24gijq1dTyGkM=
-github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
 github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb h1:zGWFAtiMcyryUHoUjUJX0/lt1H2+i2Ka2n+D3DImSNo=
 github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
 github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0=
 github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ=
-github.com/xeipuuv/gojsonschema v1.2.0 h1:LhYJRs+L4fBtjZUfuSZIKGeVu0QRy8e5Xi7D17UxZ74=
-github.com/xeipuuv/gojsonschema v1.2.0/go.mod h1:anYRn/JVcOK2ZgGU+IjEV4nwlhoK5sQluxsYJ78Id3Y=
 github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8 h1:nIPpBwaJSVYIxUFsDv3M8ofmx9yWTog9BfvIu0q41lo=
 github.com/xi2/xz v0.0.0-20171230120015-48954b6210f8/go.mod h1:HUYIGzjTL3rfEspMxjDjgmT5uz5wzYJKVo23qUhYTos=
 github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ=

diff --git a/main.go b/main.go
@@ -5,16 +5,21 @@
 package main
 
 import (
-	_ "embed"
+	"embed"
 
 	"github.com/defenseunicorns/zarf/src/cmd"
 	"github.com/defenseunicorns/zarf/src/config"
+	"github.com/defenseunicorns/zarf/src/pkg/packager/lint"
 )
 
 //go:embed cosign.pub
 var cosignPublicKey string
 
+//go:embed zarf.schema.json
+var zarfSchema embed.FS
+
 func main() {
 	config.CosignPublicKey = cosignPublicKey
+	lint.ZarfSchema = zarfSchema
 	cmd.Execute()
 }
diff --git a/src/cmd/prepare.go b/src/cmd/prepare.go
@@ -17,6 +17,7 @@ import (
 	"github.com/defenseunicorns/zarf/src/config/lang"
 	"github.com/defenseunicorns/zarf/src/pkg/message"
 	"github.com/defenseunicorns/zarf/src/pkg/packager"
+	"github.com/defenseunicorns/zarf/src/pkg/packager/lint"
 	"github.com/defenseunicorns/zarf/src/pkg/transform"
 	"github.com/defenseunicorns/zarf/src/pkg/utils"
 	"github.com/defenseunicorns/zarf/src/pkg/utils/helpers"
@@ -205,6 +206,34 @@ var prepareGenerateConfigFile = &cobra.Command{
 	},
 }
 
+var lintCmd = &cobra.Command{
+	Use:     "lint [ DIRECTORY ]",
+	Args:    cobra.MaximumNArgs(1),
+	Aliases: []string{"l"},
+	Short:   lang.CmdPrepareLintShort,
+	Long:    lang.CmdPrepareLintLong,
+	Run: func(cmd *cobra.Command, args []string) {
+		baseDir := ""
+		if len(args) > 0 {
+			baseDir = args[0]
+		} else {
+			var err error
+			baseDir, err = os.Getwd()
+			if err != nil {
+				message.Fatalf(err, lang.CmdPrepareLintErr, err.Error())
+			}
+		}
+		validator, err := lint.ValidateZarfSchema(baseDir)
+		if err != nil {
+			message.Fatal(err, err.Error())
+		}
+		validator.DisplayFormattedMessage()
+		if !validator.IsSuccess() {
+			os.Exit(1)
+		}
+	},
+}
+
 func init() {
 	v := common.InitViper()
 
@@ -213,6 +242,7 @@ func init() {
 	prepareCmd.AddCommand(prepareComputeFileSha256sum)
 	prepareCmd.AddCommand(prepareFindImages)
 	prepareCmd.AddCommand(prepareGenerateConfigFile)
+	prepareCmd.AddCommand(lintCmd)
 
 	prepareComputeFileSha256sum.Flags().StringVarP(&extractPath, "extract-path", "e", "", lang.CmdPrepareFlagExtractPath)
 

diff --git a/src/config/config.go b/src/config/config.go
@@ -6,6 +6,7 @@ package config
 
 import (
 	"crypto/tls"
+	"embed"
 	"fmt"
 	"net/http"
 	"os"
@@ -92,6 +93,7 @@ var (
 	NoColor bool
 
 	CosignPublicKey string
+	ZarfSchema      embed.FS
 
 	// Timestamp of when the CLI was started
 	operationStartTime  = time.Now().Unix()

diff --git a/src/config/lang/english.go b/src/config/lang/english.go
@@ -359,6 +359,10 @@ $ zarf package publish ./path/to/dir oci://my-registry.com/my-namespace
 	CmdPrepareFlagGitAccount    = "User or organization name for the git account that the repos are created under."
 	CmdPrepareFlagKubeVersion   = "Override the default helm template KubeVersion when performing a package chart template"
 
+	CmdPrepareLintShort = "Verifies the package schema"
+	CmdPrepareLintLong  = "Verifies the package schema and warns the user if they have variables that won't be evaluated"
+	CmdPrepareLintErr   = "Unable to lint package: %s"
+
 	// zarf tools
 	CmdToolsShort = "Collection of additional tools to make airgap easier"
 

diff --git a/src/pkg/message/connect.go b/src/pkg/message/connect.go
@@ -8,22 +8,22 @@ import (
 	"fmt"
 
 	"github.com/defenseunicorns/zarf/src/types"
-	"github.com/pterm/pterm"
 )
 
 // PrintConnectStringTable prints a table of connect strings.
 func PrintConnectStringTable(connectStrings types.ConnectStrings) {
 	Debugf("message.PrintConnectStringTable(%#v)", connectStrings)
 
 	if len(connectStrings) > 0 {
-		list := pterm.TableData{{"     Connect Command", "Description"}}
+		connectData := [][]string{}
 		// Loop over each connectStrings and convert to pterm.TableData
 		for name, connect := range connectStrings {
-			name = fmt.Sprintf("     zarf connect %s", name)
-			list = append(list, []string{name, connect.Description})
+			name = fmt.Sprintf("zarf connect %s", name)
+			connectData = append(connectData, []string{name, connect.Description})
 		}
 
 		// Create the table output with the data
-		_ = pterm.DefaultTable.WithHasHeader().WithData(list).Render()
+		header := []string{"Connect Command", "Description"}
+		Table(header, connectData)
 	}
 }
diff --git a/src/pkg/message/credentials.go b/src/pkg/message/credentials.go
@@ -34,37 +34,32 @@ func PrintCredentialTable(state *types.ZarfState, componentsToDeploy []types.Dep
 	// Set output to os.Stderr to avoid creds being printed in logs
 	pterm.SetDefaultOutput(os.Stderr)
 
-	pterm.Println()
-	loginTableHeader := pterm.TableData{
-		{"     Application", "Username", "Password", "Connect", "Get-Creds Key"},
-	}
-
-	loginTable := pterm.TableData{}
+	loginData := [][]string{}
 	if state.RegistryInfo.InternalRegistry {
-		loginTable = append(loginTable, pterm.TableData{
-			{"     Registry", state.RegistryInfo.PushUsername, state.RegistryInfo.PushPassword, "zarf connect registry", RegistryKey},
-			{"     Registry (read-only)", state.RegistryInfo.PullUsername, state.RegistryInfo.PullPassword, "zarf connect registry", RegistryReadKey},
-		}...)
+		loginData = append(loginData,
+			[]string{"Registry", state.RegistryInfo.PushUsername, state.RegistryInfo.PushPassword, "zarf connect registry", RegistryKey},
+			[]string{"Registry (read-only)", state.RegistryInfo.PullUsername, state.RegistryInfo.PullPassword, "zarf connect registry", RegistryReadKey},
+		)
 	}
 
 	for _, component := range componentsToDeploy {
 		// Show message if including logging stack
 		if component.Name == "logging" {
-			loginTable = append(loginTable, pterm.TableData{{"     Logging", config.ZarfLoggingUser, state.LoggingSecret, "zarf connect logging", LoggingKey}}...)
+			loginData = append(loginData, []string{"Logging", config.ZarfLoggingUser, state.LoggingSecret, "zarf connect logging", LoggingKey})
 		}
 		// Show message if including git-server
 		if component.Name == "git-server" {
-			loginTable = append(loginTable, pterm.TableData{
-				{"     Git", state.GitServer.PushUsername, state.GitServer.PushPassword, "zarf connect git", GitKey},
-				{"     Git (read-only)", state.GitServer.PullUsername, state.GitServer.PullPassword, "zarf connect git", GitReadKey},
-				{"     Artifact Token", state.ArtifactServer.PushUsername, state.ArtifactServer.PushToken, "zarf connect git", ArtifactKey},
-			}...)
+			loginData = append(loginData,
+				[]string{"Git", state.GitServer.PushUsername, state.GitServer.PushPassword, "zarf connect git", GitKey},
+				[]string{"Git (read-only)", state.GitServer.PullUsername, state.GitServer.PullPassword, "zarf connect git", GitReadKey},
+				[]string{"Artifact Token", state.ArtifactServer.PushUsername, state.ArtifactServer.PushToken, "zarf connect git", ArtifactKey},
+			)
 		}
 	}
 
-	if len(loginTable) > 0 {
-		loginTable = append(loginTableHeader, loginTable...)
-		_ = pterm.DefaultTable.WithHasHeader().WithData(loginTable).Render()
+	if len(loginData) > 0 {
+		header := []string{"Application", "Username", "Password", "Connect", "Get-Creds Key"}
+		Table(header, loginData)
 	}
 
 	// Restore the log file if it was specified

diff --git a/src/pkg/message/message.go b/src/pkg/message/message.go
@@ -318,6 +318,28 @@ func Truncate(text string, length int, invert bool) string {
 	return textEscaped
 }
 
+// Table prints a padded table containing the specified header and data
+func Table(header []string, data [][]string) {
+	pterm.Println()
+
+	if len(header) > 0 {
+		header[0] = fmt.Sprintf("     %s", header[0])
+	}
+
+	table := pterm.TableData{
+		header,
+	}
+
+	for _, row := range data {
+		if len(row) > 0 {
+			row[0] = fmt.Sprintf("     %s", row[0])
+		}
+		table = append(table, pterm.TableData{row}...)
+	}
+
+	pterm.DefaultTable.WithHasHeader().WithData(table).Render()
+}
+
 func debugPrinter(offset int, a ...any) {
 	printer := pterm.Debug.WithShowLineNumber(logLevel > 2).WithLineNumberOffset(offset)
 	now := time.Now().Format(time.RFC3339)
-Original file line number
+Diff line change
@@ Expand Up / @@ -37,6 +37,7 @@ metadata: @@
     :::tip
     If you are using an Integrated Development Environment (such as [VS Code](../3-create-a-zarf-package/8-vscode.md)) to create and edit the `zarf.yaml` file, you can install or reference the [`zarf.schema.json`](https://github.com/defenseunicorns/zarf/blob/main/zarf.schema.json) file to get error checking and autocomplete.
+    Additionally, you can run `zarf prepare lint <directory>` to validate aginst the [`zarf.schema.json`](https://github.com/defenseunicorns/zarf/blob/main/zarf.schema.json)
     :::
@@ Expand Down @@