Merge pull request #263 from allaboutapps/mwr/go-1.19-bullseye

Mwr/go 1.19 bullseye

.github/workflows/build-test.yml

@@ -36,9 +36,9 @@ jobs:
     steps:
       - uses: actions/[email protected]
       - name: docker build (target builder)
-        run: docker build --target builder --file Dockerfile --tag allaboutapps.dev/aw/go-starter:builder-${GITHUB_SHA} .
+        run: DOCKER_BUILDKIT=1 docker build --target builder --file Dockerfile --tag allaboutapps.dev/aw/go-starter:builder-${GITHUB_SHA} .
       - name: docker build (target app)
-        run: docker build --target app --file Dockerfile --tag allaboutapps.dev/aw/go-starter:app-${GITHUB_SHA} .
+        run: DOCKER_BUILDKIT=1 docker build --target app --file Dockerfile --tag allaboutapps.dev/aw/go-starter:app-${GITHUB_SHA} .
       - name: trivy scan
         uses: aquasecurity/trivy-action@master
         with:

.trivyignore

@@ -1,4 +0,0 @@
-# Allow https://nvd.nist.gov/vuln/detail/CVE-2020-26160 (JWT unused, still waiting for child deps upgrade)
-CVE-2020-26160
-# https://avd.aquasec.com/nvd/2022/cve-2022-27664/ golang.org/x/net is a peer dep of many of our dependencies, require Go 1.18.6+
-CVE-2022-27664

CHANGELOG.md

@@ -9,7 +9,44 @@
 - Please follow the update process in *[I just want to update / upgrade my project!](https://github.com/allaboutapps/go-starter/wiki/FAQ#i-just-want-to-update--upgrade-my-project)*.
 
 ## Unreleased
-- ...
+- Switch [from Go 1.17.9 to Go 1.19.3](https://go.dev/doc/devel/release#go1.19) (requires `./docker-helper.sh --rebuild`).
+  - Major: Update base docker image from debian buster to bullseye
+  - Minor: [Bump github.com/darold/pgFormatter from 5.2 to 5.3](https://github.com/darold/pgFormatter/releases/tag/v5.3)
+  - Minor: [Bump github.com/gotestyourself/gotestsum from 1.8.0 to 1.9.0](https://github.com/gotestyourself/gotestsum/releases/tag/v1.9.0)
+  - Minor: [Bump github.com/golangci/golangci-lint from 1.45.2 to 1.50.1](https://github.com/golangci/golangci-lint/releases/tag/v1.50.1)
+  - Minor: [Bump github.com/uw-labs/lichen from 0.1.5 to 0.1.7](https://github.com/uw-labs/lichen/releases/tag/v0.1.7)
+  - Minor: [Bump github.com/watchexec/watchexec from 1.18.11 to 1.20.6](https://github.com/watchexec/watchexec/releases/tag/v1.20.6)
+  - Minor: [Bump github.com/mikefarah/yq from 4.24.2 to 4.30.5](https://github.com/mikefarah/yq/releases/tag/v4.30.5)
+- Major: Upgrade distroless app image from base-debian10 to base-debian11
+- Major: Dockerfile is now build to support amd64 and arm64 architecture
+- Improve speed of `make swagger` when dealing with many files in `/api` by generating to a docker volume instead of the host filesystem, rsyncing only to changes into `/internal/types`. Furthermore split our swagger type generation and validation into two separate make targets, that can run concurrently (requires `./docker-helper.sh --rebuild`).
+  - Note that `/app/api/tmp`, `/app/tmp` and `/app/bin` are now baked by proper docker volumes when using our `docker-compose.yml`/`./docker-helper.sh --up`. You **cannot** remove these directories directly inside the container (but its contents) and you can also no longer see its files on your host machine directly! 
+- Fix `make check-gen-dirs` false positives hidden files.
+- Allow to trace/benchmark `Makefile` targets execution by using a custom shell wrapper for make execution. See `SHELL` and `.SHELLFLAGS` within `Makefile` and the custom `rksh` script in the root working directory. Usage: `MAKE_TRACE_TIME=true make <target>`
+- `go.mod` changes:
+  - Minor: [Bump github.com/BurntSushi/toml from 1.1.0 to 1.2.1](https://github.com/BurntSushi/toml/releases/tag/v1.2.1)
+  - Minor: [Bump github.com/gabriel-vasile/mimetype from 1.4.0 to 1.4.1](https://github.com/gabriel-vasile/mimetype/releases/tag/v1.4.1)
+  - Minor: [Bump github.com/go-openapi/errors from 0.20.2 to 0.20.3](https://github.com/go-openapi/errors/releases/tag/v0.20.3)
+  - Minor: [Bump github.com/go-openapi/runtime from 0.23.3 to 0.25.0](https://github.com/go-openapi/runtime)
+  - Minor: [Bump github.com/go-openapi/strfmt from 0.21.2 to 0.21.3](https://github.com/go-openapi/strfmt/releases/tag/v0.21.3)
+  - Minor: [Bump github.com/go-openapi/swag from 0.21.1 to 0.22.3](https://github.com/go-openapi/swag/releases/tag/v0.22.3)
+  - Minor: [Bump github.com/go-openapi/validate from 0.21.0 to 0.22.0](https://github.com/go-openapi/validate/releases/tag/v0.22.0)
+  - Minor: [Bump github.com/labstack/echo/v4 from 4.7.2 to 4.9.1](https://github.com/labstack/echo/releases/tag/v4.9.1) (Fixing CVE-2022-40083)
+  - Minor: [Bump github.com/lib/pq from 1.10.5 to 1.10.7](https://github.com/lib/pq/releases/tag/v1.10.7)
+  - Minor: [Bump github.com/nicksnyder/go-i18n/v2 from 2.2.0 to 2.2.1](https://github.com/nicksnyder/go-i18n/releases/tag/v2.2.1)
+  - Minor: [Bump github.com/rogpeppe/go-internal from 1.8.1 to 1.9.0](https://github.com/rogpeppe/go-internal/releases/tag/v1.9.0)
+  - Minor: [Bump github.com/rs/zerolog from 1.26.1 to 1.28.0](https://github.com/rs/zerolog/releases/tag/v1.28.0)
+  - Minor: [Bump github.com/rubenv/sql-migrate from 1.1.1 to 1.2.0](https://github.com/rubenv/sql-migrate/releases/tag/v1.2.0)
+  - Minor: [Bump github.com/spf13/cobra from 1.4.0 to 1.6.1](https://github.com/spf13/cobra/releases/tag/v1.6.1)
+  - Minor: [Bump github.com/spf13/viper from 1.10.1 to 1.14.0](https://github.com/spf13/viper/releases/tag/v1.14.0)
+  - Minor: [Bump github.com/stretchr/testify from 1.7.1 to 1.8.1](https://github.com/stretchr/testify/releases/tag/v1.8.1)
+  - Minor: [Bump github.com/subosito/gotenv from 1.2.0 to 1.4.1](https://github.com/subosito/gotenv/releases/tag/v1.4.1)
+  - Minor: [Bump github.com/volatiletech/sqlboiler/v4 from 4.9.2 to v4.13.0](https://github.com/volatiletech/sqlboiler/blob/master/CHANGELOG.md#v4130---2022-08-28)
+  - Minor: [Bump github.com/volatiletech/strmangle from 0.0.2 to 0.0.4](https://github.com/volatiletech/strmangle/releases/tag/v0.0.4) (changes in enum generation might require manual changes, minor changes)
+  - Minor: [Bump golang.org/x/crypto from v0.0.0-20220411220226-7b82a4e95df4 to 0.3.0](https://cs.opensource.google/go/x/crypto)
+  - Minor: [Bump golang.org/x/sys from v0.0.0-20220412211240-33da011f77ad to 0.2.0](https://cs.opensource.google/go/x/sys)
+  - Minor: [Bump golang.org/x/text from 0.3.7 to 0.4.0](https://cs.opensource.google/go/x/text) (Fixing CVE-2022-32149)
+  - Minor: [Bump google.golang.org/api from 0.74.0 to 0.103.0](https://github.com/googleapis/google-api-go-client/compare/v0.80.0...v0.103.0)
 
 ## 2022-09-13
 - Hotfix: Previously there was a chance of recursive error wrapping within our [`internal/api/router/error_handler.go`](https://github.com/allaboutapps/go-starter/blob/master/internal/api/router/error_handler.go) in combination with `*echo.HTTPError`. We currently disable this wrapping (as not used anyways) and will schedule a cleaner update regarding this error augmentation approach.

Dockerfile

@@ -4,7 +4,7 @@
 # --- https://hub.docker.com/_/golang
 # --- https://github.com/microsoft/vscode-remote-try-go/blob/master/.devcontainer/Dockerfile
 ### -----------------------
-FROM golang:1.17.9-buster AS development
+FROM golang:1.19.3-bullseye AS development
 
 # Avoid warnings by switching to noninteractive
 ENV DEBIAN_FRONTEND=noninteractive
@@ -15,8 +15,8 @@ ENV MAKEFLAGS "-j 8 --no-print-directory"
 # postgresql-support: Add the official postgres repo to install the matching postgresql-client tools of your stack
 # https://wiki.postgresql.org/wiki/Apt
 # run lsb_release -c inside the container to pick the proper repository flavor
-# e.g. stretch=>stretch-pgdg, buster=>buster-pgdg
-RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ buster-pgdg main" \
+# e.g. stretch=>stretch-pgdg, buster=>buster-pgdg, bullseye=>bullseye-pgdg
+RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ bullseye-pgdg main" \
     | tee /etc/apt/sources.list.d/pgdg.list \
     && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc \
     | apt-key add -
@@ -58,6 +58,7 @@ RUN apt-get update \
     postgresql-client-12 \
     icu-devtools \
     tmux \
+    rsync \
     # --- END DEVELOPMENT ---
     # 
     && apt-get clean \
@@ -78,9 +79,9 @@ ENV LANG en_US.UTF-8
 # https://github.com/darold/pgFormatter/releases
 RUN mkdir -p /tmp/pgFormatter \
     && cd /tmp/pgFormatter \
-    && wget https://github.com/darold/pgFormatter/archive/v5.2.tar.gz \
-    && tar xzf v5.2.tar.gz \
-    && cd pgFormatter-5.2 \
+    && wget https://github.com/darold/pgFormatter/archive/v5.3.tar.gz \
+    && tar xzf v5.3.tar.gz \
+    && cd pgFormatter-5.3 \
     && perl Makefile.PL \
     && make && make install \
     && rm -rf /tmp/pgFormatter 
@@ -89,27 +90,29 @@ RUN mkdir -p /tmp/pgFormatter \
 # https://github.com/gotestyourself/gotestsum/releases
 RUN mkdir -p /tmp/gotestsum \
     && cd /tmp/gotestsum \
-    && wget https://github.com/gotestyourself/gotestsum/releases/download/v1.8.0/gotestsum_1.8.0_linux_amd64.tar.gz \
-    && tar xzf gotestsum_1.8.0_linux_amd64.tar.gz \
+    && ARCH="$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)" \
+    && wget "https://github.com/gotestyourself/gotestsum/releases/download/v1.9.0/gotestsum_1.9.0_linux_${ARCH}.tar.gz" \
+    && tar xzf "gotestsum_1.9.0_linux_${ARCH}.tar.gz" \
     && cp gotestsum /usr/local/bin/gotestsum \
     && rm -rf /tmp/gotestsum 
 
 # go linting: (this package should NOT be installed via go get)
 # https://github.com/golangci/golangci-lint#binary
 # https://github.com/golangci/golangci-lint/releases
 RUN curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh \
-    | sh -s -- -b $(go env GOPATH)/bin v1.45.2
+    | sh -s -- -b $(go env GOPATH)/bin v1.50.1
 
 # go swagger: (this package should NOT be installed via go get) 
 # https://github.com/go-swagger/go-swagger/releases
-RUN curl -o /usr/local/bin/swagger -L'#' \
-    "https://github.com/go-swagger/go-swagger/releases/download/v0.29.0/swagger_linux_amd64" \
+RUN ARCH="$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)" \
+    && curl -o /usr/local/bin/swagger -L'#' \
+    "https://github.com/go-swagger/go-swagger/releases/download/v0.29.0/swagger_linux_${ARCH}" \
     && chmod +x /usr/local/bin/swagger
 
 # lichen: go license util 
 # TODO: Install from static binary as soon as it becomes available.
 # https://github.com/uw-labs/lichen/tags
-RUN go install github.com/uw-labs/[email protected].5
+RUN go install github.com/uw-labs/[email protected].7
 
 # cobra-cli: cobra cmd scaffolding generator
 # TODO: Install from static binary as soon as it becomes available.
@@ -120,18 +123,19 @@ RUN go install github.com/spf13/[email protected]
 # https://github.com/watchexec/watchexec/releases
 RUN mkdir -p /tmp/watchexec \
     && cd /tmp/watchexec \
-    && wget https://github.com/watchexec/watchexec/releases/download/cli-v1.18.11/watchexec-1.18.11-x86_64-unknown-linux-musl.tar.xz \
-    && tar xf watchexec-1.18.11-x86_64-unknown-linux-musl.tar.xz \
-    && cp watchexec-1.18.11-x86_64-unknown-linux-musl/watchexec /usr/local/bin/watchexec \
+    && wget https://github.com/watchexec/watchexec/releases/download/v1.20.6/watchexec-1.20.6-$(arch)-unknown-linux-musl.tar.xz \
+    && tar xf watchexec-1.20.6-$(arch)-unknown-linux-musl.tar.xz \
+    && cp watchexec-1.20.6-$(arch)-unknown-linux-musl/watchexec /usr/local/bin/watchexec \
     && rm -rf /tmp/watchexec
 
 # yq
 # https://github.com/mikefarah/yq/releases
 RUN mkdir -p /tmp/yq \
     && cd /tmp/yq \
-    && wget https://github.com/mikefarah/yq/releases/download/v4.24.2/yq_linux_amd64.tar.gz \
-    && tar xzf yq_linux_amd64.tar.gz \
-    && cp yq_linux_amd64 /usr/local/bin/yq \
+    && ARCH="$(arch | sed s/aarch64/arm64/ | sed s/x86_64/amd64/)" \
+    && wget "https://github.com/mikefarah/yq/releases/download/v4.30.5/yq_linux_${ARCH}.tar.gz" \
+    && tar xzf "yq_linux_${ARCH}.tar.gz" \
+    && cp "yq_linux_${ARCH}" /usr/local/bin/yq \
     && rm -rf /tmp/yq 
 
 # gsdev
@@ -186,6 +190,7 @@ ENV PATH $PATH:$GOBIN
 FROM development as builder
 WORKDIR /app
 COPY Makefile /app/Makefile
+COPY --chmod=0755 rksh /app/rksh
 COPY go.mod /app/go.mod
 COPY go.sum /app/go.sum
 RUN make modules
@@ -205,7 +210,7 @@ RUN make go-build
 # https://github.com/GoogleContainerTools/distroless/blob/master/base/README.md
 # The :debug image provides a busybox shell to enter (base-debian10 only, not static).
 # https://github.com/GoogleContainerTools/distroless#debug-images
-FROM gcr.io/distroless/base-debian10:debug as app
+FROM gcr.io/distroless/base-debian11:debug as app
 
 # FROM debian:buster-slim as app
 # RUN apt-get update \

Makefile

@@ -62,12 +62,14 @@ check-handlers: ##- (opt) Checks if implemented handlers match their spec (path)
 #    ^// Code generated .* DO NOT EDIT\.$
 check-gen-dirs: ##- (opt) Ensures internal/models|types only hold generated files.
 	@echo "make check-gen-dirs"
-	@grep -R -L '^// Code generated .* DO NOT EDIT\.$$' --exclude ".DS_Store" ./internal/types/ && echo "Error: Non generated file(s) in ./internal/types!" && exit 1 || exit 0
-	@grep -R -L '^// Code generated .* DO NOT EDIT\.$$' --exclude ".DS_Store" ./internal/models/ && echo "Error: Non generated file(s) in ./internal/models!" && && exit 1 || exit 0
+	@find ./internal/types -name ".*" -prune -o -type f -print | xargs -L1 grep -L '// Code generated .* DO NOT EDIT\.' \
+		|| (echo "Error: Non generated file(s) in ./internal/types!" && exit 1)
+	@find ./internal/models -name ".*" -prune -o -type f -print | xargs -L1 grep -L '// Code generated .* DO NOT EDIT\.' \
+		|| (echo "Error: Non generated file(s) in ./internal/models!" && exit 1)
 
 check-script-dir: ##- (opt) Ensures all scripts/**/*.go files have the "//go:build scripts" build tag set.
 	@echo "make check-script-dir"
-	@grep -R --include=*.go -L '//go:build scripts' ./scripts && echo "Error: Found unset '//go:build scripts' in ./scripts/**/*.go!" && exit 1 || exit 0
+	@find ./scripts -type f -name '*.go' | xargs -L1 grep -L '//go:build scripts' || (echo "Error: Found unset '//go:build scripts' in ./scripts/**/*.go!" && exit 1)
 
 # https://github.com/gotestyourself/gotestsum#format 
 # w/o cache https://github.com/golang/go/issues/24573 - see "go help testflag"
@@ -90,7 +92,7 @@ test-update-golden: ##- Refreshes all golden files / snapshot tests by running t
 
 # note that we explicitly don't want to use a -coverpkg=./... option, per pkg coverage take precedence
 go-test-by-pkg: ##- (opt) Run tests, output by package.
-	gotestsum --format pkgname-and-test-fails --jsonfile /tmp/test.log -- -race -cover -count=1 -coverprofile=/tmp/coverage.out ./...
+	gotestsum --format pkgname-and-test-fails --format-hide-empty-pkg --jsonfile /tmp/test.log -- -race -cover -count=1 -coverprofile=/tmp/coverage.out ./...
 
 go-test-by-name: ##- (opt) Run tests, output by testname.
 	gotestsum --format testname --jsonfile /tmp/test.log -- -race -cover -count=1 -coverprofile=/tmp/coverage.out ./...
@@ -132,7 +134,7 @@ modules: ##- (opt) Cache packages as specified in go.mod.
 
 # https://marcofranssen.nl/manage-go-tools-via-go-modules/
 tools: ##- (opt) Install packages as specified in tools.go.
-	@cat tools.go | grep _ | awk -F'"' '{print $$2}' | xargs -P $$(nproc) -L 1 -tI % go install %
+	@cat tools.go | grep _ | awk -F'"' '{print $$2}' | xargs -P $$(nproc) -tI % go install %
 
 tidy: ##- (opt) Tidy our go.sum file.
 	go mod tidy
@@ -233,9 +235,8 @@ watch-sql: ##- Watches *.sql files in /migrations and runs 'make sql-regenerate'
 # --- Swagger
 ### -----------------------
 
-swagger: ##- Runs make swagger-concat, swagger-lint-ref-siblings, and swagger-server.
+swagger: ##- Runs make swagger-concat and swagger-server.
 	@$(MAKE) swagger-concat
-	@$(MAKE) swagger-lint-ref-siblings
 	@$(MAKE) swagger-server
 
 # Any sibling elements of a $ref are ignored. This is because $ref works by replacing itself and everything on its level with the definition it is pointing at.
@@ -262,8 +263,8 @@ swagger-lint-ref-siblings: ##- (opt) Checks api/**/*.[yml|yaml] for invalid usag
 # https://goswagger.io/usage/flatten.html
 swagger-concat: ##- (opt) Regenerates api/swagger.yml based on api/paths/*.
 	@echo "make swagger-concat"
-	@rm -rf api/tmp
 	@mkdir -p api/tmp
+	@rm -rf api/tmp/*
 	@swagger mixin \
 		--output=api/tmp/tmp.yml \
 		--format=yaml \
@@ -276,24 +277,32 @@ swagger-concat: ##- (opt) Regenerates api/swagger.yml based on api/paths/*.
 		-q
 	@sed -i '1s@^@# // Code generated by "make swagger"; DO NOT EDIT.\n@' api/swagger.yml
 
+swagger-server: swagger-generate swagger-lint-ref-siblings swagger-validate ##- (opt) Lint/validate api/swagger.yml and generate /internal/types.
+
 # https://goswagger.io/generate/server.html
 # Note that we first flag all files to delete (as previously generated), regenerate, then delete all still flagged files
 # This allows us to ensure that any filewatchers (VScode) don't panic as these files are removed.
 # --keep-spec-order is broken (/tmp spec resolving): https://github.com/go-swagger/go-swagger/issues/2216
-swagger-server: ##- (opt) Regenerates internal/types based on api/swagger.yml.
-	@echo "make swagger-server"
-	@grep -R -L '^// Code generated .* DO NOT EDIT\.$$$$' ./internal/types \
-		| xargs sed -i '1s#^#// DELETE ME; DO NOT EDIT.\n#'
+swagger-generate: ##- (opt) Generate swagger /internal/types.
+	@echo "make swagger-generate"
+	@rm -rf tmp/testdata/types
+	@mkdir -p tmp/testdata/types
 	@swagger generate server \
 		--allow-template-override \
 		--template-dir=api/templates \
 		--spec=api/swagger.yml \
-		--server-package=internal/types \
-		--model-package=internal/types \
+		--server-package=tmp/testdata/types \
+		--model-package=tmp/testdata/types \
 		--exclude-main \
+		--skip-validation \
 		--config-file=api/config/go-swagger-config.yml \
 		-q
-	@find internal/types -type f -exec grep -q '^// DELETE ME; DO NOT EDIT\.$$' {} \; -delete
+	@find tmp/testdata/types -type f -exec sed -i "s|${GO_MODULE_NAME}/tmp/testdata/types|${GO_MODULE_NAME}/internal/types|g" {} \;
+	rsync -au --size-only --ignore-times --delete tmp/testdata/types/ internal/types/
+
+swagger-validate: ##- (opt) Validate api/swagger.yml.
+	@echo "make swagger-validate"
+	@swagger validate --skip-warnings --stop-on-error -q api/swagger.yml
 
 watch-swagger: ##- Watches *.yml|yaml|gotmpl files in /api and runs 'make swagger' on modifications.
 	@echo "Watching /api/**/*.yml|yaml|gotmpl. Use Ctrl-c to stop a run or exit."
@@ -359,8 +368,8 @@ git-merge-go-starter: ##- Merges upstream GIT_GO_STARTER_TARGET into current HEA
 
 clean: ##- Cleans ./tmp and ./api/tmp folder.
 	@echo "make clean"
-	@rm -rf tmp 2> /dev/null
-	@rm -rf api/tmp 2> /dev/null
+	@rm -rf tmp/* 2> /dev/null
+	@rm -rf api/tmp/* 2> /dev/null
 
 get-module-name: ##- Prints current go module-name (pipeable).
 	@echo "${GO_MODULE_NAME}"
@@ -443,5 +452,11 @@ LDFLAGS = $(eval LDFLAGS := "\
 # https://www.gnu.org/software/make/manual/html_node/One-Shell.html
 # required to ensure make fails if one recipe fails (even on parallel jobs) and on pipefails
 .ONESHELL:
-SHELL = /bin/bash
-.SHELLFLAGS = -cEeuo pipefail
+
+# # normal POSIX bash shell mode
+# SHELL = /bin/bash
+# .SHELLFLAGS = -cEeuo pipefail
+
+# wrapped make time tracing shell, use it via MAKE_TRACE_TIME=true make <target>
+SHELL = /app/rksh
+.SHELLFLAGS = $@