* Fix #16 and #22 (#23)

* * Fix #16 Resolve all deps inside containerized build
- Refactored Dockerfile to build a plugin in a container
- Add a comment to the merged `go.mod` file which contains the GlooE version
- Copy the go.mod file to the final layer so developers can download it after a succesful build.
- If the build fails during the `build-plugins` phase the usesd `go.mod` wil be printed to stdout for analysis.
- Changed the `build-plugins` rule to run local.
- Added `build` rule to run all other targets in a container

* Fix #22 Improve scripts to auto-upgrade go.mod based on gloo dependencies
- Removed public compare-deps code.
- Refactored and keep the private compare_dep code as a final check after merging the module files.
- If there are non matching deps, print the suggestions to stdout and abort, otherwise overwrite the go.mod
- Created `resolve-deps` rule to merge non matching dependencies to plugin's go.mod
  Using the following merge strategy:
    - Use plugin.mod as base
    - In plugin.mod, copy all matching require names and version from gloo.mod
    - In plugin.mod, copy all matching replace names with version from gloo.mod, where plugin's replace has a version defined
    - In plugin.mod, remove all replace entries which has the matching require name in gloo.mod

* Added tmp to ignore list
* Added/updated tests
* Updated documentation
* The go-control-plane not required in the go.mod.
* Made the run image configurable
* Added changelog for breaking change
* Changed changelog type
* Changed changelog
* Removed redundant arguments
* Removed redundant arguments
* Use the right dep version for impl.go
* processed review comments
* Update cloudbuild
* fixed: #23 (comment)
* fixed:
- #23 (comment)
- #23 (comment)
* rework:
- #23 (comment)
* Ensure env vars are being picked up in build command, as makefile runs each line in target in its own shell subprocess
* Go mod tidy
* Some fixes to ensure the verify script runs during container build process
* Fix runtime plugin mismatch by ensuring unquoted gc flags get passed to build command, also update to later GlooE (unrelated)
* Update go.mod and go.sum so plugin can be loaded succesffuly
* Run resolve deps
* Go mod tidy
* - Verify ARGs are set
- run verify script from Makefile
- Parmaterized the plugin build name and folder
* - go mod tidy
* - Cleanup variable
- Made variable configurable
* - Updated and rearranged docs
* - Updated and rearranged docs
* - Updated and rearranged docs
- Renamed var
* Minor correctness updates
* Minor cleanups
* Fix for older versions of GlooE (1.3.3 and older). Will need to add some special logic to support go module builds
* Bump default GlooE version to 1.3.8
* Getting go build image is a prereq to being able to docker build, since we need to know base image
* Shouldn't need to copy over current directory, since go mod vendor is in the make target now
* Allow setting env vars to build both new and old versions of Gloo plugins
* Less strict pre-reqs for targets
* Fix make target for tests
* Update readme to reflect changes
* Better style to have tests in different package than the code it's testing
* Remove commented out code
* Write script to determine gloo build mode
* Go mod tidy and goimports
* No need for working directory passthrough
* Codify GlooE version to build mode logic -- all we need to provide is GlooE version now to make build
* Update cloudbuild to reflect new target name

.gitignore

@@ -1,6 +1,4 @@
 .idea
 **.so
 _glooe
-mismatched_dependencies.json
-suggestions
-plugin_dependencies
+vendor

Dockerfile

@@ -1,40 +1,53 @@
-# This stage is parametrized to replicate the same environment Gloo Enterprise was built in.
+# Prepare the build environment.
+# Use this stage to add certificates and set proxies
 # All ARGs need to be set via the docker `--build-arg` flags.
 ARG GO_BUILD_IMAGE
+ARG RUN_IMAGE
 FROM $GO_BUILD_IMAGE AS build-env
 
-# This must contain the value of the `gcflag` build flag that Gloo was built with
-ARG GC_FLAGS
-# This must contain the path to the plugin verification script
-ARG VERIFY_SCRIPT
+# This stage is parametrized to replicate the same environment Gloo Enterprise was built in.
+# It is important to use the same container to build the plugin that Gloo Enterprise was
+# built in to ensure that the same go version and linker is used during compilation.
+# All ARGs need to be set via the docker `--build-arg` flags.
+FROM build-env as build
+ARG GLOOE_VERSION
+ARG STORAGE_HOSTNAME
+ARG PLUGIN_MODULE_PATH
+
+ENV GONOSUMDB=*
+ENV GLOOE_VERSION=$GLOOE_VERSION
 
-# Fail if VERIFY_SCRIPT not set
 # We don't have the same check as on GC_FLAGS as empty values are allowed there
-RUN if [[ ! $VERIFY_SCRIPT ]]; then echo "Required VERIFY_SCRIPT build argument not set" && exit 1; fi
+RUN if [ ! $GLOOE_VERSION ]; then echo "Required GLOOE_VERSION build argument not set" && exit 1; fi
+RUN if [ ! $STORAGE_HOSTNAME ]; then echo "Required STORAGE_HOSTNAME build argument not set" && exit 1; fi
 
 # Install packages needed for compilation
-RUN apk add --no-cache gcc musl-dev git
+RUN apk add --no-cache gcc musl-dev git make
 
-# Copy the repository to the image and set it as the working directory. The GOPATH here is `/go`.
-# The directory chosen here is arbitrary and does not influence the plugins compatibility with Gloo.
-ADD . /go/src/github.com/solo-io/ext-auth-plugin-examples/
-WORKDIR /go/src/github.com/solo-io/ext-auth-plugin-examples
+# Sets working dir to the correct directory
+# /go/src to support older versions of Gloo that built plugins with go modules disabled (i.e., gopath builds)
+WORKDIR /go/src/$PLUGIN_MODULE_PATH
 
-# Build plugins with CGO enabled, passing the GC_FLAGS flags
-ENV CGO_ENABLED=1 GOARCH=amd64 GOOS=linux
-RUN go build -buildmode=plugin -gcflags="$GC_FLAGS" -o plugins/RequiredHeader.so plugins/required_header/plugin.go
+# Resolve dependencies and ensure dependency version usage
+COPY Makefile go.mod go.sum ./
+COPY pkg ./pkg
+COPY scripts ./scripts
+COPY plugins ./plugins
 
-# Run the script to verify that the plugin(s) can be loaded by Gloo
-RUN chmod +x $VERIFY_SCRIPT
-RUN $VERIFY_SCRIPT -pluginDir plugins -manifest plugins/plugin_manifest.yaml
+RUN make resolve-deps
+RUN echo "// Generated for GlooE $GLOOE_VERSION" | cat - go.mod > go.new && mv go.new go.mod
+# Compile and verify the plugin can be loaded by Gloo
+RUN make build-plugin || { echo "Used module:" | cat - go.mod; exit 1; }
 
 # This stage builds the final image containing just the plugin .so files. It can really be any linux/amd64 image.
-FROM alpine:3.10.1
+FROM $RUN_IMAGE
+ARG PLUGIN_MODULE_PATH
 
 # Copy compiled plugin file from previous stage
 RUN mkdir /compiled-auth-plugins
-COPY --from=build-env /go/src/github.com/solo-io/ext-auth-plugin-examples/plugins/RequiredHeader.so /compiled-auth-plugins/
+COPY --from=build /go/src/$PLUGIN_MODULE_PATH/plugins/*.so /compiled-auth-plugins/
+COPY --from=build /go/src/$PLUGIN_MODULE_PATH/go.mod /compiled-auth-plugins/
 
 # This is the command that will be executed when the container is run.
 # It has to copy the compiled plugin file(s) to a directory.
-CMD cp /compiled-auth-plugins/* /auth-plugins/
+CMD cp /compiled-auth-plugins/*.so /auth-plugins/

Makefile

@@ -4,17 +4,48 @@ format:
 	goimports -w -e plugins scripts
 
 #----------------------------------------------------------------------------------
-# Retrieve GlooE build information
+# Set build variables
 #----------------------------------------------------------------------------------
-GLOOE_DIR := _glooe
-_ := $(shell mkdir -p $(GLOOE_DIR))
 
 # Set this variable to the version of GlooE you want to target
-GLOOE_VERSION ?= 1.3.4
+GLOOE_VERSION ?= 1.3.8
+
+# Set this variable to the name of your build plugin
+PLUGIN_BUILD_NAME ?= RequiredHeader.so
+
+# Set this variable to the image name and tag of your plugin
+PLUGIN_IMAGE ?= gloo-ext-auth-plugins:$(GLOOE_VERSION)
+
+# Set this variable to the name of your plugin
+PLUGIN_NAME ?= required_header
+
+# Set this variable to the base image name for the container that will have the compiled plugin
+RUN_IMAGE ?= alpine:3.11
 
 # Set this variable to the hostname of your custom (air gapped) storage server
 STORAGE_HOSTNAME ?= storage.googleapis.com
 
+GLOOE_DIR := _glooe
+_ := $(shell mkdir -p $(GLOOE_DIR))
+
+PLUGIN_MODULE_PATH := $(shell grep module go.mod | cut -d ' ' -f 2-)
+
+#----------------------------------------------------------------------------------
+# Build an docker image which contains the plugin framework and plugin implementation
+#----------------------------------------------------------------------------------
+.PHONY: build
+build: $(GLOOE_DIR)/build_env
+	docker build --no-cache \
+		--build-arg GO_BUILD_IMAGE=$(call get_glooe_var,GO_BUILD_IMAGE) \
+		--build-arg RUN_IMAGE=$(RUN_IMAGE) \
+		--build-arg GLOOE_VERSION=$(GLOOE_VERSION) \
+		--build-arg STORAGE_HOSTNAME=$(STORAGE_HOSTNAME) \
+		--build-arg PLUGIN_MODULE_PATH=$(PLUGIN_MODULE_PATH) \
+		-t $(PLUGIN_IMAGE) .
+
+#----------------------------------------------------------------------------------
+# Retrieve GlooE build information
+#----------------------------------------------------------------------------------
 .PHONY: get-glooe-info
 get-glooe-info: $(GLOOE_DIR)/dependencies $(GLOOE_DIR)/verify-plugins-linux-amd64 $(GLOOE_DIR)/build_env
 
@@ -34,9 +65,12 @@ $(GLOOE_DIR)/build_env:
 get-plugin-dependencies: go.mod go.sum
 	go list -m all > plugin_dependencies
 
-.PHONY: compare-deps
-compare-deps: get-plugin-dependencies $(GLOOE_DIR)/dependencies
-	go run scripts/compare_deps/main.go plugin_dependencies $(GLOOE_DIR)/dependencies
+#----------------------------------------------------------------------------------
+# Compare and merge mon matching dependencies against GlooE
+#----------------------------------------------------------------------------------
+.PHONY: resolve-deps
+resolve-deps: go.mod $(GLOOE_DIR)/dependencies
+	go run scripts/resolve_deps/main.go go.mod $(GLOOE_DIR)/dependencies
 
 #----------------------------------------------------------------------------------
 # Build plugins
@@ -48,20 +82,44 @@ define get_glooe_var
 $(shell grep $(1) $(GLOOE_DIR)/build_env | cut -d '=' -f 2-)
 endef
 
-.PHONY: build-plugins
-build-plugins: $(GLOOE_DIR)/build_env $(GLOOE_DIR)/verify-plugins-linux-amd64
-	docker build --no-cache \
-		--build-arg GO_BUILD_IMAGE=$(call get_glooe_var,GO_BUILD_IMAGE) \
-		--build-arg GC_FLAGS=$(call get_glooe_var,GC_FLAGS) \
-		--build-arg VERIFY_SCRIPT=$(GLOOE_DIR)/verify-plugins-linux-amd64 \
-		.
+.PHONY: build-plugin
+build-plugin: compile-plugin verify-plugin
+
+.PHONY: compile-plugin
+compile-plugin: $(GLOOE_DIR)/build_env
+	# If using older versions of GlooE (1.3.3 or 1.4.0-beta2, or earlier) must build with GO111MODULE=off, also:
+	#   De-vendor all the dependencies and move them to the GOPATH, so they will be loaded from there.
+	#   We need this so that the import paths for any library shared between the plugins and Gloo are the same.
+	#
+	#   For example, if we were to vendor the ext-auth-plugin dependency, the ext-auth-server would load the plugin interface
+	#   as `GLOOE_REPO/vendor/github.com/solo-io/ext-auth-plugins/api.ExtAuthPlugin`, while the plugin
+	#   would instead implement `THIS_REPO/vendor/github.com/solo-io/ext-auth-plugins/api.ExtAuthPlugin`. These would be seen
+	#   by the go runtime as two different types, causing Gloo to fail.
+	#   Also, some packages cause problems if loaded more than once. For example, loading `golang.org/x/net/trace` twice
+	#   causes a panic (see here: https://github.com/golang/go/issues/24137). By flattening the dependencies this way we
+	#   prevent these sorts of problems.
+	# else just build with go modules
+	if go run scripts/determine_gloo_build_mode/main.go "v${GLOOE_VERSION}" | grep -q gomod; then \
+		echo "building plugin with go modules enabled"; \
+		GO111MODULE=on CGO_ENABLED=1 GOARCH=amd64 GOOS=linux go build -buildmode=plugin -gcflags=$(call get_glooe_var,GC_FLAGS) -o plugins/$(PLUGIN_BUILD_NAME) plugins/$(PLUGIN_NAME)/plugin.go; \
+	else \
+		echo "building plugin with go modules disabled"; \
+		go mod vendor; \
+		cp -a vendor/. /go/src/ && rm -rf vendor; \
+		GO111MODULE=off CGO_ENABLED=1 GOARCH=amd64 GOOS=linux go build -buildmode=plugin -gcflags=$(call get_glooe_var,GC_FLAGS) -o plugins/$(PLUGIN_BUILD_NAME) plugins/$(PLUGIN_NAME)/plugin.go; \
+	fi
+
+.PHONY: verify-plugin
+verify-plugin: $(GLOOE_DIR)/verify-plugins-linux-amd64
+	chmod +x $(GLOOE_DIR)/verify-plugins-linux-amd64
+	$(GLOOE_DIR)/verify-plugins-linux-amd64 -pluginDir plugins -manifest plugins/plugin_manifest.yaml
 
 .PHONY: build-plugins-for-tests
 build-plugins-for-tests: $(EXAMPLES_DIR)/required_header/RequiredHeader.so
 
 $(EXAMPLES_DIR)/required_header/RequiredHeader.so: $(SOURCES)
 	go build -buildmode=plugin -o $(EXAMPLES_DIR)/required_header/RequiredHeader.so $(EXAMPLES_DIR)/required_header/plugin.go
 
+.PHONY: clean
 clean:
 	rm -rf _glooe
-	rm suggestions mismatched_dependencies.json plugin_dependencies