From 558d394ebe0f107b7b8804f6351747e076da9a06 Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Thu, 20 Oct 2022 23:47:40 +1300 Subject: [PATCH 1/8] Updated README Updated Readme with changes recommended by jack --- README.md | 68 +++++++++++++++++++++++++++++-------------------------- 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/README.md b/README.md index 60e1b4c..6edeaf3 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,35 @@ # Swarm Stack Updater -The Swarm Stack Updater an automated tool built in shell that can be run periodically on a docker swarm to search for and identify any out of date stacks. If any changes are found, this tool automatically downloads, updates, and redeploys software back onto the swarm. +The Swarm Stack Updater an automated tool built in shell that can be run periodically on a Docker Swarm to search for and identify any out of date stacks. If any changes are found, this tool automatically downloads, updates, and redeploys software back onto the swarm. -This tool has been developed in conjuction with the University of Canterbury Computer Science Education Research Group as a University of Canterbury Sofware Engineering final year project. +This tool has been specifically enginneered to find and update full stacks by downloading Docker Compose files from GitHub. This was after finding tools such as [Watchtower](https://github.com/containrrr/watchtower) or [Shepard](https://github.com/djmaze/shepherd) which only aimed to update individual Docker containers by updating images. Using Compose files means whole application stacks can be updated and can allow for configuration changes as well. -## How to use: -### Deploying Using Docker -* First create a config file matching this description: +## Requirements +* The tool makes use of the Github API and Container Registry. You may have to update the script to work for other +repositories and container registries. +* Website must have a status url that returns a JSON object that contains both +sha commit hash and a tag version. +* ```{"VERSION_NUMBER": "3.10.0", "GIT_SHA": "a49a111d"}``` +* Deployment on swarm requires Swarm Cronjob running and deployed on swarm +this can be found here https://github.com/crazy-max/swarm-cronjob +* From the bottom of the compose file you can see that a secret is required. This should be a ```github_access_token``` to allow for multiple requests. More about docker secrets can be found here: https://docs.docker.com/engine/swarm/secrets/ + +* Your docker images should also contain some extra information in their manifest files. This is so the app is able to identify whether a new image is available to download. + +Manifest JSON should contain at least: +``` +... +labels: { + "org.opencontainers.image.revision": "" +} +... +``` + +## Deployment: + +1. First create a config file so that the Swarm Stack Updater is able to recognise what it needs to update. This should contain the stack's name, whether you are deploying the tool in a development environment, the websites url that you are updating as well as some repository information. + +This should look something like this: ``` --- : @@ -18,11 +41,13 @@ This tool has been developed in conjuction with the University of Canterbury Com user: ``` -* Next save that config file as "swarm_updater_config" in your swarm, this can be done
-by typing ```docker config create swarm_updater_config ``` +2. Next save that config file as "swarm_updater_config" and set it as a config file in your swarm, this can be done
+by typing ```docker config create swarm_updater_config ``` or can be setup as part of step 3 in your deployment compose file. + +3. Deploy on your swarm swarm using Docker Compose. Make sure you include an env file if you need external varibles for deployment. Also to ensure that the tool can access Docker commands for your swarm add the volume ```/var/run/docker.sock:/var/run/docker.sock``` to give the tool access to the Docker daemon. -* Finally Deploy on swarm using docker compose: +An example of a compose file can be seen below: ``` version: '3.8' @@ -58,6 +83,8 @@ secrets: external: true ``` +## Development + ### Running Locally The tool has been designed specifically to run using docker swarm or compose but it can be run locally if required. I have tryed my best to remove all issues when running locally but it may not be perfect. @@ -72,30 +99,7 @@ git submodule update --init ``` This will pull all the required tools for running and creating tests for this application. - - -## Requirements -* The tool makes use of the github api and container registry. You may have to update the script to work for other -repositories and container registries. -* Website must have a status url that returns a JSON object that contains both -sha commit hash and a tag version. -* ```{"VERSION_NUMBER": "3.10.0", "GIT_SHA": "a49a111d"}``` -* Deployment on swarm requires swarm cronjob running and deployed on swarm -this can be found here https://github.com/crazy-max/swarm-cronjob -* From the bottom of the compose file you can see that a secret is required. This should be a github_access_token to allow for multiple requests. More about -docker secrets can be found here: https://docs.docker.com/engine/swarm/secrets/ -* Your docker images should also contain some extra information in their manifest files. This is so the app is able to identify whether a new image is available to download. - -Manifest JSON should contain at least: -``` -... -labels: { - "org.opencontainers.image.revision": "" -} -... -``` - -## Testing +### Testing BATS Core is being used to test some of the swarm stack updaters functions. This hopefully ensures that main functionailly of the application is kept consistant as updates and changes are preformed to the tool. ### Running Tests From d52db6398881e1c17c146cf108e0f7359e75deaf Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Fri, 21 Oct 2022 00:48:50 +1300 Subject: [PATCH 2/8] Added more description to running the tool locally --- README.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6edeaf3..488e964 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ This tool has been specifically enginneered to find and update full stacks by do ## Requirements +* Needs to be run in a docker swarm, this can either be locally or on a remove server. * The tool makes use of the Github API and Container Registry. You may have to update the script to work for other repositories and container registries. * Website must have a status url that returns a JSON object that contains both @@ -89,15 +90,17 @@ secrets: The tool has been designed specifically to run using docker swarm or compose but it can be run locally if required. I have tryed my best to remove all issues when running locally but it may not be perfect. -* First Clone the repository using following the command: +1. Clone the repository using following the command: ``` https://github.com/uccser/swarm-stack-updater.git ``` -* Then use the following command to pull BATS, which is used for automated testing: +2. Use the following command to pull BATS, which is used for automated testing. This will pull all the required tools for running and creating tests for this application. ``` git submodule update --init ``` -This will pull all the required tools for running and creating tests for this application. +3. Ensure that you have a configuration file created and have the requried environment varibles defined. To define environment varibles use the command ```export ENV_VAR=```. + +4. Run the swarm updater by running the command ./src/swarm_stack_updater.sh. You may need to set to to be executible this can be done by typing the command ```chmod +x /src/swarm_stack_updater.sh``` ### Testing BATS Core is being used to test some of the swarm stack updaters functions. This hopefully ensures that main functionailly of the application is kept consistant as updates and changes are preformed to the tool. From b3d3ea63ab0dab9d05e60b2b901440e7f5142bba Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Fri, 21 Oct 2022 00:49:24 +1300 Subject: [PATCH 3/8] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 488e964..f6a811c 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ git submodule update --init ``` 3. Ensure that you have a configuration file created and have the requried environment varibles defined. To define environment varibles use the command ```export ENV_VAR=```. -4. Run the swarm updater by running the command ./src/swarm_stack_updater.sh. You may need to set to to be executible this can be done by typing the command ```chmod +x /src/swarm_stack_updater.sh``` +4. Run the swarm updater by running the command ```./src/swarm_stack_updater.sh.``` You may need to set to to be executible this can be done by typing the command ```chmod +x /src/swarm_stack_updater.sh``` ### Testing BATS Core is being used to test some of the swarm stack updaters functions. This hopefully ensures that main functionailly of the application is kept consistant as updates and changes are preformed to the tool. From 67bc1be783a1de423ef6573a8069d0377edfff58 Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Sun, 23 Oct 2022 10:52:04 +1300 Subject: [PATCH 4/8] Apply suggestions from code review Co-authored-by: Jack Morgan --- README.md | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index f6a811c..ff1c93a 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,24 @@ # Swarm Stack Updater The Swarm Stack Updater an automated tool built in shell that can be run periodically on a Docker Swarm to search for and identify any out of date stacks. If any changes are found, this tool automatically downloads, updates, and redeploys software back onto the swarm. -This tool has been specifically enginneered to find and update full stacks by downloading Docker Compose files from GitHub. This was after finding tools such as [Watchtower](https://github.com/containrrr/watchtower) or [Shepard](https://github.com/djmaze/shepherd) which only aimed to update individual Docker containers by updating images. Using Compose files means whole application stacks can be updated and can allow for configuration changes as well. +This tool has been specifically enginneered to find and update full Docker Stacks by downloading Docker Compose files from GitHub. +This was after finding tools such as [Watchtower](https://github.com/containrrr/watchtower) or [Shepard](https://github.com/djmaze/shepherd) which only aimed to update individual Docker containers by updating images. +Using Compose files means whole application stacks can be updated and can allow for configuration changes as well. ## Requirements -* Needs to be run in a docker swarm, this can either be locally or on a remove server. -* The tool makes use of the Github API and Container Registry. You may have to update the script to work for other -repositories and container registries. -* Website must have a status url that returns a JSON object that contains both -sha commit hash and a tag version. -* ```{"VERSION_NUMBER": "3.10.0", "GIT_SHA": "a49a111d"}``` -* Deployment on swarm requires Swarm Cronjob running and deployed on swarm -this can be found here https://github.com/crazy-max/swarm-cronjob -* From the bottom of the compose file you can see that a secret is required. This should be a ```github_access_token``` to allow for multiple requests. More about docker secrets can be found here: https://docs.docker.com/engine/swarm/secrets/ - -* Your docker images should also contain some extra information in their manifest files. This is so the app is able to identify whether a new image is available to download. +* Can only be run in a Docker Swarm. +* The tool makes use of the Github API and Container Registry. + You may have to update the script to work for other repositories and container registries. +* Website must have a status url that returns a JSON object that contains both SHA-1 commit hash and a tag version. + ```{"VERSION_NUMBER": "7.1.0", "GIT_SHA": "634d994d771fa23f65bc735bba32317ca71b374d"}` +* Deployment requires [Swarm Cronjob](https://github.com/crazy-max/swarm-cronjob) running and deployed on the Swarm. +* A secret is required for multiple GitHub requests (see the Compose file). + This must be named ```github_access_token```. + More about Docker Secrets can be found in the [Docker documentation](https://docs.docker.com/engine/swarm/secrets/). + +* The Docker images must contain additional information in their manifest files. + This is so the tool can identify whether a new image is available to download. Manifest JSON should contain at least: ``` @@ -25,10 +28,12 @@ labels: { } ... ``` +An [example is available within the CS Unplugged repostiory](https://github.com/uccser/cs-unplugged/blob/develop/infrastructure/production/django/Dockerfile#L28-L36). ## Deployment: -1. First create a config file so that the Swarm Stack Updater is able to recognise what it needs to update. This should contain the stack's name, whether you are deploying the tool in a development environment, the websites url that you are updating as well as some repository information. +1. First create a config file so that the Swarm Stack Updater is able to recognise what it needs to update. + This should contain the stack's name, whether you are deploying the tool in a development environment, the websites url that you are updating as well as some repository information. This should look something like this: ``` @@ -94,13 +99,15 @@ The tool has been designed specifically to run using docker swarm or compose but ``` https://github.com/uccser/swarm-stack-updater.git ``` -2. Use the following command to pull BATS, which is used for automated testing. This will pull all the required tools for running and creating tests for this application. +2. Use the following command to pull BATS, which is used for automated testing. + This will pull all the required tools for running and creating tests for this application. ``` git submodule update --init ``` 3. Ensure that you have a configuration file created and have the requried environment varibles defined. To define environment varibles use the command ```export ENV_VAR=```. -4. Run the swarm updater by running the command ```./src/swarm_stack_updater.sh.``` You may need to set to to be executible this can be done by typing the command ```chmod +x /src/swarm_stack_updater.sh``` +4. Run the swarm updater by running the command ```./src/swarm_stack_updater.sh```. + You may need to set to to be executible this can be done by typing the command ```chmod +x /src/swarm_stack_updater.sh``` ### Testing BATS Core is being used to test some of the swarm stack updaters functions. This hopefully ensures that main functionailly of the application is kept consistant as updates and changes are preformed to the tool. From 8b505daae3f67b8da990a3132c692e067c223ffe Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Sun, 23 Oct 2022 11:21:55 +1300 Subject: [PATCH 5/8] Update README.md --- README.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index ff1c93a..0cc5e1a 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Using Compose files means whole application stacks can be updated and can allow * The tool makes use of the Github API and Container Registry. You may have to update the script to work for other repositories and container registries. * Website must have a status url that returns a JSON object that contains both SHA-1 commit hash and a tag version. - ```{"VERSION_NUMBER": "7.1.0", "GIT_SHA": "634d994d771fa23f65bc735bba32317ca71b374d"}` + ```{"VERSION_NUMBER": "7.1.0", "GIT_SHA": "634d994d771fa23f65bc735bba32317ca71b374d"}``` * Deployment requires [Swarm Cronjob](https://github.com/crazy-max/swarm-cronjob) running and deployed on the Swarm. * A secret is required for multiple GitHub requests (see the Compose file). This must be named ```github_access_token```. @@ -47,11 +47,22 @@ This should look something like this: user: ``` -2. Next save that config file as "swarm_updater_config" and set it as a config file in your swarm, this can be done
-by typing ```docker config create swarm_updater_config ``` or can be setup as part of step 3 in your deployment compose file. +2. Next save that config file as "swarm_updater_config" and set it as a config file in your swarm, this can be done by typing the following command (or can be setup as part of step 3 in your deployment compose file). +``` +docker config create swarm_updater_config +``` -3. Deploy on your swarm swarm using Docker Compose. Make sure you include an env file if you need external varibles for deployment. Also to ensure that the tool can access Docker commands for your swarm add the volume ```/var/run/docker.sock:/var/run/docker.sock``` to give the tool access to the Docker daemon. +3. Deploy on your swarm swarm using Docker Compose. +* Make sure you include an env file if you need external varibles for deployment. +* To ensure that the tool can use docker commands add the volume ```/var/run/docker.sock:/var/run/docker.sock``` to give the tool access to the Docker daemon. +* The following relates to [Swarm Cronjob](https://github.com/crazy-max/swarm-cronjob). Here the tool has been setup to run every 2 mins and will not re run if the tool is already running. +``` + labels: + - "swarm.cronjob.enable=true" + - "swarm.cronjob.schedule=*/2 * * * *" # Testing Update Every Minute + - "swarm.cronjob.skip-running=true" +``` An example of a compose file can be seen below: ``` From b31470069a0e052d31c9ebac59e1d8f4032ff0fb Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Sun, 23 Oct 2022 11:23:49 +1300 Subject: [PATCH 6/8] Update README.md --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0cc5e1a..ae2f725 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ docker config create swarm_updater_config 3. Deploy on your swarm swarm using Docker Compose. * Make sure you include an env file if you need external varibles for deployment. * To ensure that the tool can use docker commands add the volume ```/var/run/docker.sock:/var/run/docker.sock``` to give the tool access to the Docker daemon. -* The following relates to [Swarm Cronjob](https://github.com/crazy-max/swarm-cronjob). Here the tool has been setup to run every 2 mins and will not re run if the tool is already running. +* In order to configure [Swarm Cronjob](https://github.com/crazy-max/swarm-cronjob) Docker Labels are used. In the following the tool has been setup to run every 2 mins and will not re run if already running. ``` labels: - "swarm.cronjob.enable=true" @@ -64,7 +64,7 @@ docker config create swarm_updater_config - "swarm.cronjob.skip-running=true" ``` -An example of a compose file can be seen below: +An example of a full compose file to setup the Swarm Stack Updater can be seen below: ``` version: '3.8' From 64694eb1452120e21458cd1a19285e0d74c21e17 Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Sun, 23 Oct 2022 11:26:31 +1300 Subject: [PATCH 7/8] Update README.md --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index ae2f725..72054e9 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,9 @@ Using Compose files means whole application stacks can be updated and can allow * The tool makes use of the Github API and Container Registry. You may have to update the script to work for other repositories and container registries. * Website must have a status url that returns a JSON object that contains both SHA-1 commit hash and a tag version. - ```{"VERSION_NUMBER": "7.1.0", "GIT_SHA": "634d994d771fa23f65bc735bba32317ca71b374d"}``` + ``` + {"VERSION_NUMBER": "7.1.0", "GIT_SHA": "634d994d771fa23f65bc735bba32317ca71b374d"} + ``` * Deployment requires [Swarm Cronjob](https://github.com/crazy-max/swarm-cronjob) running and deployed on the Swarm. * A secret is required for multiple GitHub requests (see the Compose file). This must be named ```github_access_token```. @@ -20,7 +22,7 @@ Using Compose files means whole application stacks can be updated and can allow * The Docker images must contain additional information in their manifest files. This is so the tool can identify whether a new image is available to download. -Manifest JSON should contain at least: +* Manifest JSON should contain at least: ``` ... labels: { From fa75d4e8f6aed8d6b2f1108f26edff58e2e7db72 Mon Sep 17 00:00:00 2001 From: Samuel Burtenshaw <38925982+Burtens@users.noreply.github.com> Date: Mon, 24 Oct 2022 11:08:32 +1300 Subject: [PATCH 8/8] Update README.md --- README.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 72054e9..9e00978 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Swarm Stack Updater The Swarm Stack Updater an automated tool built in shell that can be run periodically on a Docker Swarm to search for and identify any out of date stacks. If any changes are found, this tool automatically downloads, updates, and redeploys software back onto the swarm. -This tool has been specifically enginneered to find and update full Docker Stacks by downloading Docker Compose files from GitHub. +This tool has been specifically engineered to find and update full Docker Stacks by downloading Docker Compose files from GitHub. This was after finding tools such as [Watchtower](https://github.com/containrrr/watchtower) or [Shepard](https://github.com/djmaze/shepherd) which only aimed to update individual Docker containers by updating images. Using Compose files means whole application stacks can be updated and can allow for configuration changes as well. @@ -30,7 +30,7 @@ labels: { } ... ``` -An [example is available within the CS Unplugged repostiory](https://github.com/uccser/cs-unplugged/blob/develop/infrastructure/production/django/Dockerfile#L28-L36). +An [example is available within the CS Unplugged repository](https://github.com/uccser/cs-unplugged/blob/develop/infrastructure/production/django/Dockerfile#L28-L36). ## Deployment: @@ -119,8 +119,14 @@ git submodule update --init ``` 3. Ensure that you have a configuration file created and have the requried environment varibles defined. To define environment varibles use the command ```export ENV_VAR=```. -4. Run the swarm updater by running the command ```./src/swarm_stack_updater.sh```. - You may need to set to to be executible this can be done by typing the command ```chmod +x /src/swarm_stack_updater.sh``` +4. Run the swarm updater by running the command +``` +./src/swarm_stack_updater.sh +``` +You may need to set to to be executible this can be done by typing the command +``` +chmod +x /src/swarm_stack_updater.sh +``` ### Testing BATS Core is being used to test some of the swarm stack updaters functions. This hopefully ensures that main functionailly of the application is kept consistant as updates and changes are preformed to the tool.