Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Redesign Build Deployment Process (External) (e-mission#125)
* Create image_build_push.yml * Added docker build and push for dashboard and notebook images + Updated viz_scripts Dockerfile viz_scripts Dockerfile contains ENV variables from docker-compose as well. Added docker image commands to image_build_push yml for dashboard and notebook images * Changed sed to jq + Renamed docker image in image_push 1. Sed to jq change to make it consistent to what is being used in internal repos. 2. Renamed image pushed to docker hub. * Added TODO to change image push branch Currently the branch specified - "image-push-merge" is available locally on my system. I use it to test the automated docker image push mechanism whenever any changes are merged to this branch. Once, everything looks good, need to change this to master or main as per the repo. * Removed printing Docker username Had added it initially for testing purposes. Can remove now so it doesn't expose any sensitive info. * Added cert.sh + Modified Dockerfiles 1. docker/cert.sh - Uses new environment variable PROD_STAGE variable to determine whether in staging / production environment and only then install certificates, else skip. 2. Dockerfile Added environment variables here, keeping the same default ENV values as the ones in docker-compose.yml. Not adding in docker/Dockerfile.dev, since this change is being done primarily with the objective to aid with the automated build and push to Dockerhub. This pushed image would then be the one that would be used in the internal environments as the base image, which would be based on the non-dev Dockerfile. * Removing ENV variables from Dockerfile The other ENV vars that were added in Docker-compose files are currently present in the Dockerfile in the external repo root. Removing them from there and can add them as per requirement. For local testing, add them when “docker run” is used using the -e flag. For usage in stage / production, can be set by cloud team in AWS Codebuild as they currently do. CRON_MODE CRON_MODE can be moved to setting when docker run command is executed using -e flag. This is because this is required in start_notebook.sh in CMD layer in Dockerfile which is executed when the container is run. Hence, CRON_MODE value can be supplied later and not needed to be embedded in image. PROD_STAGE Not setting PROD_TRUE in docker run command since it is required during docker image build to decide whether to add certificates or not. Hence, adding it in the Dockerfile so it’s available during docker build. * Removed sed / jq usage from start scripts Can directly set DB_HOST since we can now use environment variables. No need to use jq or sed to replace file contents and copy over files. * Changing base image to build from redesign server image Created a new branch image-push-merge for e-mission-server in my forked repo. Also modified image push workflow to build and push docker image to docker hub on push to image-push-merge branch. Doing this since admin-dash was failing when was building from internal repo since this was still referring to old server code. Changed Dockerfile and docker/Dockerfile.dev in public-dash and admin-dash to build from this new image mukuflash03/e-mission-server:image-push-merge_2024-04-12--01-01 Redesigned server image is built from the image-push-merge branch on my personal forked repository. This branch has a workflow run set up to build the docker image and push it to Dockerhub whenever a push or merge happens to image-push-merge branch. Currently, I've been pushing to image-push and then creating a PR to merge into image-push-merge. Doing this, so admin-dash and public-dash can build from the latest redesigned server code. This is not the latest server code but the latest changes from my redesign PR. * Bumped up base server image tag Changing to build from base server image from my personal Dockerhub repository with redesigned server code. Will need to change to build from Shankari's Dockerhub repository, once all changes are final. * Bump up base server image tag * Artifact download test - 1 Added working code from join repo to fetch docker image tags using artifact download. * Bumped up server image tag Bumped up after fixing "url" KeyError bug in this commit in server repo: MukuFlash03/e-mission-server@e778b3f * Artifact + Matrix - 1 Added push and workflow_dispatch trigger handling to fetch appropriate docker image tags. Push trigger uses tag from artifact upload while workflow_dispatch uses tag from input parameters. Check commit from admin-dash here: MukuFlash03/op-admin-dashboard@39b289f * Artifact + Matrix - 2 Updating Dockerfiles to use ARG environment variable with latest timestamp that will be passed through: - `docker build --build-arg` command in Github actions in the workflow for automated pushing to Docker hub. - `args: ` config field in docker-compose which will need to be set manually by developers locally. Also, changing branch in fetch_runID and Dockerfiles to tags-combo-approach. * Artifact + Matrix - 3 Public-dash was failing as I had incorrectly replaced the docker build and push commands with the commands used for admin-dash in its YAML file. Public-dash has two images: frontend (dashboard) and viz_scripts (notebook-server) with their separate Dockerfiles. Hence, was getting the error: ERROR: failed to solve: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount264844780/Dockerfile: no such file or directory https://github.com/MukuFlash03/em-public-dashboard/actions/runs/8917300905/job/24490148189 * Artifact + Matrix - 4 For public-dash, admin-dash where ARGS are now being used, need to add the args under build command in the docker compose files. Gives error if arg is at the same hierarchical level as build. Also, public-dash docker-compose.yml (non-dev) version changed to have build: context, dockerfile ; unlike only build: frontend. This allows adding args under build. Similar to how currently being built in docker-compose.dev.yml. Also, args to be added under notebook-server and not dashboard since viz_scripts builds off of server image and not frontend, which is a node image. * Artifact + Matrix - 5 Adding .env file which stores only docker image timestamp for the latest dockerhub e-mission-server image already pushed. .env file overwritten in both types of trigger events - push and workflow_dispatch. Added commit and push github actions as well for pushing latest changes to the .env file made via the workflow. Lastly, docker-compose now also mentions the ENV variable name to be read from the .env file for the ARG value in the Dockerfile. No changes required in the Dockerfiles. Had to remove .env from the .gitignore file. * Updated docker image tag in .env to the latest timestamp: * Updated docker image tag in .env to the latest timestamp: 2024-05-03--14-37 * Added TODOs in github actions workflow YAML file Reminder for things to change as per master branch of e-mission-server once changes are finalized. * Artifact + Matrix - 6 Previous Push event triggers run failed Error occurred in GitHub actions git add, commit, push step. If file with no changes operated upon, it leaves an error: “nothing to commit, working tree clean Error: Process completed with exit code 1.” Need to fix. —— Quick fix is to make changes to .env file only if workflow_dispatch event is the trigger. Don’t do anything for push event. So, in case anyone modifies .env file on their own by using their own timestamp during testing, and pushes it as a part of their PR, then Shankari will have to ask them to revert the changes. Else, their custom timestamp will make it to the repo code base. Found something: https://www.reddit.com/r/github/comments/ju3ipr/commit_from_github_action_only_when_changes_exist/ It should work but there’s a drawback of using “exit 0” - it will mask all errors generated during “git commit”. This is bad and we won’t be able to see the reason why something wrong happened as the workflow would be shown as successful with a green tick. Found a solution with git diff: https://github.com/simonw/til/blob/main/github-actions/commit-if-file-changed.md $ git diff --quiet || (git add README.md && git commit -m "Updated README") However, I won’t be able to log any message saying that no changes to commit, tag not modified. Hence, will possibly use just “git diff —quiet” with an if-else block. Expected results: - Push event triggers workflow. - It writes DOCKER_IMAGE_TAG_1 fetched from last successful completed run to .env file. - It sees that there is a difference in the latest committed .env file in the dashboard repo which includes older timestamp. - Hence it runs git commit part of the script to reset to latest server timestamp. * Updated docker image tag in .env file to the latest timestamp * Updated docker image tag in .env file to the latest timestamp * Updated docker image tag in .env file to the latest timestamp * Updated docker image tag in .env file to the latest timestamp * Cleanup image_build_push.yml Preparing the yml for merge with main. * Polishing image_build_push.yml Prepped for merge with main * Polishing fetch_runID.py Prepping for merge with main. * Cert copy Per Shankari's permission, copying the cert file every time the viz_scripts dockerfile is run instead of deciding when to copy it based on an environment variable. * Delete viz_scripts/docker/cert.sh Removing cert.sh per Shankari's comments * Update docker-compose.yml Removing unnecessary change * Replacing docker build in image_build_push.yml with docker compose Removing the `docker build` command from image_build_push.yml and replacing it with `docker compose`. Since tag names cannot be specified as an input in compose, the names of the tags have to be set in the docker-compose.yml file. Additionally, these tags have to be input as environment variables ahead of the `docker compose` part of the command. I also renamed DOCKER_IMAGE_TAG to SERVER_IMAGE_TAG, since this is a more meaningful name. I will implement this change across the other repositories that use this variable name. * Delete .env * Remove whitespace changes to docker-compose.dev.yml Removing unnecessary whitespace changes. * Removing redundant pip install * Updating tag name * Adding artifact upload Adding a step to upload an artifact with the newly created image tag. This way, the file can be pulled into the internal repo with GH API. * Removing DB_HOST fallback in start_notebook.sh Fallback unnecessary * DOCKER_IMAGE_TAG --> SERVER_IMAGE_TAG rename * Testing workflow I want to see if passing in the tags to the docker-compose in the way I'm trying to will actually work. We'll see! * Testing workflow (again) Trying to test the workflow again. * Push test + adding rename step * dash to dot * change FROM context * Reverting changes made to test Reverting the changes that I made to test the workflow with docker compose. * Switching to build prod instead of dev * Modified tag variable names to be more relevant These store tags differently depending on the trigger event - Push OR Workflow dispatch * Updated username to clarify that env file is being updated * Re-added .env file This had been deleted in this commit: MukuFlash03@27c2f42 Added it with latest server image tag. Hence forth, once PRs are merged, it will read updated tags. * Changing images to use docker-compose DEV file Earlier commit had changed it to use PROD file but this file doesn't exist. The prod version of the file we have is docker-compose.yml instead. Using dev version only. * Tag and Push frontend dashboard image only on Push event Workflow dispatch event is triggered on changes to server image. This should build the notebook server image as it uses the server image. But frontend is Javascript nodejs based image and is unrelated to the server. Check this review comment: e-mission#125 (comment) * Hardcoded WEB_SERVER_HOST to 0.0.0.0 Check comment e-mission#125 (comment) * Changing to use non-dev version Based on this commit e-mission@82cf4e9 * Removing artifact method This was mainly needed for Push event but since Workflow dispatch event would be setting the latest server image tag in .env file, the push event can read from this file directly. * Added suffix to frontend image --------- Co-authored-by: Mahadik, Mukul Chandrakant <[email protected]> Co-authored-by: GitHub Action <[email protected]> Co-authored-by: Natalie Schultz <[email protected]>
- Loading branch information
4 people
authored
Aug 14, 2024
1 parent
3d2b1a5
commit ac676e9