[SPIKE] Explore Argo CD Blue/Green and/or zero-downtime Deployment for Next.js servers on EKS

[olivereri]

Requirements

In an effort to ensure Next-Build infrastructure is immutable, meaning changes are not made in Production rather they are introduced only from the build stage, AP should explore the Blue-Green deployment method offered in ArgoCD.

Noting here that the intention is to minimize/eliminate downtime of the system; there is not a before/after testing requirement.

Acceptance criteria

Preview Give feedback

1. Create Next-Build Container Image Workflow #16857
   5 of 5
   Accelerated Publishing rollover +2
   
2. Determine length of downtime when new image is deployed
3. ArgoCD Blue-Green Deployment is investigated as a viable deployment method.
4. ArgoCD Blue-Green Deployment is implemented for Dev

Background & implementation details

https://argoproj.github.io/argo-rollouts/features/bluegreen/

[timcosgrove]

Questions that will inform this will be done in #17223.

[timcosgrove]

We should observe if the server becomes unresponsive/down, and how long that period is.

[timcosgrove]

Can we sync with CMS Deploy to minimize preview downtime?

[jschmidt-civicactions]

Examples for ArgoCD blue/green rollout strategy:
https://github.com/argoproj/argo-rollouts/tree/master/examples/helm-blue-green

[nfpappas-oddball]

Changes were made to Staging. Everything seems to work. To test I created a production tag. This triggered a new image to be built. When the image was built, i synced the project in argocd. Argo noticed the change and started creating new pods. It left 2 pods available , removed 2 pods, and stood 2 pods up.

[nfpappas-oddball]

As testing was successful , I proceeded to make this change to all helm charts for next. Testing for those pieces yielded similar results to prod.