Move Notify System Code to Helmfile/AWS Secrets Manager

[P0NDER0SA]

Description

As a developer of notify, I would like all of our kubernetes deployments to be managed by Helmfile so that all of our kubernetes code is in one location. I would also like to have all secrets automated from a single source of truth (1pass -> AWS Secrets) and remove our dependency on encrypted .env files

WHY are we building?

• Provides the ability to deploy dynamic environments
• Provides the ability to run "diff" commands against environments
• Reduces the amount of code that Notify has to manage
• Keeps code DRY

WHAT are we building?

• Add Document Download entry to the helmfile
• source secrets for helmfile from AWS Secrets Manager
• source configuration values from helmfile overrides where necessary
• adjust GitHub actions to work with these changes
• This work is scoped to the DEV environment.

VALUE created by our solution

• Increased reliability via reduced chance of accidental changes during releases
• Increased velocity due to reduction in maintenance
• better secrets management that falls in line with our OKRs

Acceptance Criteria

• Notify System components are deployed and managed with helmfile
• All secrets required by Document Download are sourced from 1pass -> terraform -> aws secrets manager and accessed using a kubernetes secret
• The Notify System Stack still works

Not in scope

• Deployment in staging and production: we should only deploy to dev until we figure out the deployment steps.

QA Steps

• verify that the Notify System Pods and other K8s components are functional and running properly (might need some guidance on the best way to QA this!)

[P0NDER0SA]

Began work on this. I have a couple of different ways I could handle it, so I will have a conversation with the team to decide which route i'll take. I've already bootstrapped the project so it's ready for migration.

[P0NDER0SA]

cds-snc/notification-manifests#3137

Draft PR

[jimleroyer]

Currently blocked by release of other components, so waiting on other tasks.