📖 CaDeT Spike Cleanup and Productionise

[jhpyke]

User Story

As a maintainer of CaDeT's airflow deploys
I need to have a single, clean image
So that I can use it, along with passed arguments at call time, to both deploy and retry tables in CaDeT.

Value / Purpose

This will give us a good basis for the public trial of Airflow-Based Deployments.

Useful Contacts

@jhpyke

User Types

No response

Hypothesis

If we make a clean image
Then we'll have a good technical base with which to create our Airflow Deployments.

Proposal

Migrating away from the Spike nature of our investigation, we should be:

• Publishing an image called airflow-deployment-cadet or similar.
• Said image should be generic, allowing environment variables to be passed as the job is called to specify which tags are being deployed, and whether retries are to be attempted.
• The image should be able to write to a relevant slack channel with a message that specfies it is an Airflow deployment, as opposed to a github one as it does currently.
• Able to accept an argument that specifies the region queries should be executed in. This should default to EU-West-1
• Creating an Airflow DAG of the form cadet_deployments.<name_of_workflow> for any of our test targets. Initially, for ourselves, this should be deploy weekly only to limit costs.
• DAG runs should run at a schedule that is aprox 12 hours offset from their current schedule in github actions, and target sandpit during the dual running period to avoid overstraining source datasets.

Additional Information

Separately to this, I will create an distribute a form to users allowing them to opt in to this trial period of dual running.

Definition of Done

• Image publishing repo exists that meets specs above
• Deploy weekly DAG is able to run and consume that
• DAGs are created for any respondents to the form.