📖 Airflow 3.0 - DAGFactory Setup for Standard Users

[jhpyke]

User Story

As a maintainer of Airflow
I want users to be able to easily scaffold standardised DAGs
So that I can maintain the platform effectively by reducing variance in how users are running DAGs, thereby making them easier to troubleshoot.

Value / Purpose

DAG Factory allows us to provide users with a similarly simple mechanism to IAM Builder, but for the construction of DAGs. As such we should look to integrate this more fully with our specific Analytical Platform setup. This will allow us to standardise the pods users are running, thus reducing the possibility space for people on support when resolving issues.

Useful Contacts

No response

User Types

No response

Hypothesis

If we... [do a thing]
Then... [this will happen]

Proposal

We should seek to create a template and a standard operator for Airflow, which allow for:

• A standard template for a 1 image, 1 task job
• A standard operator that creates a pre-configured kubernetes pod operator using a sensible set of requests and limits.
• This should be tested in a multi-image, multi-task setup, to ensure that all DAG Factory functionality can work in our new airflow environment.
• Any documentation that's specific to our setup should be generated.

Additional Information

No response

Definition of Done

• Creation of AP_Standard_Operator for Airflow
• Implementation of DAG Factory for generation of DAGs within the correct MWAA env.
• User docs have been created