Airflow CaDeT Spike follow-up - Further Testing and Expansion

[jhpyke]

In a followup to #5857 , we now look to expand the functionality of the Airflow CaDeT deployment to prove it can achieve parity with our existing deployment mechanisms. As such, we should look to expand the functionality being tested beyond the MVP.

Functionality to test:

• Can we replicate the existing retry functionality, where the retry is a separate, follow-up task.
• Can we use a DAG to trigger another DAG and deploy dependent tables (A deployment that waits for a previous deployment to finish)
• Can we post the status of a workflow (on failure) to a slack channel per the existing Slack integrations in Github Actions.
• Can we deploy on an hourly schedule as a minimum interval while maintaining stability.
• Can we do a large deployment while maintaining stability (Check we won't hit memory limits with any of our deploys).

Once we've tested these, we'll be confident we can create a setup that maintains parity with our existing github actions deployments.

[jnayak-moj]

The work is in progress.
made changes to the airflow docker image for the spike to use a different repo other than main.
added retry functionality in the deploy steps in the image
used the latest docker image in the airflow dag
testing is in progress.

[jnayak-moj]

The first task retry functionality is added, testing is in progress. The tests are passing when there are no models to retry. Also skip logic is tested. Working on testing the scenarios for the failed models for retry.

[jnayak-moj]

The retry scenario is passing now. The airflow dag is successfully executing the retry job.
main job:
https://23f37892-d1d1-4d9f-a03d-b8a53581fd20.c0.eu-west-1.airflow.amazonaws.com/log?execution_date=2025-01-15T16%3A45%3A01.749691%2B00%3A00&task_id=cadet-sandpit-pipeline-deploy-s3&dag_id=development-sandpit.deploy_sandpit_s3_version&map_index=-1

Retry job:
https://23f37892-d1d1-4d9f-a03d-b8a53581fd20.c0.eu-west-1.airflow.amazonaws.com/log?execution_date=2025-01-15T16%3A45%3A01.749691%2B00%3A00&task_id=cadet-sandpit-pipeline-deploy-retry&dag_id=development-sandpit.deploy_sandpit_s3_version&map_index=-1

[jnayak-moj]

Regarding the second requirement od a DAG triggering another DAG, the parent dag is able to trigger the child dag and the child dag is getting triggered but waiting in an execution queue indefinitely. Currently I am trying to debug this behaviour.

Parent DAG:
https://23f37892-d1d1-4d9f-a03d-b8a53581fd20.c0.eu-west-1.airflow.amazonaws.com/dags/development-sandpit.parent_dag/grid?search=development-sandpit.parent_dag&dag_run_id=manual__2025-01-15T16%3A45%3A13.895128%2B00%3A00

Child DAG:
https://23f37892-d1d1-4d9f-a03d-b8a53581fd20.c0.eu-west-1.airflow.amazonaws.com/dags/development-sandpit.child_dag/grid?search=development-sandpit.child_dag&dag_run_id=manual__2025-01-15T16%3A45%3A25.084254%2B00%3A00

[jnayak-moj]

regarding the 4th requirement if scheduling, I have tested the scheduling of the airflow DAG and its scheduling correctly as expected but the stability need to be tested with a long running deployment task.

Scheduled Job:
https://23f37892-d1d1-4d9f-a03d-b8a53581fd20.c0.eu-west-1.airflow.amazonaws.com/dags/development-sandpit.parent_dag/grid?search=development-sandpit.parent_dag&dag_run_id=scheduled__2025-01-15T15%3A20%3A00%2B00%3A00