Running Amazon Managed Workflows for Apache Airflow (MWAA) locally on Finch

[austinvazquez]

TL; DR

Please give a plus one on each of these issues if you would like out of the box support for AWS MWAA on Finch.

• Enable mwaa runner using Finch aws/aws-mwaa-local-runner#312
• Enable running local AWS Managed Workflows for Apache Airflow #569

If you are willing to accept a bleeding edge patch, either checkout my draft pull request or apply my publicly available patch to your local runner.

curl https://gist.githubusercontent.com/austinvazquez/c9a1de9086c62e00980ab7cab3f4f491/raw/5d3d3ef258d746ca04157819fb3c895f86d83346/enable-mwaa-v2.6.3-local-runner-on-finch.patch
 | git apply

For the detail oriented...

Got this question via direct message and wanted to share with the broader community. Effectively it went something like this:

Can I run Amazon Managed Workflows for Apache Airflow (MWAA) locally on Finch like I can with Docker?

My initial reaction was "I do not think anyone has tried... but I also do not see why not" <10 minutes later I had something working...ish. There is two small blockers to what I have so far.

1. When building the container image used by the application, I had to disable --compress due to Finch not supporting the feature flag.
2. When starting the application, I had to drop the compose up flag --abort-on-container-exit due to Finch not supporting the feature flag.

Other than that I was able to get their application running on Finch. At least, to the best of my knowledge. (This is the first time I've seen/used the tool before)

So yeah feel free to pull the patch and try it out yourself. Happy to get some feedback on it or if you would like something a little more polished give some +1s to the issues listed at the top so I can advocate to my bosses for more time to work on this. The two blockers are minor IMO. The first really just means a larger container image and the second means manually killing the container. e.g.

finch container ls
finch container stop <CONTAINER_ID>