Skip to content

Commit

Permalink
Turn off scarf based telemetry data collection (aws#149)
Browse files Browse the repository at this point in the history
*Issue #, if available:*

*Description of changes:*
Turning off scarf based telemetry
https://airflow.apache.org/docs/apache-airflow/2.10.1/configurations-ref.html#usage-data-collection

Basically adding an extra environment variable for this

Also updated readme to show 2.10.1 support

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.

---------

Co-authored-by: Michael Fu <[email protected]>
  • Loading branch information
michaelfu1029 and Michael Fu authored Oct 2, 2024
1 parent 155f0e0 commit 2ebece3
Show file tree
Hide file tree
Showing 3 changed files with 19 additions and 6 deletions.
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) uses to run
You can also use it locally if you want to run a MWAA-like environment for testing, experimentation,
and development purposes.

Currently, Airflow v2.9.2 is supported. Future versions in parity with Amazon MWAA will be added as
Currently, Airflow v2.9.2 and v2.10.1 are supported. Future versions in parity with Amazon MWAA will be added as
well. _Notice, however, that we do not plan to support previous Airflow versions supported by MWAA._

## Using the Airflow Image
Expand All @@ -26,7 +26,7 @@ To experiment with the image using a vanilla Docker setup, follow these steps:
python3 create_venvs.py --target <development | production>
```

3. Build the Airflow v2.9.2 Docker image
3. Build a supported Airflow version Docker image
- `cd <amazon-mwaa-docker-images path>/images/airflow/2.9.2`
- Update `run.sh` file with your account ID, environment name and account credentials. The permissions associated
with the provided credentials will be assigned to the Airflow components that would be started with the next step.
Expand All @@ -46,8 +46,7 @@ Airflow should be up and running now. You can access the web server on your loca

When you build the Docker images of a certain Airflow version, using either `build.sh` or `run.sh`
(which automatically also calls `build.sh` for you), multiple Docker images will actually be
generated. For example, for Airflow 2.9 (the only currently supported version), you will notice the
following images:
generated. For example, for Airflow 2.9, you will notice the following images:

| Repository | Tag |
| --------------------------------- | ----------------------------- |
Expand Down
16 changes: 15 additions & 1 deletion images/airflow/2.10.1/python/mwaa/config/airflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,19 @@ def _get_opinionated_airflow_secrets_config() -> Dict[str, str]:
"AIRFLOW__SECRETS__BACKEND_KWARGS": json.dumps(connection_lookup_pattern),
}

def _get_opinionated_airflow_usage_data_config() -> Dict[str, str]:
"""
Retrieve the environment variables for Airflow's usage data configuration section.
This config can be overridden by the user.
:returns A dictionary containing the environment variables.
"""

return {
"AIRFLOW__USAGE_DATA_COLLECTION__ENABLED": "False",
}


def _get_essential_airflow_webserver_config() -> Dict[str, str]:
"""
Expand Down Expand Up @@ -310,12 +323,13 @@ def get_opinionated_airflow_config() -> Dict[str, str]:
"""
Retrieve the environment variables required to set Airflow configurations.
These environment variables are essential and cannot be overridden by the customer.
These environment variables can be overridden by the customer.
:returns A dictionary containing the environment variables.
"""
return {
**_get_opinionated_airflow_core_config(),
**_get_opinionated_airflow_scheduler_config(),
**_get_opinionated_airflow_secrets_config(),
**_get_opinionated_airflow_usage_data_config(),
}
2 changes: 1 addition & 1 deletion images/airflow/2.9.2/python/mwaa/config/airflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -310,7 +310,7 @@ def get_opinionated_airflow_config() -> Dict[str, str]:
"""
Retrieve the environment variables required to set Airflow configurations.
These environment variables are essential and cannot be overridden by the customer.
These environment variables can be overridden by the customer.
:returns A dictionary containing the environment variables.
"""
Expand Down

0 comments on commit 2ebece3

Please sign in to comment.