Skip to content

Commit

Permalink
[Docs] Add limitation about copying target dir files to remote (#1305)
Browse files Browse the repository at this point in the history
It’s currently observed that users may assume, after reading the
[documentation](https://astronomer.github.io/astronomer-cosmos/configuration/cosmos-conf.html#remote-target-path),
that all files from the target directory of the dbt project are uploaded
to the remote target directory. However, this isn’t yet the case. The
`remote_target_path` configuration, introduced in PR
#1224, currently
only uploads files from the compiled directory within the target
directory of the dbt project—and only when the execution mode is set to
`ExecutionMode.AIRFLOW_ASYNC`. This PR updates the documentation for
this configuration to clarify this behavior and indicate that additional
files will become available as we expand support for uploading them
within the operators.

closes: #1291
  • Loading branch information
pankajkoti authored Nov 6, 2024
1 parent 1dd0364 commit 489195a
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions docs/configuration/cosmos-conf.rst
Original file line number Diff line number Diff line change
Expand Up @@ -130,8 +130,12 @@ This page lists all available Airflow configurations that affect ``astronomer-co

`remote_target_path`_:
(Introduced since Cosmos 1.7.0) The path to the remote target directory. This is the directory designated to
remotely copy & store in the files generated and stored by dbt in the dbt project's target directory. The value
for the remote target path can be any of the schemes that are supported by the
remotely copy & store in the files generated and stored by dbt in the dbt project's target directory.
While this remote path is intended to copy files from the dbt project’s target directory, Cosmos currently only
supports copying files from the ``compiled`` directory within the ``target`` folder — and only when the execution
mode is set to ``ExecutionMode.AIRFLOW_ASYNC``. Future releases will add support for copying additional files from
the target directory.
The value for the remote target path can be any of the schemes that are supported by the
`Airflow Object Store <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html>`_
feature introduced in Airflow 2.8.0 (e.g. ``s3://your_s3_bucket/target_dir/``, ``gs://your_gs_bucket/target_dir/``,
``abfs://your_azure_container/cache_dir``, etc.)
Expand Down

0 comments on commit 489195a

Please sign in to comment.