This repository has been archived by the owner on Jul 24, 2024. It is now read-only.
Decouple hook executor from the operator, add Cloud Build execution and adjust testing #45
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dinigo
changed the title
The Airflow bash executor will run the command isolated in the worker. To be able to locate the profiles and code we have to tell dbt with the new implemented flag `dbt run --project-dir /path/to/my/project_dir --profiles-dir /path/to/profile.yaml` `
That comes useful when running the command inside cloud run, since the entrypoint for the command is already `dbt`, and this `dbt` is already in the PATH of the image
Local execution might not take place in the same folder. It depends on the underlying hook implementation. This way we ensure we're pointing to the folder with an absolute path even if run outside the project folder
dinigo
changed the title
|
Hello! The functionality is there. We've tested it with a DAG like this with DAG('dbt_dag_example', start_date=datetime(2021, 10, 1)) as dag:
# If no hook is provided, it defaults to the local execution hook
DbtRunOperator(
task_id='run_dbt_cli',
project_dir=abspath('./jaffle-shop'),
profiles_dir=abspath('.'),
dbt_bin='/home/airflow/.local/bin/dbt',
# local environment need to define the env variable for auth
env={ 'GOOGLE_APPLICATION_CREDENTIALS': '/run/secrets/gcp_credentials.json'},
)
DbtRunOperator(
task_id='run_dbt_cloud_build',
project_dir='jaffle-shop',
profiles_dir='jaffle-shop',
dbt_hook=DbtCloudBuildHook(
project_id='datatonic-uk-dop-dev',
gcs_staging_location='gs://some_bucket/jaffle-shop.tar.gz',
)
) |
Also make the command build function more idiomatic by using `append` to append single commands (flags) and `extend` when we want to append several.
Merged
|
@andrewrjones @etoulas : Could you assist me in getting this PR merged? if this repo is not going to be maintained I'll be happy to do so |
This was referenced Oct 22, 2021
|
This was closed when I renamed the main branch In my repo :/ |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #44
This extracts the execution from the
DbtHook. Defaults the Operators behaviour. It does not break the API. But in case the user needs another environment you can pass it a different hook with custom configuration.