Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Protobuf upgrade breaks MWAA #11222

Open
2 tasks done
MazrimT opened this issue Jan 17, 2025 · 1 comment
Open
2 tasks done

[Bug] Protobuf upgrade breaks MWAA #11222

MazrimT opened this issue Jan 17, 2025 · 1 comment
Labels
bug Something isn't working triage

Comments

@MazrimT
Copy link

MazrimT commented Jan 17, 2025

Is this a new bug in dbt-core?

  • I believe this is a new bug in dbt-core
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

All versions of dbt-core is not installable in amazon MWAA after pr #10969 was merged.
Reason is that MWAA enforces airflow constraints https://raw.githubusercontent.com/apache/airflow/constraints-2.10.3/constraints-3.11.txt which specifies only a single version of protobuf==4.25.5
Even one step newer airflow 2.10.4 and with python 3.12 still only have that specific version.
This means dbt-core can not install on MWAA.

Expected Behavior

dbt-core (or any adapter using it) should not require such high version of protobuf and at a minimum require only the version that airflow has as constraints.

Steps To Reproduce

Try to install dbt-core (or dbt-snowflake) with the following at the top of requirements.txt file:
--constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.10.3/constraints-3.11.txt

Relevant log output

Environment

- OS:
- Python:
- dbt:

Which database adapter are you using with dbt?

No response

Additional Context

No response

@MazrimT MazrimT added bug Something isn't working triage labels Jan 17, 2025
@aleksandrgordienko
Copy link

Just observing and, unfortunately, not able to help with the issue itself…

Have you ever considered installation of dbt outside of Airflow? I found it quite difficult to always keep dependencies in sync for Airflow and dbt in one installation. If you split this two from each other, you can upgrade them separately and maintain dependencies separately two

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

2 participants