Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ELE-47] feature: allow glue adapter #757

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Jeremynadal33
Copy link

@Jeremynadal33 Jeremynadal33 commented Oct 1, 2024

Partially resolves ELE-47

Description

This adds the needed macros to run both the dbt-data-reliability logging process in a dbt-glue set up and the macros needed for by the Elementary CLI.

It uses the Delta file format by default and therefore needs a glue environment that can use Delta. I add a new issue for other file formats.

However, since we use Delta Lake file format, we need to use a specific Glue Session :

Can anyone tell me how I could add disclaimers about it ?

FYI : I am sure this PR needs testing but I don't know how to do them, I would be grateful for advises on that 🙏

Tests that were run locally

@haritamar
Copy link
Collaborator

Hi @Jeremynadal33 !

I haven't had the chance to review the PR yet, but I wanted to comment on testing.
Please see the instructions here - you should see here how to run our integration tests with py.test.

Please let me know if you have any questions.

Cheers,
Itamar

@Jeremynadal33
Copy link
Author

Hi there,
Thanks a lot for your reactivity! I actually saw the test suit after submitting the PR... Haha. We are currently testing it and will get back to it as soon as we are done !

Thanks again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[ELE-47] AWS Glue Integration
2 participants