Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add key configs for adapters using microbatch #6582

Merged
merged 12 commits into from
Dec 5, 2024

Conversation

mirnawong1
Copy link
Contributor

@mirnawong1 mirnawong1 commented Dec 3, 2024

this pr adds support forincremental stategy microbatch, which requires additional configs over the standard set:
dbt-postgres requires a unique_key
dbt-bigquery requires a partition_by
dbt-spark requires a [partition_by]

Resolves #6576


🚀 Deployment available! Here are the direct links to the updated files:

@mirnawong1 mirnawong1 requested a review from a team as a code owner December 3, 2024 15:17
Copy link

vercel bot commented Dec 3, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
docs-getdbt-com ✅ Ready (Inspect) Visit Preview Dec 5, 2024 10:05am

@github-actions github-actions bot added content Improvements or additions to content Docs team Authored by the Docs team @dbt Labs size: small This change will take 1 to 2 days to address labels Dec 3, 2024
Copy link
Contributor

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment

website/docs/docs/build/incremental-microbatch.md Outdated Show resolved Hide resolved
@github-actions github-actions bot added size: medium This change will take up to a week to address and removed size: small This change will take 1 to 2 days to address labels Dec 4, 2024
@mirnawong1 mirnawong1 changed the title add key requirements about adapter requirement add key configs for adapters using microbatch Dec 4, 2024
Copy link
Contributor

@nataliefiann nataliefiann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hiya @mirnawong1

I've approved this for you. I've made a non-blocking suggestion but yet to commit.

I'll also leave this for @matthewshaver to review.

Kind Regards
Natalie


<Lightbox src="/img/docs/building-a-dbt-project/microbatch/event_time.png" title="The event_time column configures the real-world time of this record"/>

### Required configs for specific adapters
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a separate header for this @QMalcolm 🙏

Copy link
Contributor

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for doing this work ❤️

@mirnawong1 mirnawong1 merged commit 0a3c0f3 into current Dec 5, 2024
9 checks passed
@mirnawong1 mirnawong1 deleted the add-microbatch-adapter-config branch December 5, 2024 10:31
Copy link
Contributor

github-actions bot commented Dec 5, 2024

Check your created Crawler

Check your created index on your Algolia Application

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content Improvements or additions to content Docs team Authored by the Docs team @dbt Labs size: medium This change will take up to a week to address trigger-crawl
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core] Some adapter incremental strategies require additional config keys
4 participants