Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental Sync Support for All Streams #17

Open
dluftspring opened this issue May 24, 2022 · 3 comments
Open

Incremental Sync Support for All Streams #17

dluftspring opened this issue May 24, 2022 · 3 comments

Comments

@dluftspring
Copy link
Owner

Right now the tap only supports full refresh syncs. We should add support for incremental syncs.

@KPhans
Copy link
Collaborator

KPhans commented Dec 14, 2022

Looks like theres a need for this now. Will look into this and see it's feasible

@dluftspring
Copy link
Owner Author

Hey @KPhans ! long time no talk - let me know if you need any assistance otherwise just throw a PR up and i'm happy to take a look

@dluftspring
Copy link
Owner Author

May have over thought this. I think if you just figure out what the timestamp field is on each stream and add the replication_key property to each stream in streams.py this should just work out of the box after the first run. Depending on where you have the metadata hooked up to you'll just have to log in and make sure the bookmarks are being persisted properly to the database

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants