Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support aerospike batching #2403

Merged
merged 8 commits into from
Mar 7, 2024
Merged

Conversation

karolisg
Copy link
Contributor

No description provided.

@Jesse-Bakker
Copy link
Contributor

The sinks currently kind of assume BatchInsert is only used by snapshotting, because replication will have updates and deletes as well. I see we currently assume all operations are inserts?

@karolisg
Copy link
Contributor Author

The sinks currently kind of assume BatchInsert is only used by snapshotting, because replication will have updates and deletes as well. I see we currently assume all operations are inserts?

Yes, at the moment we support only inserts. Is it possible to use BatchInsert + Update + Delete operations?

@Jesse-Bakker
Copy link
Contributor

Yes, that's possible. It might be more efficient if you have mostly inserts (for example, for oracle, batch inserts use a fast path), but if there is a mix, using non-batch inserts will be more efficient (because a batch insert will always round-trip to the database, and to maintain ordering, the previous mixed batch will have to be flushed to the db as well, but a mix of inserts, updates and deletes will combine all these operations into a single batch).

So, long story short, batch insert is mostly for insert-only workloads. For mixed workloads, batch insert is less efficient on the sink side

@karolisg
Copy link
Contributor Author

Yes, that's possible. It might be more efficient if you have mostly inserts (for example, for oracle, batch inserts use a fast path), but if there is a mix, using non-batch inserts will be more efficient (because a batch insert will always round-trip to the database, and to maintain ordering, the previous mixed batch will have to be flushed to the db as well, but a mix of inserts, updates and deletes will combine all these operations into a single batch).

So, long story short, batch insert is mostly for insert-only workloads. For mixed workloads, batch insert is less efficient on the sink side

Got it. Using simple insert now.

@karolisg karolisg force-pushed the feat/support-aerospike-batching branch from 8c30ee0 to da02e19 Compare March 4, 2024 17:35
@karolisg karolisg requested a review from chubei March 5, 2024 12:23
@karolisg karolisg force-pushed the feat/support-aerospike-batching branch from 10da44b to 1e945ed Compare March 5, 2024 13:35
@chubei chubei merged commit f86cfd4 into main Mar 7, 2024
3 of 4 checks passed
@chubei chubei deleted the feat/support-aerospike-batching branch March 7, 2024 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants