Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

connector: add a debezium. prefix for CDC WITH OPTIONS #14176

Closed
xxchan opened this issue Dec 25, 2023 · 3 comments
Closed

connector: add a debezium. prefix for CDC WITH OPTIONS #14176

xxchan opened this issue Dec 25, 2023 · 3 comments
Assignees
Milestone

Comments

@xxchan
Copy link
Member

xxchan commented Dec 25, 2023

          > CDC needs to pass many options directly to debezium

Do you mean that all unknown fields will be passed to Debezium? If so, it sounds terrible and we should introduce a common prefix for it, such as debezium. cc. @StrikeW

Originally posted by @fuyufjh in #13654 (comment)

@StrikeW
Copy link
Contributor

StrikeW commented Dec 25, 2023

To be clear, I agree to add a common debezium. prefix for properties that are not listed in our document (aka unknown fields) but provided by the underlying debezium connector. For example, right now our doc lists 8 property for the mysql-cdc connector (https://docs.risingwave.com/docs/current/ingest-from-mysql-cdc/). And user can pass debezium.snapshot.mode property to WITH to control the snapshot behavior of the debezium connector.

@StrikeW StrikeW self-assigned this Dec 25, 2023
Copy link
Contributor

This issue has been open for 60 days with no activity. Could you please update the status? Feel free to continue discussion or close as not planned.

@StrikeW
Copy link
Contributor

StrikeW commented Apr 8, 2024

unknown fields without the debezium. prefix will be ignored and would not pass to Debezium.

private static Map<String, String> extractDebeziumProperties(
Map<String, String> userProperties) {
// retain only debezium properties if any
var dbzProps = new HashMap<String, String>();
for (var entry : userProperties.entrySet()) {
var key = entry.getKey();
if (key.startsWith(DBZ_PROPERTY_PREFIX)) {
// remove the prefix
dbzProps.put(key.substring(DBZ_PROPERTY_PREFIX.length()), entry.getValue());
}
}
return dbzProps;
}

@StrikeW StrikeW modified the milestones: release-1.9, release-1.8 Apr 8, 2024
@StrikeW StrikeW closed this as completed Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants