Update the S3 source connector to use a later version of Kafka #383
Conversation
Signed-off-by: Aindriu Lavelle <[email protected]>
| @@ -21,7 +21,7 @@ plugins { id("aiven-apache-kafka-connectors-all.java-conventions") } | |||
| val amazonS3Version by extra("2.29.34") | |||
| val amazonSTSVersion by extra("2.29.34") | |||
| val s3mockVersion by extra("0.2.6") | |||
| val testKafkaVersion by extra("3.7.1") | |||
| val kafkaVersion by extra("3.7.2") | |||
There was a problem hiding this comment.
As this version is applied only on tests, we had it named with test prefix.
There was a problem hiding this comment.
Yes, I changed it to apply to all kafka versions so everything will stay in sync going forward.
There was a problem hiding this comment.
Missing updates :
testImplementation(apache.kafka.connect.api)
testImplementation(apache.kafka.connect.runtime)
testImplementation(apache.kafka.connect.json)
Considering previous discussions on not doing this, tagging @ivanyu @AnatolyPopov
Other options :
- Should we apply this change at repo root level (settings.gradle.kts) gets applied on all connectors
- Apply on s3 source and s3 sink
- not make this change at all
There was a problem hiding this comment.
Hi @muralibasani
testImplementation(apache.kafka.connect.api)
testImplementation(apache.kafka.connect.runtime)
testImplementation(apache.kafka.connect.json)
are updated on lines 83-85
I decided here to make this change only to the S3-source-connector in this PR as I am working on verifying the changes at a root level for the sink connectors.
I was hoping to have a discussion on this on Tuesday when people are back from their holidays, as to the benefits for the sink connector.
But for the source connector which can operate independently with this change, we gain access to https://kafka.apache.org/documentation/#connect_exactlyoncesourceconnectors as long as upgrade to at least 3.3.0 which is a very powerful tool for the source connectors.
There was a problem hiding this comment.
@aindriu-aiven While I am generally in favor of upgrades (and had proposed this myself three months ago), there are a couple of considerations to keep in mind:
Compatibility: We've been developing the S3 source connector to align (formats and everything) with the S3 sink (and not operate independently always). If Kafka clusters running the S3 sink are on an older version and they attempt to use this S3 source, the connector may fail to load. This backward compatibility is critical for users who wish to integrate both connectors.
Exactly-once semantics: This is a separate concern. Even with the relevant configurations enabled, the code would still require updates to handle operations like abort, commit, and transaction boundaries. Without these changes, the value of enabling this feature is limited.
Previously this was on Version 1.1.0 and upgrading to be on version 3.7.2