s3 object summary iterator improvements #340
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Claudenw
force-pushed
the
KCON-51_S3ObjectSummaryIterator_improvements
branch
3 times, most recently
from
b6ccad4 to
f3ac5a5
Compare
aindriu-aiven
changed the title
RyanSkraba
force-pushed
the
s3-source-release
branch
from
fcad7e7 to
a31b0ca
Compare
added 7 commits
December 10, 2024 10:18
# Conflicts: # s3-source-connector/src/main/java/io/aiven/kafka/connect/s3/source/S3SourceTask.java # s3-source-connector/src/main/java/io/aiven/kafka/connect/s3/source/utils/FileReader.java # s3-source-connector/src/main/java/io/aiven/kafka/connect/s3/source/utils/RecordProcessor.java # s3-source-connector/src/main/java/io/aiven/kafka/connect/s3/source/utils/SourceRecordIterator.java # s3-source-connector/src/test/java/io/aiven/kafka/connect/s3/source/utils/RecordProcessorTest.java
Claudenw
force-pushed
the
KCON-51_S3ObjectSummaryIterator_improvements
branch
from
6c4e125 to
6d33432
Compare
…S3ObjectSummaryIterator_improvements
| * @param request | ||
| * the request object that defines the starting position for the object summary retrieval. | ||
| */ | ||
| @SuppressFBWarnings(value = "EI_EXPOSE_REP2", justification = "stores mutable AmazeonS3 and ListObjectsV2Request objects") |
There was a problem hiding this comment.
NIT: Amazon misspelt
| /** the ObjectRequest initially to start the iteration from later to retrieve more records */ | ||
| private final ListObjectsV2Request request; | ||
|
|
||
| /** The last key seen by this process. This allows us to restart when a new file is dropped in the direcotry */ |
There was a problem hiding this comment.
I would say not necessarily when a new file is dropped, more like just let us query S3 from after the last known processed file.
There was a problem hiding this comment.
maybe a nit, I just think wording is misleading.
| objectListing = s3Client.listObjectsV2(request); | ||
| innerIterator = objectListing.getObjectSummaries().iterator(); | ||
| } | ||
| if (!this.innerIterator.hasNext()) { |
There was a problem hiding this comment.
This is a nice efficiency brought to the iterator so we only call when we run out of object summaries 👍
| private static final String PATTERN_TOPIC_KEY = "topicName"; | ||
| private static final String PATTERN_PARTITION_KEY = "partitionId"; | ||
|
|
||
| private static final Pattern FILE_DEFAULT_PATTERN = Pattern.compile("(?<topicName>[^/]+?)-" |
There was a problem hiding this comment.
I think we need to create an issue for how this is hardcoded and how we are handling it, not sure if @RyanSkraba has this handled in his work
|
This PR is outdated and subsumed into other changes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changed iterator inside FileReader to prepare to handle offsets and other reset operations.
Reduced the custom code overhead.
Added ability to use Predicates to filter S3ObjectSummary objects.