[Pull-based Ingestion][WIP] Introduce the new pull-based ingestion engine, APIs, and Kafka plugin #16958
Conversation
yupeng9
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
dblock,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli,
shwetathareja,
sohami,
VachaShah,
jainankitk and
linuxpi
as code owners
github-actions
bot
added
enhancement
|
❌ Gradle check result for 16dd9d0: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
| String clientId = engineConfig.getIndexSettings().getNodeName() | ||
| + "-" | ||
| + engineConfig.getIndexSettings().getIndex().getName() | ||
| + "-" | ||
| + engineConfig.getShardId().getId(); |
There was a problem hiding this comment.
Should we use ids instead of names like index uuid, node id etc
There was a problem hiding this comment.
this is mainly for monitoring and operation, for example, kafka supports quota set by client-id. as long as we can uniquely identify a streaming consumer, it's sufficient. any suggestion?
There was a problem hiding this comment.
Curious how would the FGAC security model work, espl with security plugin which intercepts transport actions to validate if authorised users can perform bulk actions on certain indices. Is the intent to handle permissions at a Kafka "partition level"
Another aspect is maintaining Kafka checkpoints durably, I'm yet to read that part but would be good to understand how are we handling fail overs and recoveries
server/src/main/java/org/opensearch/plugins/IngestionConsumerPlugin.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/ingest/package-info.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/indices/ingest/StreamPoller.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/index/engine/IngestionEngine.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
…cessing Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
Signed-off-by: Yupeng Fu <[email protected]>
yupeng9
force-pushed
the
pull-ingestion
branch
from
ad8a00f to
6d86683
Compare
|
❌ Gradle check result for 6d86683: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Description
This PR implements the basics of the pull-based ingestion described in this RFC, including:
Currently WIP, and there are a few improvements to make and test coverage to increase
Related Issues
Resolves #16927 #16929 #16928
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.