Skip to content

Commit

Permalink
Add supervisor actions (#16276)
Browse files Browse the repository at this point in the history
* Add supervisor actions

* Update text

* Update text

* Update after review

* Update after review
  • Loading branch information
ektravel authored Apr 24, 2024
1 parent 6bca406 commit ceb6646
Show file tree
Hide file tree
Showing 3 changed files with 74 additions and 23 deletions.
13 changes: 7 additions & 6 deletions docs/api-reference/supervisor-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import TabItem from '@theme/TabItem';
-->

This topic describes the API endpoints to manage and monitor supervisors for Apache Druid.
The topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.

In this topic, `http://ROUTER_IP:ROUTER_PORT` is a placeholder for your Router service address and port. Replace it with the information for your deployment. For example, use `http://localhost:8888` for quickstart deployments.

Expand Down Expand Up @@ -3247,13 +3248,13 @@ Host: http://ROUTER_IP:ROUTER_PORT

The supervisor must be running for this endpoint to be available.

Resets the specified supervisor. This endpoint clears all stored offsets in Kafka or sequence numbers in Kinesis, prompting the supervisor to resume data reading. The supervisor restarts from the earliest or latest available position, depending on the platform: offsets in Kafka or sequence numbers in Kinesis.
After clearing all stored offsets in Kafka or sequence numbers in Kinesis, the supervisor kills and recreates active tasks,
Resets the specified supervisor. This endpoint clears supervisor metadata, prompting the supervisor to resume data reading. The supervisor restarts from the earliest or latest available position, depending on the value of the `useEarliestOffset` property.
After clearing all stored offsets, the supervisor kills and recreates active tasks,
so that tasks begin reading from valid positions.

Use this endpoint to recover from a stopped state due to missing offsets in Kafka or sequence numbers in Kinesis. Use this endpoint with caution as it may result in skipped messages and lead to data loss or duplicate data.
Use this endpoint to recover from a stopped state due to missing offsets. Use this endpoint with caution as it may result in skipped messages and lead to data loss or duplicate data.

The indexing service keeps track of the latest persisted offsets in Kafka or sequence numbers in Kinesis to provide exactly-once ingestion guarantees across tasks. Subsequent tasks must start reading from where the previous task completed for Druid to accept the generated segments. If the messages at the expected starting offsets in Kafka or sequence numbers in Kinesis are no longer available, the supervisor refuses to start and in-flight tasks fail. Possible causes for missing messages include the message retention period elapsing or the topic being removed and re-created. Use the `reset` endpoint to recover from this condition.
The indexing service keeps track of the latest persisted offsets to provide exactly-once ingestion guarantees across tasks. Subsequent tasks must start reading from where the previous task completed for Druid to accept the generated segments. If the messages at the expected starting offsets are no longer available, the supervisor refuses to start and in-flight tasks fail. Possible causes for missing messages include the message retention period elapsing or the topic being removed and re-created. Use the `reset` endpoint to recover from this condition.

#### URL

Expand Down Expand Up @@ -3322,7 +3323,7 @@ The supervisor must be running for this endpoint to be available.

Resets the specified offsets for partitions without resetting the entire set.

This endpoint clears only the specified offsets in Kafka or sequence numbers in Kinesis, prompting the supervisor to resume reading data from the specified offsets.
This endpoint clears only the stored offsets, prompting the supervisor to resume reading data from the specified offsets.
If there are no stored offsets, the specified offsets are set in the metadata store.

After resetting stored offsets, the supervisor kills and recreates any active tasks pertaining to the specified partitions,
Expand Down Expand Up @@ -3429,7 +3430,7 @@ when the supervisor's tasks restart, they resume reading from `{"0": 100, "1": 1

### Terminate a supervisor

Terminates a supervisor and its associated indexing tasks, triggering the publishing of their segments. When terminated, a tombstone marker is placed in the database to prevent reloading on restart.
Terminates a supervisor and its associated indexing tasks, triggering the publishing of their segments. When you terminate a supervisor, Druid places a tombstone marker in the metadata store to prevent reloading on restart.

The terminated supervisor still exists in the metadata store and its history can be retrieved.

Expand Down
Binary file added docs/assets/supervisor-actions.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading

0 comments on commit ceb6646

Please sign in to comment.