diff --git a/docs/api-reference/dynamic-configuration-api.md b/docs/api-reference/dynamic-configuration-api.md index e725eea5d698..b884ca6b955c 100644 --- a/docs/api-reference/dynamic-configuration-api.md +++ b/docs/api-reference/dynamic-configuration-api.md @@ -3,8 +3,12 @@ id: dynamic-configuration-api title: Dynamic configuration API sidebar_label: Dynamic configuration --- +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + + - -This document describes the API endpoints to retrieve and manage the dynamic configurations for the [Coordinator](../configuration/index.md#overlord-dynamic-configuration) and [Overlord](../configuration/index.md#dynamic-configuration) in Apache Druid. +This document describes the API endpoints to retrieve and manage dynamic configurations for the [Coordinator](../design/coordinator.md) and [Overlord](../design/overlord.md) in Apache Druid. -In this document, `http://SERVICE_IP:SERVICE_PORT` is a placeholder for the server address of deployment and the service port. For example, on the quickstart configuration, replace `http://ROUTER_IP:ROUTER_PORT` with `http://localhost:8888`. +In this topic, `http://ROUTER_IP:ROUTER_PORT` is a placeholder for your Router service address and port. +Replace it with the information for your deployment. +For example, use `http://localhost:8888` for quickstart deployments. ## Coordinator dynamic configuration -The Coordinator has dynamic configurations to tune certain behavior on the fly, without requiring a service restart. For more information, see [Coordinator Dynamic Configuration](../configuration/index.md#dynamic-configuration). +The Coordinator has dynamic configurations to tune certain behavior on the fly, without requiring a service restart. +For information on the supported properties, see [Coordinator dynamic configuration](../configuration/index.md#dynamic-configuration). + +### Get dynamic configuration + +Retrieves the current Coordinator dynamic configuration. Returns a JSON object with the dynamic configuration properties and values. + +#### URL + +GET /druid/coordinator/v1/config + +#### Responses + + + + + + +*Successfully retrieved dynamic configuration* + + + + +--- + +#### Sample request + + + + + + +```shell +curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config" +``` + + + + + +```HTTP +GET /druid/coordinator/v1/config HTTP/1.1 +Host: http://ROUTER_IP:ROUTER_PORT +``` + + + + +#### Sample response + +
+Click to show sample response + +```json +{ + "millisToWaitBeforeDeleting": 900000, + "mergeBytesLimit": 524288000, + "mergeSegmentsLimit": 100, + "maxSegmentsToMove": 100, + "replicantLifetime": 15, + "replicationThrottleLimit": 500, + "balancerComputeThreads": 1, + "killDataSourceWhitelist": [], + "killPendingSegmentsSkipList": [], + "maxSegmentsInNodeLoadingQueue": 500, + "decommissioningNodes": [], + "decommissioningMaxPercentOfMaxSegmentsToMove": 70, + "pauseCoordination": false, + "replicateAfterLoadTimeout": false, + "maxNonPrimaryReplicantsToLoad": 2147483647, + "useRoundRobinSegmentAssignment": true, + "smartSegmentLoading": true, + "debugDimensions": null +} +``` + +
-### Update Coordinator dynamic configuration +### Update dynamic configuration -Update Coordinator dynamic worker configuration. Pass the dynamic configuration spec as a JSON request body. For information on constructing a dynamic configuration spec, see [Dynamic configuration](../configuration/index.md#dynamic-configuration). +Submits a JSON-based dynamic configuration spec to the Coordinator. +For information on the supported properties, see [Dynamic configuration](../configuration/index.md#dynamic-configuration). #### URL POST /druid/coordinator/v1/config +#### Header parameters + +The endpoint supports a set of optional header parameters to populate the `author` and `comment` fields in the configuration history. + +* `X-Druid-Author` + * Type: String + * A string representing the author making the configuration change. +* `X-Druid-Comment` + * Type: String + * A string describing the update. + #### Responses - + + + - -*Successfully updated dynamic configuration* +*Successfully updated dynamic configuration* - + + --- #### Sample request - + + + - ```shell curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config" \ @@ -83,7 +180,9 @@ curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config" \ }' ``` - + + + ```HTTP POST /druid/coordinator/v1/config HTTP/1.1 @@ -114,50 +213,72 @@ Content-Length: 683 } ``` - + + #### Sample response A successful request returns an HTTP `200 OK` message code and an empty response body. -### Get Coordinator dynamic configuration +### Get dynamic configuration history -Retrieves current Coordinator dynamic configuration. Returns a JSON object with the dynamic configuration properties and values. For information on the response properties, see [Dynamic configuration](../configuration/index.md#dynamic-configuration). +Retrieves the history of changes to Coordinator dynamic configuration over an interval of time. Returns an empty array if there are no history records available. #### URL -GET /druid/coordinator/v1/config +GET /druid/coordinator/v1/config/history + +#### Query parameters + +The endpoint supports a set of optional query parameters to filter results. + +* `interval` + * Type: String (ISO-8601) + * Limit the results to the specified time interval. Delimited with `/`. For example, `2023-07-13/2023-07-19`. You can specify the default value of `interval` by setting `druid.audit.manager.auditHistoryMillis` in Coordinator `runtime.properties`. If not specified, `interval` defaults to one week. + +* `count` + * Type: Integer + * Limit the number of results to the last `n` entries. #### Responses - + - + -*Successfully retrieved dynamic configuration* - +*Successfully retrieved history* + + + + --- #### Sample request - +The following example retrieves the dynamic configuration history between `2022-07-13` and `2024-07-19`. The response is limited to 10 entries. + + + + - ```shell -curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config" +curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/history?interval=2022-07-13%2F2024-07-19&count=10" ``` - + + + ```HTTP -GET /druid/coordinator/v1/config HTTP/1.1 +GET /druid/coordinator/v1/config/history?interval=2022-07-13/2024-07-19&count=10 HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` - + + #### Sample response @@ -165,81 +286,71 @@ Host: http://ROUTER_IP:ROUTER_PORT Click to show sample response ```json -{ - "millisToWaitBeforeDeleting": 900000, - "mergeBytesLimit": 524288000, - "mergeSegmentsLimit": 100, - "maxSegmentsToMove": 5, - "percentOfSegmentsToConsiderPerMove": 100.0, - "useBatchedSegmentSampler": true, - "replicantLifetime": 15, - "replicationThrottleLimit": 10, - "balancerComputeThreads": 1, - "emitBalancingStats": false, - "killDataSourceWhitelist": [], - "killPendingSegmentsSkipList": [], - "maxSegmentsInNodeLoadingQueue": 100, - "decommissioningNodes": [], - "decommissioningMaxPercentOfMaxSegmentsToMove": 70, - "pauseCoordination": false, - "replicateAfterLoadTimeout": false, - "maxNonPrimaryReplicantsToLoad": 2147483647, - "useRoundRobinSegmentAssignment": true -} +[ + { + "key": "coordinator.config", + "type": "coordinator.config", + "auditInfo": { + "author": "", + "comment": "", + "ip": "127.0.0.1" + }, + "payload": "{\"millisToWaitBeforeDeleting\":900000,\"mergeBytesLimit\":524288000,\"mergeSegmentsLimit\":100,\"maxSegmentsToMove\":5,\"replicantLifetime\":15,\"replicationThrottleLimit\":10,\"balancerComputeThreads\":1,\"killDataSourceWhitelist\":[],\"killPendingSegmentsSkipList\":[],\"maxSegmentsInNodeLoadingQueue\":100,\"decommissioningNodes\":[],\"decommissioningMaxPercentOfMaxSegmentsToMove\":70,\"pauseCoordination\":false,\"replicateAfterLoadTimeout\":false,\"maxNonPrimaryReplicantsToLoad\":2147483647,\"useRoundRobinSegmentAssignment\":true,\"smartSegmentLoading\":true,\"debugDimensions\":null}", + "auditTime": "2023-10-03T20:59:51.622Z" + } +] ``` -### Get Coordinator dynamic configuration history +## Overlord dynamic configuration -Retrieves the history of changes to Coordinator dynamic configuration over an interval of time. Returns an empty array if there are no history records available. +The Overlord has dynamic configurations to tune how Druid assigns tasks to workers. +For information on the supported properties, see [Overlord dynamic configuration](../configuration/index.md#overlord-dynamic-configuration). -#### URL +### Get dynamic configuration -GET /druid/coordinator/v1/config/history +Retrieves the current Overlord dynamic configuration. +Returns a JSON object with the dynamic configuration properties and values. +Returns an empty response body if there is no current Overlord dynamic configuration. -#### Query parameters - -* `interval` (optional) - * Type: ISO 8601 - * Limits the number of results to the specified time interval. Delimited with `/`. For example, `2023-07-13/2023-07-19`. +#### URL -* `count` (optional) - * Type: Int - * Limits the number of results to the last `n` entries. +GET /druid/indexer/v1/worker #### Responses - - - + -*Successfully retrieved history* + - +*Successfully retrieved dynamic configuration* ---- + + #### Sample request -The following example retrieves the dynamic configuration history between `2022-07-13` and `2024-07-19`. The response is limited to 10 entries. + - + - ```shell -curl "http://ROUTER_IP:ROUTER_PORT/druid/coordinator/v1/config/history?interval=2022-07-13%2F2024-07-19&count=10" +curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker" ``` - + + + ```HTTP -GET /druid/coordinator/v1/config/history?interval=2022-07-13/2024-07-19&count=10 HTTP/1.1 +GET /druid/indexer/v1/worker HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` - + + #### Sample response @@ -247,29 +358,25 @@ Host: http://ROUTER_IP:ROUTER_PORT Click to show sample response ```json -[ - { - "key": "coordinator.config", - "type": "coordinator.config", - "auditInfo": { - "author": "console", - "comment": "testing config change", - "ip": "127.0.0.1" - }, - "payload": "{\"millisToWaitBeforeDeleting\":900000,\"mergeBytesLimit\":524288000,\"mergeSegmentsLimit\":100,\"maxSegmentsToMove\":5,\"percentOfSegmentsToConsiderPerMove\":100.0,\"useBatchedSegmentSampler\":true,\"replicantLifetime\":15,\"replicationThrottleLimit\":10,\"balancerComputeThreads\":1,\"emitBalancingStats\":false,\"killDataSourceWhitelist\":[],\"killPendingSegmentsSkipList\":[],\"maxSegmentsInNodeLoadingQueue\":100,\"decommissioningNodes\":[],\"decommissioningMaxPercentOfMaxSegmentsToMove\":70,\"pauseCoordination\":false,\"replicateAfterLoadTimeout\":false,\"maxNonPrimaryReplicantsToLoad\":2147483647,\"useRoundRobinSegmentAssignment\":true}", - "auditTime": "2023-08-12T07:51:36.306Z" - } -] +{ + "type": "default", + "selectStrategy": { + "type": "fillCapacityWithCategorySpec", + "workerCategorySpec": { + "categoryMap": {}, + "strong": true + } + }, + "autoScaler": null +} ``` - - -## Overlord dynamic configuration -The Overlord can be dynamically configured to specify how tasks are assigned to workers. For more information, see [Overlord dynamic configuration](../configuration/index.md#overlord-dynamic-configuration). + -### Update Overlord dynamic configuration. +### Update dynamic configuration -Updates overlord dynamic worker configuration. Pass the dynamic configuration spec as a JSON request body. For information on constructing an Overlord dynamic configuration spec, see [Overlord dynamic configuration](../configuration/index.md#overlord-dynamic-configuration). +Submits a JSON-based dynamic configuration spec to the Overlord. +For information on the supported properties, see [Overlord dynamic configuration](../configuration/index.md#overlord-dynamic-configuration). #### URL @@ -277,32 +384,35 @@ Updates overlord dynamic worker configuration. Pass the dynamic configuration sp #### Header parameters -The endpoint supports a set of optional header parameters to populate the `author` and `comment` fields in the configuration history. +The endpoint supports a set of optional header parameters to populate the `author` and `comment` fields in the configuration history. -* `X-Druid-Author` (optional) +* `X-Druid-Author` * Type: String * A string representing the author making the configuration change. -* `X-Druid-Comment` (optional) +* `X-Druid-Comment` * Type: String * A string describing the update. #### Responses - + + + - -*Successfully updated dynamic configuration* +*Successfully updated dynamic configuration* - + + --- #### Sample request - + + + - ```shell curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker" \ @@ -320,7 +430,9 @@ curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker" \ }' ``` - + + + ```HTTP POST /druid/indexer/v1/worker HTTP/1.1 @@ -341,98 +453,45 @@ Content-Length: 196 } ``` - + + #### Sample response A successful request returns an HTTP `200 OK` message code and an empty response body. -### Get Overlord dynamic configuration - -Returns current Overlord dynamic configuration. Returns an empty response body if there are no current Overlord dynamic configuration. - -#### URL - -GET /druid/indexer/v1/worker - -#### Responses - - - - - -*Successfully retrieved dynamic configuration* - - - -#### Sample request - - - - - -```shell -curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker" -``` - - - -```HTTP -GET /druid/indexer/v1/worker HTTP/1.1 -Host: http://ROUTER_IP:ROUTER_PORT -``` - - - -#### Sample response - -
- Click to show sample response - -```json -{ - "type": "default", - "selectStrategy": { - "type": "fillCapacityWithCategorySpec", - "workerCategorySpec": { - "categoryMap": {}, - "strong": true - } - }, - "autoScaler": null -} -``` -
- - -### Get Overlord dynamic configuration history - -Retrieves history of changes to Overlord dynamic configuration. Returns an empty array if there are no history records available. +### Get dynamic configuration history +Retrieves the history of changes to Overlord dynamic configuration over an interval of time. Returns an empty array if there are no history records available. #### URL GET /druid/indexer/v1/worker/history + #### Query parameters -* `interval` (optional) - * Type: ISO 8601 - * Limits the number of results to the specified time interval. Delimited with `/`. For example, `2023-07-13/2023-07-19`. +The endpoint supports a set of optional query parameters to filter results. -* `count` (optional) - * Type: Int - * Limits the number of results to the last `n` entries. +* `interval` + * Type: String (ISO-8601) + * Limit the results to the specified time interval. Delimited with `/`. For example, `2023-07-13/2023-07-19`. You can specify the default value of `interval` by setting `druid.audit.manager.auditHistoryMillis` in Overlord `runtime.properties`. If not specified, `interval` defaults to one week. + +* `count` + * Type: Integer + * Limit the number of results to the last `n` entries. #### Responses - + + + - -*Successfully retrieved history* +*Successfully retrieved history* - + + --- @@ -440,22 +499,26 @@ Retrieves history of changes to Overlord dynamic configuration. Returns an empty The following example retrieves the dynamic configuration history between `2022-07-13` and `2024-07-19`. The response is limited to 10 entries. - + + + - ```shell curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/worker/history?interval=2022-07-13%2F2024-07-19&count=10" ``` - + + + ```HTTP GET /druid/indexer/v1/worker/history?interval=2022-07-13%2F2024-07-19&count=10 HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` - + + #### Sample response @@ -473,10 +536,11 @@ Host: http://ROUTER_IP:ROUTER_PORT "ip": "127.0.0.1" }, "payload": "{\"type\":\"default\",\"selectStrategy\":{\"type\":\"fillCapacityWithCategorySpec\",\"workerCategorySpec\":{\"categoryMap\":{},\"strong\":true}},\"autoScaler\":null}", - "auditTime": "2023-08-15T20:32:06.899Z" + "auditTime": "2023-10-03T21:49:49.991Z" } ] ``` + ### Get an array of worker nodes in the cluster @@ -487,34 +551,40 @@ Returns an array of all the worker nodes in the cluster along with its correspon #### Responses - + - + -*Successfully retrieved worker nodes* - +*Successfully retrieved worker nodes* + + + --- #### Sample request - + + + - ```shell curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/workers" ``` - + + + ```HTTP GET /druid/indexer/v1/workers HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` - + + #### Sample response @@ -527,7 +597,7 @@ Host: http://ROUTER_IP:ROUTER_PORT "worker": { "scheme": "http", "host": "localhost:8091", - "ip": "192.168.1.4", + "ip": "198.51.100.0", "capacity": 2, "version": "0", "category": "_default_worker_category" @@ -536,16 +606,18 @@ Host: http://ROUTER_IP:ROUTER_PORT "currParallelIndexCapacityUsed": 0, "availabilityGroups": [], "runningTasks": [], - "lastCompletedTaskTime": "2023-08-15T18:19:37.254Z", + "lastCompletedTaskTime": "2023-09-29T19:13:05.505Z", "blacklistedUntil": null } ] ``` + ### Get scaling events Returns Overlord scaling events if auto-scaling runners are in use. +Returns an empty response body if there are no Overlord scaling events. #### URL @@ -553,41 +625,41 @@ Returns Overlord scaling events if auto-scaling runners are in use. #### Responses - + - + -*Successfully retrieved scaling events* - +*Successfully retrieved scaling events* + + + --- #### Sample request - + + + - ```shell curl "http://ROUTER_IP:ROUTER_PORT/druid/indexer/v1/scaling" ``` - + + + ```HTTP GET /druid/indexer/v1/scaling HTTP/1.1 Host: http://ROUTER_IP:ROUTER_PORT ``` - + + #### Sample response -
- Click to show sample response - -```json - -``` -
\ No newline at end of file +A successful request returns a `200 OK` response and an array of scaling events.