Adding documentation for search-shard-routing opensearch-project#7507

Signed-off-by: AntonEliatra <[email protected]>

_search-plugins/search-shard-routing.md

@@ -0,0 +1,164 @@
+---
+layout: default
+title: Search shard routing
+nav_order: 210
+---
+
+# Search shard routing
+
+To ensure redundancy and improve search performance, OpenSearch replicates index data across various shards located on different nodes. When a search query is executed, OpenSearch routes the request to a node containing either primary or replica shard of the index. This technique is known as _search shard routing_.
+
+## Adaptive replica selection
+
+In OpenSearch, in order to improve latency, search requests are routed using _Adaptive replica selection_, which chooses appropriate nodes based on number of factors, such as:
+
+- The time it took particular node to run previous requests.
+- Latency between coordinating node and selected node.
+- The size of the queue of node's search threadpool.
+
+You can turn off this feature using following curl command with user which has permissions to call REST APIs, mode details on REST API can be found at [REST management API settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/security-settings/#rest-management-api-settings):
+
+```json
+curl -X PUT "https://<opensearch-host>:<port>/_cluster/settings" -u admin:<password> -H "Content-Type: application/json" -d '{
+  "persistent": {
+    "cluster.routing.use_adaptive_replica_selection": false
+  }
+}'
+```
+
+In this case, OpenSearch routing will use _round-robin_ which can impact the search latency negatively.
+{: .note}
+
+## Selecting nodes/shards for searches
+
+OpenSearch uses all nodes to choose best routing for search requests, however, there might be reasons why you want to manually select which nodes or shards the search request is sent to, possible reasons could be:
+
+- Using caching of previous searches.
+- Dedicating specific hardware for searches.
+- Using only local nodes for searches.
+
+You can use the _preference_ parameter in the search query to indicate the destination for the search, see example below:
+
+1. `_primary`: Forces the search to execute only on primary shards.
+```
+GET /my-index/_search?preference=_primary
+```
+2. `_primary_first`: Prefers primary shards but will use replica shards if the primary is not available.
+```
+GET /my-index/_search?preference=_primary_first
+```
+3. `_replica`: Forces the search to execute only on replica shards.
+```
+GET /my-index/_search?preference=_replica
+```
+4. `_replica_first`: Prefers replica shards but will use primary shards if no replicas are available.
+```
+GET /my-index/_search?preference=_replica_first
+```
+5. `_only_nodes:<node-id>,<node-id>`: Limits the search to execute only on specific nodes by their IDs.
+```
+GET /my-index/_search?preference=_only_nodes:node-1,node-2
+```
+6. `_prefer_nodes:<node-id>,<node-id>`: Prefers to execute the search on specific nodes but will use other nodes if the preferred ones are not available.
+```
+GET /my-index/_search?preference=_prefer_nodes:node-1,node-2
+```
+7. `_shards:<shard-id>,<shard-id>`: Limits the search to specific shards.
+```
+GET /my-index/_search?preference=_shards:0,1
+```
+8. `_local`: Executes the search on the local node if the data is available there, which can reduce latency.
+```
+GET /my-index/_search?preference=_local
+```
+9. Custom string: You can use any custom string as the preference value. This custom string ensures that requests with the same string are routed to the same shards consistently, which can be useful for caching.
+```
+GET /my-index/_search?preference=custom_string
+```
+
+## Custom routing during index and search
+
+### Indexing with routing
+When you index a document, OpenSearch calculates a hash of the routing value and uses this hash to determine the shard on which the document will be stored. If you don't specify a routing value, OpenSearch uses the document's ID to calculate the hash.
+
+Following is an example index operation using a routing value:
+```
+curl -X POST "https://<opensearch-host>:<port>/index1/_doc/1?routing=user1" -H 'Content-Type: application/json' -d'
+{
+  "name": "John Doe",
+  "age": 20
+}
+'
+```
+In the above example, the document with ID 1 is indexed with the routing value user1. All documents with the same routing value will be stored on the same shard.
+
+### Searching with routing
+
+When you search for documents, specifying the same routing value ensures that the search request is routed to the appropriate shard. This can significantly improve performance by reducing the number of shards that need to be queried.
+
+Here’s an example of searching with a specific routing value:
+```
+curl -X GET "https://<opensearch-host>:<port>/index1/_search?routing=user1" -H 'Content-Type: application/json' -d'
+{
+  "query": {
+    "match": {
+      "name": "John Doe"
+    }
+  }
+}
+'
+```
+In the above example, the search query is routed to the shard that contains documents indexed with the routing value user1.
+
+## Concurrent shard request
+
+Hitting a large number of shards simultaneously during a search can significantly impact the CPU and memory consumption. By default, OpenSearch does not reject these requests, however there are number of methods that can be used to mitigate this risk.
+
+### max_concurrent_shard_requests
+
+You can use parameter `max_concurrent_shard_requests` with the search request to limit it to maximum number of shards that can be queried, see the following example:
+
+```
+curl -X GET "https://<opensearch-host>:<port>/index1/_search?max_concurrent_shard_requests=12" -u admin:<password> -H 'Content-Type: application/json' -d'
+{
+  "query": {
+    "match_all": {}
+  }
+}
+'
+```
+
+### action.search.shard_count.limit
+
+You can define `action.search.shard_count.limit` in `opensearch.yml` file or dynamically using cluster settings. Any search request that attempts to query more than 1000 shards will be rejected with an error. This helps prevent a single search request from consuming too many resources, which can degrade the performance of the entire cluster. See example below:
+
+```
+curl -X PUT "https://<opensearch-host>:<port>/_cluster/settings" -u admin:<password> -H 'Content-Type: application/json' -d'
+{
+  "transient": {
+    "action.search.shard_count.limit": 1000
+  }
+}
+```
+
+### Search thread pool
+
+OpenSearch uses thread pools to manage the execution of various tasks, including search operations. The search thread pool is specifically used for search requests. You can adjust the size and queue capacity of the search thread pool by adding the following configuration to `opensearch.yml`
+file:
+```
+thread_pool.search.size: 100
+thread_pool.search.queue_size: 1000
+```
+
+#### How They Work Together
+
+**Thread Assignment**: If there are available threads in the search thread pool, the request is immediately assigned to a thread and begins processing.
+
+**Queueing**: If all threads in the search thread pool are busy, the request is placed in the queue.
+
+**Rejection**: If the queue is full (i.e., the number of queued requests reaches the queue_size limit), additional incoming search requests are rejected until there is space available in the queue.
+
+You can check the currently configured search thread pool by running the following curl request:
+```
+curl -X GET "https://localhost:9200/_cat/thread_pool/search?v&h=id,name,active,rejected,completed,size,queue_size" -u admin:<password>
+```