Skip to content

Commit

Permalink
Add General Guidelines for concurrent segment search
Browse files Browse the repository at this point in the history 
Signed-off-by: Jay Deng <[email protected]>
  • Loading branch information
@jed326
jed326 authored and Jay Deng committed Jun 17, 2024
1 parent 90ae402 commit a6a7e8a
Showing 1 changed file with 8 additions and 0 deletions.
8 changes: 8 additions & 0 deletions _search-plugins/concurrent-segment-search.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,14 @@ The `search.concurrent.max_slice_count` setting can take the following valid val
- `0`: Use the default Lucene mechanism.
- Positive integer: Use the max target slice count mechanism. Usually, a value between 2 and 8 should be sufficient.

## General Guidelines

Check failure on line 85 in _search-plugins/concurrent-segment-search.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'General Guidelines' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'General Guidelines' is a heading and should be in sentence case.", "location": {"path": "_search-plugins/concurrent-segment-search.md", "range": {"start": {"line": 85, "column": 4}}}, "severity": "ERROR"}
Concurrent segment search helps to improve the performance of search requests at the cost of consuming more resources such as CPU, JVM, etc. It is important to try your workload to understand if the domain is sized correctly for concurrent segment search or not. We recommend the following guidelines to follow to enable concurrent segment search:

Check warning on line 86 in _search-plugins/concurrent-segment-search.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.LatinismsElimination] Using 'etc.' is unnecessary. Remove.

Raw Output:
{"message": "[OpenSearch.LatinismsElimination] Using 'etc.' is unnecessary. Remove.", "location": {"path": "_search-plugins/concurrent-segment-search.md", "range": {"start": {"line": 86, "column": 137}}}, "severity": "WARNING"}

* Start with a slice count of 2 and measure the performance of your workload. If resource utilization is exceeding the recommended values then consider scaling your cluster. Generally based on our testing we have observed that if your workload is already consuming more than 50% of the CPU resources, then you will need to scale your domain for concurrent segment search.
* With a slice count of 2 if you still have available resources in the domain, then you can increase the slice count to higher number such as 4 or 6 and observe the search latency along with resource utilization on the domain.
* With a large number of clients sending search requests in parallel, usually a lower slice count works better. This can again be reflected in the CPU utilization, since more clients means higher TPS which will translate to higher usage of resources.


## Limitations

The following aggregations do not support the concurrent search model. If a search request contains one of these aggregations, the request will be executed using the non-concurrent path even if concurrent segment search is enabled at the cluster level or index level.
Expand Down

0 comments on commit a6a7e8a

Please sign in to comment.