-
Notifications
You must be signed in to change notification settings - Fork 506
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* add kstem token filter docs #8150 Signed-off-by: Anton Rubin <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Anton Rubin <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit ac852d6) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
- Loading branch information
1 parent
994ac78
commit 5627929
Showing
2 changed files
with
93 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,92 @@ | ||
--- | ||
layout: default | ||
title: KStem | ||
parent: Token filters | ||
nav_order: 220 | ||
--- | ||
|
||
# KStem token filter | ||
Check failure on line 8 in _analyzers/token-filters/kstem.md GitHub Actions / vale[vale] _analyzers/token-filters/kstem.md#L8
Raw output
Check failure on line 8 in _analyzers/token-filters/kstem.md GitHub Actions / vale[vale] _analyzers/token-filters/kstem.md#L8
Raw output
|
||
|
||
The `kstem` token filter is a stemming filter used to reduce words to their root forms. The filter is a lightweight algorithmic stemmer designed for the English language that performs the following stemming operations: | ||
|
||
- Reduces plurals to their singular form. | ||
- Converts different verb tenses to their base form. | ||
- Removes common derivational endings, such as "-ing" or "-ed". | ||
|
||
The `kstem` token filter is equivalent to the a `stemmer` filter configured with a `light_english` language. It provides a more conservative stemming compared to other stemming filters like `porter_stem`. | ||
|
||
The `kstem` token filter is based on the Lucene KStemFilter. For more information, see the [Lucene documentation](https://lucene.apache.org/core/9_10_0/analysis/common/org/apache/lucene/analysis/en/KStemFilter.html). | ||
|
||
## Example | ||
|
||
The following example request creates a new index named `my_kstem_index` and configures an analyzer with a `kstem` filter: | ||
|
||
```json | ||
PUT /my_kstem_index | ||
{ | ||
"settings": { | ||
"analysis": { | ||
"filter": { | ||
"kstem_filter": { | ||
"type": "kstem" | ||
} | ||
}, | ||
"analyzer": { | ||
"my_kstem_analyzer": { | ||
"type": "custom", | ||
"tokenizer": "standard", | ||
"filter": [ | ||
"lowercase", | ||
"kstem_filter" | ||
] | ||
} | ||
} | ||
} | ||
}, | ||
"mappings": { | ||
"properties": { | ||
"content": { | ||
"type": "text", | ||
"analyzer": "my_kstem_analyzer" | ||
} | ||
} | ||
} | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
## Generated tokens | ||
|
||
Use the following request to examine the tokens generated using the analyzer: | ||
|
||
```json | ||
POST /my_kstem_index/_analyze | ||
{ | ||
"analyzer": "my_kstem_analyzer", | ||
"text": "stops stopped" | ||
} | ||
``` | ||
{% include copy-curl.html %} | ||
|
||
The response contains the generated tokens: | ||
|
||
```json | ||
{ | ||
"tokens": [ | ||
{ | ||
"token": "stop", | ||
"start_offset": 0, | ||
"end_offset": 5, | ||
"type": "<ALPHANUM>", | ||
"position": 0 | ||
}, | ||
{ | ||
"token": "stop", | ||
"start_offset": 6, | ||
"end_offset": 13, | ||
"type": "<ALPHANUM>", | ||
"position": 1 | ||
} | ||
] | ||
} | ||
``` |