Update documentation for new AD settings #4835
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Jonah Calvo <[email protected]>
JonahCalvo
requested review from
cwillum,
hdhalter,
kolchfa-aws,
Naarcha-AWS,
vagimeli,
ananzh,
seanneumann,
AMoo-Miki and
natebower
as code owners
sudiptoguha
reviewed
Aug 18, 2023
| @@ -18,6 +18,10 @@ You can configure the anomaly detector processor by specifying a key and the opt | |||
| | :--- | :--- | :--- | | |||
| | `keys` | Yes | A non-ordered `List<String>` that is used as input to the ML algorithm to detect anomalies in the values of the keys in the list. At least one key is required. | |||
| | `mode` | Yes | The ML algorithm (or model) used to detect anomalies. You must provide a mode. See [random_cut_forest mode](#random_cut_forest-mode). | |||
| | `identification_keys` | No | If provided, anomalies will be detected within each unique instance of this key. For example, providing `ip` here will have anomalies detected seperately for each unique IP address. | |||
| | `cardinality_limit` | No | If using `identification_keys`, a new ML model will be created for every degree of cardinality. This can cause a large amount of memory usage, so setting a limit to the number of models is useful. Defaults to 5000. | |||
| | `verbose` | No | By default, the RCF algorithm will alert once on a level shift. For example, if latency is consistently 50-100 and jumps to consistently ~1000, only one anomaly will be detected. Setting `verbose` to `true` will alert many times for such a shift. | |||
There was a problem hiding this comment.
"RCF algorithm will alert once on a level shift" -> this is too strong. How about "RCF will try to auto learn and reduce the number of anomalies. For example if latency is consistently 50-100 and jumps to consistently ~1000, only the first few points after the transition will be detected (unless there are other spikes/anomalies). Likewise, for repeated spikes to the same level, RCF will likely eliminate many of the spikes after a few initial ones. The goal of this default setting is to minimize alerts. Setting verbose to true will alert consistently on these repeated cases and may be useful in detecting anomalous behavior that lasts an extended period of time." Please free to reword as well.
There was a problem hiding this comment.
Made this change, thanks for the suggestion.
Signed-off-by: Jonah Calvo <[email protected]>
|
Hey team, could I get an approval on this updated documentation? |
|
@vagimeli - can you please review this? Thank you! |
vagimeli
reviewed
Aug 31, 2023
_data-prepper/pipelines/configuration/processors/anomaly-detector.md
Outdated
Show resolved
Hide resolved
_data-prepper/pipelines/configuration/processors/anomaly-detector.md
Outdated
Show resolved
Hide resolved
_data-prepper/pipelines/configuration/processors/anomaly-detector.md
Outdated
Show resolved
Hide resolved
@JonahCalvo @hdhalter Please see doc review comments. Once changes are made, it'll be good to go. |
…tor.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]>
…tor.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]>
…tor.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]>
Signed-off-by: Jonah Calvo <[email protected]>
|
Sorry I missed this folks. Great comments, applied changes and removed unclear usage of 'few'. Thanks! |
sudiptoguha
approved these changes
Sep 21, 2023
vagimeli
approved these changes
Sep 21, 2023
harshavamsi
pushed a commit
to harshavamsi/documentation-website
that referenced
this pull request
Oct 31, 2023
* Update documentation for new AD settings Signed-off-by: Jonah Calvo <[email protected]> * update wording for verbose Signed-off-by: Jonah Calvo <[email protected]> * Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> * Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> * Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> * Remove 'few' from description Signed-off-by: Jonah Calvo <[email protected]> --------- Signed-off-by: Jonah Calvo <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> Co-authored-by: Melissa Vagi <[email protected]>
vagimeli
added a commit
that referenced
this pull request
Dec 21, 2023
* Update documentation for new AD settings Signed-off-by: Jonah Calvo <[email protected]> * update wording for verbose Signed-off-by: Jonah Calvo <[email protected]> * Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> * Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> * Update _data-prepper/pipelines/configuration/processors/anomaly-detector.md Co-authored-by: Melissa Vagi <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> * Remove 'few' from description Signed-off-by: Jonah Calvo <[email protected]> --------- Signed-off-by: Jonah Calvo <[email protected]> Signed-off-by: Jonah Calvo <[email protected]> Co-authored-by: Melissa Vagi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Updates the Anomaly Detection docs with new configuration options.
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.