Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update expression and anomaly detector documentation #8041

Merged
merged 14 commits into from
Dec 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ You can configure `random_cut_forest` mode with the following options.
| `sample_size` | `256` | 100--2500 | The sample size used in the ML algorithm. |
| `time_decay` | `0.1` | 0--1.0 | The time decay value used in the ML algorithm. Used as the mathematical expression `timeDecay` divided by `SampleSize` in the ML algorithm. |
| `type` | `metrics` | N/A | The type of data sent to the algorithm. |
| `output_after` | 32 | N/A | Specifies the number of events to process before outputting any detected anomalies. |
| `version` | `1.0` | N/A | The algorithm version number. |

## Usage
Expand Down
64 changes: 62 additions & 2 deletions _data-prepper/pipelines/expression-syntax.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
|----------------------|-------------------------------------------------------|---------------|
| `()` | Priority expression | Left to right |
| `not`<br> `+`<br> `-`| Unary logical NOT<br>Unary positive<br>Unary negative | Right to left |
| `*`, `/` | Multiplication and division operators | Left to right |
| `+`, `-` | Addition and subtraction operators | Left to right |
| `+` | String concatenation operator | Left to right |
| `<`, `<=`, `>`, `>=` | Relational operators | Left to right |
| `==`, `!=` | Equality operators | Left to right |
| `and`, `or` | Conditional expression | Left to right |
Expand Down Expand Up @@ -78,7 +81,6 @@
<Any> or <Any>
not <Any>
```
{% include copy-curl.html %}

The following are some example conditional expressions:

Expand All @@ -91,9 +93,64 @@
```
{% include copy-curl.html %}

### Arithmetic expressions

Arithmetic expressions enable basic mathematical operations like addition, subtraction, multiplication, and division. These expressions can be combined with conditional expressions to create more complex conditional statements. The available arithmetic operators are +, -, *, and /. The syntax for using the arithmetic operators is as follows:

```
<Any> + <Any>
<Any> - <Any>
<Any> * <Any>
<Any> / <Any>
```

The following are example arithmetic expressions:

```
/value + length(/message)
/bytes / 1024
/value1 - /value2
/TimeInSeconds * 1000
```
{% include copy-curl.html %}

The following are some example arithmetic expressions used in conditional expressions :
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following are some example arithmetic expressions used in conditional expressions :
The following are example arithmetic expressions that can be used in conditional expressions:


```
/value + length(/message) > 200
/bytes / 1024 < 10
/value1 - /value2 != /value3 + /value4
```
{% include copy-curl.html %}

### String concatenation expressions

String concatenation expressions enable you to combine strings to create new strings. These concatenated strings can also be used within conditional expressions. The syntax for using string concatenation is as follows:

```
<String Variable or String Literal> + <String Variable or String Literal>
```

The following are example string concatenation expressions:

```
/name + "suffix"
"prefix" + /name
"time of " + /timeInMs + " ms"
```
{% include copy-curl.html %}

The following are example string concatenation expressions that can be used in conditional expressions:

```
/service + ".com" == /url
"www." + /service != /url
```
{% include copy-curl.html %}

### Reserved symbols

Reserved symbols are symbols that are not currently used in the expression syntax but are reserved for possible future functionality or extensions. Reserved symbols include `^`, `*`, `/`, `%`, `+`, `-`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.
Certain symbols, such as ^, %, xor, =, +=, -=, *=, /=, %=, ++, --, and ${<text>}, are reserved for future functionality or extensions. Reserved symbols include `^`, `%`, `xor`, `=`, `+=`, `-=`, `*=`, `/=`, `%=`, `++`, `--`, and `${<text>}`.

## Syntax components

Expand Down Expand Up @@ -170,6 +227,9 @@
| `()` | Priority expression | Yes | `/a==(/b==200)`<br>`/a in ({200})` | `/status in({200})` |
| `in`, `not in` | Set operators | Yes | `/a in {200}`<br>`/a not in {400}` | `/a in{200, 202}`<br>`/a not in{400}` |
| `<`, `<=`, `>`, `>=` | Relational operators | No | `/status < 300`<br>`/status>=300` | |
| `+` | String concatenation operator | No | `/status_code + /message + "suffix"`

Check failure on line 230 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 230, "column": 92}}}, "severity": "ERROR"}
| `+`, `-` | Arithmetic addition and subtraction operators | No | `/status_code + length(/message) - 2`

Check failure on line 231 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 231, "column": 93}}}, "severity": "ERROR"}
| `*`, `/` | Multiplication and division operators | No | `/status_code * length(/message) / 3`

Check failure on line 232 in _data-prepper/pipelines/expression-syntax.md

View workflow job for this annotation

GitHub Actions / style-job

[vale] reported by reviewdog 🐶 [OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks. Raw Output: {"message": "[OpenSearch.Spelling] Error: _code. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/expression-syntax.md", "range": {"start": {"line": 232, "column": 93}}}, "severity": "ERROR"}
| `=~`, `!~` | Regex equality operators | No | `/msg =~ "^\w*$"`<br>`/msg=~"^\w*$"` | |
| `==`, `!=` | Equality operators | No | `/status == 200`<br>`/status_code==200` | |
| `and`, `or`, `not` | Conditional operators | Yes | `/a<300 and /b>200` | `/b<300and/b>200` |
Expand Down
Loading