From 1febfc9242b1ce085ec74d4af5fd91fd7428b786 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 17 Nov 2023 17:01:55 -0700 Subject: [PATCH 01/26] Add new dot expander processor doc Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 149 +++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 _ingest-pipelines/processors/dot-expander.md diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md new file mode 100644 index 0000000000..ebac20d362 --- /dev/null +++ b/_ingest-pipelines/processors/dot-expander.md @@ -0,0 +1,149 @@ +--- +layout: default +title: Dot expander +parent: Ingest processors +nav_order: 65 +--- + +# Dot expander + +The `dot_expander` processor transforms fields containing dots into object fields, making them accessible to other processors in the pipeline. Without this transformation, fields with dots cannot be processed. + +The following is the syntax for the `date_index_name` processor: + +```json +{ + "dot_expander": { + "field": "field.to.expand" + } +} +``` +{% include copy-curl.html %} + +## Configuration parameters + +The following table lists the required and optional parameters for the `dot_expander` processor. + +Parameter | Required/Optional | Description | +|-----------|-----------|-----------| +`field` | Required | The field to expand into an object field. If set to `*`, all top-level fields will be expanded. | +`path` | Optional | The field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields, which are fields that are not nested within any other objects. | +`override` | Optional | The field determines how the processor handles conflicts when expanding a field that overlaps with an existing nested object. Setting `override` to `false` instructs the processor to merge the conflicting values into an array, preserving both the original and expanded values. Conversely, setting `override` to `true` causes the processor to replace the existing nested object's value with the expanded field's value. | +`description` | Optional | A brief description of the processor. | +`if` | Optional | A condition for running this processor. | +`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | +`on_failure` | Optional | A list of processors to run if the processor fails. | +`tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | + +## Using the processor + +Follow these steps to use the processor in a pipeline. + +**Step 1: Create a pipeline.** + +The following query expands two fields named `user.address.city` and `user.address.state` into nested objects named `city` and `state`: + +```json +PUT /_ingest/pipeline/dot-expander +{ + "description": "Dot expander processor", + "processors": [ + { + "dot_expander": { + "field": "user.address.city" + } + }, + { + "dot_expander":{ + "field": "user.address.state" + } + } + ] +} +``` +{% include copy-curl.html %} + +**Step 2 (Optional): Test the pipeline.** + +It is recommended that you test your pipeline before you ingest documents. +{: .tip} + +To test the pipeline, run the following query: + +```json +POST _ingest/pipeline/dot-expander/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "field": "city, state" + } + } + ] +} +``` +{% include copy-curl.html %} + +#### Response + +The following example response confirms that the pipeline is working as expected: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "field": "city, state" + }, + "_ingest": { + "timestamp": "2023-11-17T23:49:27.597933805Z" + } + } + } + ] +} +``` + +**Step 3: Ingest a document.** + +The following query ingests a document into an index named `testindex1`: + +```json +PUT testindex1/_doc/1?pipeline=dot-expander +{ + "field": "Denver, CO" +} +``` +{% include copy-curl.html %} + +**Step 4 (Optional): Retrieve the document.** + +To retrieve the document, run the following query: + +```json +GET testindex1/_doc/1 +``` +{% include copy-curl.html %} + +#### Response + +The following response confirms the document was indexed: + +```json +{ + "_index": "testindex1", + "_id": "1", + "_version": 58, + "_seq_no": 57, + "_primary_term": 30, + "found": true, + "_source": { + "field": "Denver, CO" + } +} +``` From da2d4ad9184efad4e7d0ab12429c93213737d3ee Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 28 Nov 2023 15:52:54 -0700 Subject: [PATCH 02/26] Draft content for tech review Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 166 +++++++++++++++++++ 1 file changed, 166 insertions(+) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index ebac20d362..25a59768e4 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -147,3 +147,169 @@ The following response confirms the document was indexed: } } ``` + +## Nested fields + +The processor consolidates the `user.address.city` and `user.address.state` fields by merging with an existing `address`, `city`, and `state` field nested under `user`. If the field is a scalar value, then it will turn that into an array field. Take for example the following document: + +```json +{ + "user": { + "address": { + "city": "Denver", + "state": "CO" + } + } +} +``` + +The `dot_expander` processor transforms the individual fields into arrays as follows: + +```json +{ + "user": { + "address": { + "city": ["Denver"], + "state": ["CO"] + } + } +} +``` + +If you set the `override` parameter to `true`, the value of the expanded field overrides the value of the nested object. Take for example the following configuration: + +```json +{ + "dot_expander": { + "field": "user.address.city", + "override": true + } +} +``` + +The result is the following document, in which the expanded field `user.address.city` overrides the value of the nested object `user.address`: + +```json +{ + "user": { + "address": { + "city": "Denver", + "state": "CO" + } + } +} +``` + +If the field value is set to a wildcard `*`, the processor expands all top-level dotted field names, for example: + +```json +{ + "dot_expander": { + "field": "*" + } +} +``` + +Take for example the following: + +```json +{ + "user.address.city": "Denver", + "user.address.state": "CO" +} +``` + +The `dot_expander` processor transforms that document into the following: + +```json +{ + "user": { + "address": { + "city": "Denver" + } + }, + "user": { + "address": { + "state": "CO" + } + } +} +``` + +If the field is nested within a structure without dots, use the `path` parameter to navigate the non-dotted structure, for example: + +```json +{ + "dot_expander": { + "path": "user.address", + "field": "*" + } +} +``` + +Take for example the following document: + +```json +{ + "user": { + "address": { + "city.one": "Denver", + "city.two": "Houston", + "state.one": "CO", + "state.two": "TX" + } + } +} +``` + +The `dot_expander` processor transforms the document into: + +```json +{ + "user": { + "address": { + "city": { + "one": "Denver", + "two": "Houston" + }, + "state": { + "one": "CO", + "two": "TX" + } + } + } +} +``` + +To ensure proper expansion of the `user.address.city` and `user.address.state` fields and handle conflicts with pre-existing fields, use a similar configuration as the following document: + +```json +{ + "user": { + "address": { + "city": "Denver", + "state": "CO" + } + } +} +``` + +To ensure the correct expansion of the `city` and `state` fields, the following pipeline uses the `rename` processor to prevent conflicts and allow for proper handling of scalar fields during expansion. + +```json +{ + "processors": [ + { + "rename": { + "field": "user.address", + "target_field": "user.address.original" + } + }, + { + "dot_expander": { + "field": "user.address.original" + } + } + ] +} +``` From 3e977ffa9bcb0dd90132b8726614571968853fc4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 21 Dec 2023 15:32:23 -0700 Subject: [PATCH 03/26] Address tech review feedback Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 33 ++++++-------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 25a59768e4..fe08407d8e 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -154,46 +154,33 @@ The processor consolidates the `user.address.city` and `user.address.state` fiel ```json { - "user": { - "address": { - "city": "Denver", - "state": "CO" - } + "dot_expander": { + "field": "user.address.city", + "override": true } } ``` -The `dot_expander` processor transforms the individual fields into arrays as follows: +With `override:false` (or unset), the dot-pathed top-level field gets appended to the target object node, so it transforms into: ```json { - "user": { + "user": { "address": { - "city": ["Denver"], - "state": ["CO"] + "city": ["Denver", "Boulder"], + "state": "CO" } } } ``` -If you set the `override` parameter to `true`, the value of the expanded field overrides the value of the nested object. Take for example the following configuration: - -```json -{ - "dot_expander": { - "field": "user.address.city", - "override": true - } -} -``` - -The result is the following document, in which the expanded field `user.address.city` overrides the value of the nested object `user.address`: +With `override:true`, the dot-pathed top-level field value overwrites the target object node, so it transforms into: ```json { - "user": { + "user": { "address": { - "city": "Denver", + "city": "Boulder", "state": "CO" } } From b3f912a9a6e41468cbe08edb04f1b0abdaec4454 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 21 Dec 2023 16:06:15 -0700 Subject: [PATCH 04/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index fe08407d8e..fbfc5e1d0f 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -9,7 +9,7 @@ nav_order: 65 The `dot_expander` processor transforms fields containing dots into object fields, making them accessible to other processors in the pipeline. Without this transformation, fields with dots cannot be processed. -The following is the syntax for the `date_index_name` processor: +The following is the syntax for the `dot_expander` processor: ```json { From 5c859868fdb376142b62f8825af37ed81d9dac6d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 21 Dec 2023 16:06:27 -0700 Subject: [PATCH 05/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index fbfc5e1d0f..843b4084fa 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -206,7 +206,7 @@ Take for example the following: } ``` -The `dot_expander` processor transforms that document into the following: +The `dot_expander` processor transforms that document into the following fields, all nested under user: ```json { From 64b1001686424bae330e562c9f6ec2a106163dc3 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 21 Dec 2023 16:07:00 -0700 Subject: [PATCH 06/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 843b4084fa..469ffc304d 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -150,7 +150,7 @@ The following response confirms the document was indexed: ## Nested fields -The processor consolidates the `user.address.city` and `user.address.state` fields by merging with an existing `address`, `city`, and `state` field nested under `user`. If the field is a scalar value, then it will turn that into an array field. Take for example the following document: +The `dot_expander` processor consolidates the `user.address.city` and `user.address.state` fields by merging with an existing `address`, `city`, and `state` field nested under `user`. If the field is a scalar value, then it will turn that field into an array. Take for example the following document: ```json { From 96e0306a96f948dd39688d2d755712d7cad978f5 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 21 Dec 2023 16:07:11 -0700 Subject: [PATCH 07/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 469ffc304d..881a8c6621 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -197,7 +197,7 @@ If the field value is set to a wildcard `*`, the processor expands all top-level } ``` -Take for example the following: +Take for example the following document: ```json { From c5e32ebf63aafa44ee4ca3795f753c334e156e26 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 22 Dec 2023 08:10:33 -0700 Subject: [PATCH 08/26] Address doc review feedback Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 881a8c6621..b4456644b7 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -39,7 +39,7 @@ Parameter | Required/Optional | Description | Follow these steps to use the processor in a pipeline. -**Step 1: Create a pipeline.** +### Step 1: Create a pipeline The following query expands two fields named `user.address.city` and `user.address.state` into nested objects named `city` and `state`: @@ -63,7 +63,7 @@ PUT /_ingest/pipeline/dot-expander ``` {% include copy-curl.html %} -**Step 2 (Optional): Test the pipeline.** +### Step 2 (Optional): Test the pipeline It is recommended that you test your pipeline before you ingest documents. {: .tip} @@ -109,7 +109,7 @@ The following example response confirms that the pipeline is working as expected } ``` -**Step 3: Ingest a document.** +### Step 3: Ingest a document The following query ingests a document into an index named `testindex1`: @@ -121,7 +121,7 @@ PUT testindex1/_doc/1?pipeline=dot-expander ``` {% include copy-curl.html %} -**Step 4 (Optional): Retrieve the document.** +### Step 4 (Optional): Retrieve the document To retrieve the document, run the following query: @@ -197,6 +197,7 @@ If the field value is set to a wildcard `*`, the processor expands all top-level } ``` +Take for example the following document: Take for example the following document: ```json @@ -206,7 +207,7 @@ Take for example the following document: } ``` -The `dot_expander` processor transforms that document into the following fields, all nested under user: +The `dot_expander` processor transforms that document into the following fields, all nested under `user`: ```json { From d5042e1b3b1c64084c8504ec4377d92fd5c3e873 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 22 Dec 2023 08:51:11 -0700 Subject: [PATCH 09/26] Edit line 227 Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index b4456644b7..f6d1c95bf4 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -7,7 +7,7 @@ nav_order: 65 # Dot expander -The `dot_expander` processor transforms fields containing dots into object fields, making them accessible to other processors in the pipeline. Without this transformation, fields with dots cannot be processed. +The `dot_expander` processor is a tool that helps you work with hierarchical data. It transforms fields containing dots into object fields, making them accessible to other processors in the pipeline. Without this transformation, fields with dots cannot be processed. The following is the syntax for the `dot_expander` processor: @@ -198,7 +198,7 @@ If the field value is set to a wildcard `*`, the processor expands all top-level ``` Take for example the following document: -Take for example the following document: + ```json { @@ -224,7 +224,7 @@ The `dot_expander` processor transforms that document into the following fields, } ``` -If the field is nested within a structure without dots, use the `path` parameter to navigate the non-dotted structure, for example: +If a field is nested within a structure without dots, you can use the `path` parameter to traverse the non-dotted structure. For example, if you have the field `user.address`, in your data, you can use the `dot_expander processor` expand it into an object field named `user` with a nested field named `address`, as shown in the following example: ```json { From 38c639bce99a402f6bad761ce12184ce6a408ce9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 22 Dec 2023 08:55:48 -0700 Subject: [PATCH 10/26] Edit line 227 Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index f6d1c95bf4..e67797462a 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -224,7 +224,7 @@ The `dot_expander` processor transforms that document into the following fields, } ``` -If a field is nested within a structure without dots, you can use the `path` parameter to traverse the non-dotted structure. For example, if you have the field `user.address`, in your data, you can use the `dot_expander processor` expand it into an object field named `user` with a nested field named `address`, as shown in the following example: +If a field is nested within a structure without dots, you can use the `path` parameter to traverse the non-dotted structure. For example, if you have the field `user.address`, in your data, you can use the `dot_expander processor` expand it into an object field named `user` with a nested field named `address`, as shown in the following example: ```json { @@ -235,7 +235,7 @@ If a field is nested within a structure without dots, you can use the `path` par } ``` -Take for example the following document: +Then take for example the following document: ```json { From 15006d688925500baaaca15264624ba4422f73bb Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Fri, 22 Dec 2023 10:15:24 -0700 Subject: [PATCH 11/26] Edit line 227 Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index e67797462a..cafd41ca35 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -224,7 +224,7 @@ The `dot_expander` processor transforms that document into the following fields, } ``` -If a field is nested within a structure without dots, you can use the `path` parameter to traverse the non-dotted structure. For example, if you have the field `user.address`, in your data, you can use the `dot_expander processor` expand it into an object field named `user` with a nested field named `address`, as shown in the following example: +If a field is nested within a structure without dots, you can use the `path` parameter to traverse the non-dotted structure. For example, if you have the field `user.address`, you can use the `dot_expander processor` to expand it into an object field named `user` with a nested field named `address`, as shown in the following example: ```json { From 041423b4b2b438fc6cf724a60b9ccf71b7e4f8d0 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 4 Jan 2024 14:10:58 -0700 Subject: [PATCH 12/26] Address doc review comments Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 194 +++++-------------- 1 file changed, 51 insertions(+), 143 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index cafd41ca35..a0cda38c46 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -26,9 +26,8 @@ The following table lists the required and optional parameters for the `dot_expa Parameter | Required/Optional | Description | |-----------|-----------|-----------| -`field` | Required | The field to expand into an object field. If set to `*`, all top-level fields will be expanded. | +`field` | Required | The field to expand into an object field. | `path` | Optional | The field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields, which are fields that are not nested within any other objects. | -`override` | Optional | The field determines how the processor handles conflicts when expanding a field that overlaps with an existing nested object. Setting `override` to `false` instructs the processor to merge the conflicting values into an array, preserving both the original and expanded values. Conversely, setting `override` to `true` causes the processor to replace the existing nested object's value with the expanded field's value. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running this processor. | `ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | @@ -40,25 +39,25 @@ Parameter | Required/Optional | Description | Follow these steps to use the processor in a pipeline. ### Step 1: Create a pipeline - -The following query expands two fields named `user.address.city` and `user.address.state` into nested objects named `city` and `state`: + +The following query creates a dot expander processor that will expand two fields named `user.address.city` and `user.address.state` into nested objects: ```json PUT /_ingest/pipeline/dot-expander { - "description": "Dot expander processor", - "processors": [ - { - "dot_expander": { - "field": "user.address.city" - } - }, - { - "dot_expander":{ - "field": "user.address.state" + "description": "Dot expander processor", + "processors": [ + { + "dot_expander": { + "field": "user.address.city" + } + }, + { + "dot_expander": { + "field": "user.address.state" + } } - } - ] + ] } ``` {% include copy-curl.html %} @@ -71,17 +70,18 @@ It is recommended that you test your pipeline before you ingest documents. To test the pipeline, run the following query: ```json -POST _ingest/pipeline/dot-expander/_simulate -{ - "docs": [ - { - "_index": "testindex1", - "_id": "1", - "_source": { - "field": "city, state" - } - } - ] +POST _ingest/pipeline/dot-expander/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "user.address.city": "New York", + "user.address.state": "NY" + } + } + ] } ``` {% include copy-curl.html %} @@ -98,10 +98,15 @@ The following example response confirms that the pipeline is working as expected "_index": "testindex1", "_id": "1", "_source": { - "field": "city, state" + "user": { + "address": { + "city": "New York", + "state": "NY" + } + } }, "_ingest": { - "timestamp": "2023-11-17T23:49:27.597933805Z" + "timestamp": "2024-01-04T21:00:42.053781253Z" } } } @@ -114,9 +119,10 @@ The following example response confirms that the pipeline is working as expected The following query ingests a document into an index named `testindex1`: ```json -PUT testindex1/_doc/1?pipeline=dot-expander -{ - "field": "Denver, CO" +PUT testindex1/_doc/1?pipeline=dot-expander +{ + "user.address.city": "Denver", + "user.address.state": "CO" } ``` {% include copy-curl.html %} @@ -138,99 +144,30 @@ The following response confirms the document was indexed: { "_index": "testindex1", "_id": "1", - "_version": 58, - "_seq_no": 57, - "_primary_term": 30, + "_version": 63, + "_seq_no": 62, + "_primary_term": 33, "found": true, "_source": { - "field": "Denver, CO" + "user": { + "address": { + "city": "Denver", + "state": "CO" + } + } } } ``` ## Nested fields -The `dot_expander` processor consolidates the `user.address.city` and `user.address.state` fields by merging with an existing `address`, `city`, and `state` field nested under `user`. If the field is a scalar value, then it will turn that field into an array. Take for example the following document: - -```json -{ - "dot_expander": { - "field": "user.address.city", - "override": true - } -} -``` - -With `override:false` (or unset), the dot-pathed top-level field gets appended to the target object node, so it transforms into: - -```json -{ - "user": { - "address": { - "city": ["Denver", "Boulder"], - "state": "CO" - } - } -} -``` - -With `override:true`, the dot-pathed top-level field value overwrites the target object node, so it transforms into: - -```json -{ - "user": { - "address": { - "city": "Boulder", - "state": "CO" - } - } -} -``` - -If the field value is set to a wildcard `*`, the processor expands all top-level dotted field names, for example: - -```json -{ - "dot_expander": { - "field": "*" - } -} -``` - -Take for example the following document: - - -```json -{ - "user.address.city": "Denver", - "user.address.state": "CO" -} -``` - -The `dot_expander` processor transforms that document into the following fields, all nested under `user`: - -```json -{ - "user": { - "address": { - "city": "Denver" - } - }, - "user": { - "address": { - "state": "CO" - } - } -} -``` - If a field is nested within a structure without dots, you can use the `path` parameter to traverse the non-dotted structure. For example, if you have the field `user.address`, you can use the `dot_expander processor` to expand it into an object field named `user` with a nested field named `address`, as shown in the following example: ```json { "dot_expander": { "path": "user.address", - "field": "*" + "field": "" } } ``` @@ -238,48 +175,19 @@ If a field is nested within a structure without dots, you can use the `path` par Then take for example the following document: ```json -{ - "user": { - "address": { - "city.one": "Denver", - "city.two": "Houston", - "state.one": "CO", - "state.two": "TX" - } - } -} + ``` The `dot_expander` processor transforms the document into: ```json -{ - "user": { - "address": { - "city": { - "one": "Denver", - "two": "Houston" - }, - "state": { - "one": "CO", - "two": "TX" - } - } - } -} + ``` To ensure proper expansion of the `user.address.city` and `user.address.state` fields and handle conflicts with pre-existing fields, use a similar configuration as the following document: ```json -{ - "user": { - "address": { - "city": "Denver", - "state": "CO" - } - } -} + ``` To ensure the correct expansion of the `city` and `state` fields, the following pipeline uses the `rename` processor to prevent conflicts and allow for proper handling of scalar fields during expansion. From 340f72d06c21e98258e3894ebb6c92dfa133edb4 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 4 Jan 2024 14:12:45 -0700 Subject: [PATCH 13/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index a0cda38c46..ba1a040563 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -138,7 +138,7 @@ GET testindex1/_doc/1 #### Response -The following response confirms the document was indexed: +The following response confirms that the specified fields were expanded into nested fields: ```json { From c003b673744bfb2935145a3516650682ac4ad20f Mon Sep 17 00:00:00 2001 From: Fanit Kolchina Date: Tue, 16 Jan 2024 21:08:02 -0500 Subject: [PATCH 14/26] Added path parameter and field name conflicts sections Signed-off-by: Fanit Kolchina --- _ingest-pipelines/processors/dot-expander.md | 278 +++++++++++++++---- 1 file changed, 220 insertions(+), 58 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index ba1a040563..33937ff264 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -27,7 +27,7 @@ The following table lists the required and optional parameters for the `dot_expa Parameter | Required/Optional | Description | |-----------|-----------|-----------| `field` | Required | The field to expand into an object field. | -`path` | Optional | The field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields, which are fields that are not nested within any other objects. | +`path` | Optional | The field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running this processor. | `ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | @@ -40,24 +40,24 @@ Follow these steps to use the processor in a pipeline. ### Step 1: Create a pipeline -The following query creates a dot expander processor that will expand two fields named `user.address.city` and `user.address.state` into nested objects: +The following query creates a `dot_expander` processor that will expand two fields named `user.address.city` and `user.address.state` into nested objects: ```json -PUT /_ingest/pipeline/dot-expander +PUT /_ingest/pipeline/dot-expander-pipeline { - "description": "Dot expander processor", - "processors": [ - { - "dot_expander": { - "field": "user.address.city" - } - }, - { - "dot_expander": { - "field": "user.address.state" - } - } - ] + "description": "Dot expander processor", + "processors": [ + { + "dot_expander": { + "field": "user.address.city" + } + }, + { + "dot_expander":{ + "field": "user.address.state" + } + } + ] } ``` {% include copy-curl.html %} @@ -70,18 +70,18 @@ It is recommended that you test your pipeline before you ingest documents. To test the pipeline, run the following query: ```json -POST _ingest/pipeline/dot-expander/_simulate -{ - "docs": [ - { - "_index": "testindex1", - "_id": "1", - "_source": { - "user.address.city": "New York", - "user.address.state": "NY" - } - } - ] +POST _ingest/pipeline/dot-expander-pipeline/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "user.address.city": "New York", + "user.address.state": "NY" + } + } + ] } ``` {% include copy-curl.html %} @@ -106,7 +106,7 @@ The following example response confirms that the pipeline is working as expected } }, "_ingest": { - "timestamp": "2024-01-04T21:00:42.053781253Z" + "timestamp": "2024-01-17T01:32:56.501346717Z" } } } @@ -119,10 +119,10 @@ The following example response confirms that the pipeline is working as expected The following query ingests a document into an index named `testindex1`: ```json -PUT testindex1/_doc/1?pipeline=dot-expander -{ - "user.address.city": "Denver", - "user.address.state": "CO" +PUT testindex1/_doc/1?pipeline=dot-expander-pipeline +{ + "user.address.city": "Denver", + "user.address.state": "CO" } ``` {% include copy-curl.html %} @@ -144,9 +144,9 @@ The following response confirms that the specified fields were expanded into nes { "_index": "testindex1", "_id": "1", - "_version": 63, - "_seq_no": 62, - "_primary_term": 33, + "_version": 1, + "_seq_no": 3, + "_primary_term": 1, "found": true, "_source": { "user": { @@ -159,53 +159,215 @@ The following response confirms that the specified fields were expanded into nes } ``` -## Nested fields +## The `path` parameter -If a field is nested within a structure without dots, you can use the `path` parameter to traverse the non-dotted structure. For example, if you have the field `user.address`, you can use the `dot_expander processor` to expand it into an object field named `user` with a nested field named `address`, as shown in the following example: +You can use the `path` parameter to specify the path to a dotted field within an object. For example, the following pipeline specifies the `address.city` field that is located within `user` object: ```json +PUT /_ingest/pipeline/dot-expander-pipeline { - "dot_expander": { - "path": "user.address", - "field": "" + "description": "Dot expander processor", + "processors": [ + { + "dot_expander": { + "field": "address.city", + "path": "user" + } + }, + { + "dot_expander":{ + "field": "address.state", + "path": "user" + } } + ] } ``` +{% include copy-curl.html %} -Then take for example the following document: +Simulate the pipeline as follows: ```json - +POST _ingest/pipeline/dot-expander-pipeline/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "user": { + "address.city": "New York", + "address.state": "NY" + } + } + } + ] +} ``` +{% include copy-curl.html %} The `dot_expander` processor transforms the document into: ```json - +{ + "user": { + "address": { + "city": "New York", + "state": "NY" + } + } +} ``` -To ensure proper expansion of the `user.address.city` and `user.address.state` fields and handle conflicts with pre-existing fields, use a similar configuration as the following document: +## Field name conflicts + +If there already exists a field with the same path as the path where the `dot_expander` processor should expand the value, the processor merges the two values into an array. + +Consider the following pipeline that expands the field `user.name`: ```json - +PUT /_ingest/pipeline/dot-expander-pipeline +{ + "description": "Dot expander processor", + "processors": [ + { + "dot_expander": { + "field": "user.name" + } + } + ] +} ``` +{% include copy-curl.html %} -To ensure the correct expansion of the `city` and `state` fields, the following pipeline uses the `rename` processor to prevent conflicts and allow for proper handling of scalar fields during expansion. +Simulate the pipeline with a document where there are two values with the exact same path `user.name`: ```json +POST _ingest/pipeline/dot-expander-pipeline/_simulate { - "processors": [ - { - "rename": { - "field": "user.address", - "target_field": "user.address.original" + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "user.name": "John", + "user": { + "name": "Steve" } - }, - { - "dot_expander": { - "field": "user.address.original" + } + } + ] +} +``` +{% include copy-curl.html %} + +The response shows that the values were merged into an array: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "user": { + "name": [ + "Steve", + "John" + ] + } + }, + "_ingest": { + "timestamp": "2024-01-17T01:44:57.420220551Z" } - } - ] + } + } + ] +} +``` + +If there is a field name with a same name but a different path field needs to be renamed. For example, the following simulate call returns a parse exception: + +```json +POST _ingest/pipeline/dot-expander-pipeline/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "user": "John", + "user.name": "Steve" + } + } + ] +} +``` + +To avoid the parse exception, rename the field first using the `rename` processor: + +```json +PUT /_ingest/pipeline/dot-expander-pipeline +{ + "processors" : [ + { + "rename" : { + "field" : "user", + "target_field" : "user.name" + } + }, + { + "dot_expander": { + "field": "user.name" + } + } + ] +} +``` +{% include copy-curl.html %} + +Now simulate the pipeline: + +```json +POST _ingest/pipeline/dot-expander-pipeline/_simulate +{ + "docs": [ + { + "_index": "testindex1", + "_id": "1", + "_source": { + "user": "John", + "user.name": "Steve" + } + } + ] } ``` +{% include copy-curl.html %} + +The response confirms that the fields are merged: + +```json +{ + "docs": [ + { + "doc": { + "_index": "testindex1", + "_id": "1", + "_source": { + "user": { + "name": [ + "John", + "Steve" + ] + } + }, + "_ingest": { + "timestamp": "2024-01-17T01:52:12.864432419Z" + } + } + } + ] +} +``` \ No newline at end of file From ec4477c99faa22ed07924088f2a6a6aa13c9d511 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:47:06 -0700 Subject: [PATCH 15/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 33937ff264..3b9706313d 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -26,7 +26,7 @@ The following table lists the required and optional parameters for the `dot_expa Parameter | Required/Optional | Description | |-----------|-----------|-----------| -`field` | Required | The field to expand into an object field. | +`field` | Required | The field to be expanded into an object field. | `path` | Optional | The field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running this processor. | From dd9118aa798fad3febca45d2e80cd38a1b7ea82c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:47:21 -0700 Subject: [PATCH 16/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 3b9706313d..4043d5d1a2 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -27,7 +27,7 @@ The following table lists the required and optional parameters for the `dot_expa Parameter | Required/Optional | Description | |-----------|-----------|-----------| `field` | Required | The field to be expanded into an object field. | -`path` | Optional | The field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields. | +`path` | Optional | This field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields. | `description` | Optional | A brief description of the processor. | `if` | Optional | A condition for running this processor. | `ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | From 586f7047330026bf5b591497032d624b75aab65a Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:47:29 -0700 Subject: [PATCH 17/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 4043d5d1a2..3cbe910d6f 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -32,7 +32,7 @@ Parameter | Required/Optional | Description | `if` | Optional | A condition for running this processor. | `ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | `on_failure` | Optional | A list of processors to run if the processor fails. | -`tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | +`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. | ## Using the processor From 29f3fb9b4ba8ed6843df20a8370dbb7deed4b1a9 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:47:42 -0700 Subject: [PATCH 18/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 3cbe910d6f..8e5424feaa 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -29,7 +29,7 @@ Parameter | Required/Optional | Description | `field` | Required | The field to be expanded into an object field. | `path` | Optional | This field is only required if the field to be expanded is nested within another object field. This is because the `field` parameter only recognizes leaf fields. | `description` | Optional | A brief description of the processor. | -`if` | Optional | A condition for running this processor. | +`if` | Optional | A condition for running the processor. | `ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | `on_failure` | Optional | A list of processors to run if the processor fails. | `tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. | From 026af45196b03a75a12183358a7804b71a09577f Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:47:59 -0700 Subject: [PATCH 19/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 8e5424feaa..5bddb9205e 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -161,7 +161,7 @@ The following response confirms that the specified fields were expanded into nes ## The `path` parameter -You can use the `path` parameter to specify the path to a dotted field within an object. For example, the following pipeline specifies the `address.city` field that is located within `user` object: +You can use the `path` parameter to specify the path to a dotted field within an object. For example, the following pipeline specifies the `address.city` field that is located within the `user` object: ```json PUT /_ingest/pipeline/dot-expander-pipeline From 41dda97756cd5ab19065b9dac023b903bd8f2073 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:48:40 -0700 Subject: [PATCH 20/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 5bddb9205e..5e2c8f8cdb 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -206,7 +206,7 @@ POST _ingest/pipeline/dot-expander-pipeline/_simulate ``` {% include copy-curl.html %} -The `dot_expander` processor transforms the document into: +The `dot_expander` processor transforms the document into the following: ```json { From feae23390e138b29d68bc4459f9d85fc8f9f1dcb Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:48:56 -0700 Subject: [PATCH 21/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 5e2c8f8cdb..65d65e1439 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -221,7 +221,7 @@ The `dot_expander` processor transforms the document into the following: ## Field name conflicts -If there already exists a field with the same path as the path where the `dot_expander` processor should expand the value, the processor merges the two values into an array. +If a field already exists that has the same path as the path where the `dot_expander` processor should expand the value, the processor merges the two values into an array. Consider the following pipeline that expands the field `user.name`: From 53d0d0ac7a36a713393d8d1f2b5613d82f1f7960 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:49:10 -0700 Subject: [PATCH 22/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 65d65e1439..d2ae8afc0f 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -240,7 +240,7 @@ PUT /_ingest/pipeline/dot-expander-pipeline ``` {% include copy-curl.html %} -Simulate the pipeline with a document where there are two values with the exact same path `user.name`: +Simulate the pipeline with a document containing two values with the exact same path `user.name`: ```json POST _ingest/pipeline/dot-expander-pipeline/_simulate From 3dde50bebac9f8f0e7ff8be79e9579af309c4ab0 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:49:31 -0700 Subject: [PATCH 23/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index d2ae8afc0f..e0eee9ec0c 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -305,7 +305,7 @@ POST _ingest/pipeline/dot-expander-pipeline/_simulate } ``` -To avoid the parse exception, rename the field first using the `rename` processor: +To avoid the parse exception, first rename the field by using the `rename` processor: ```json PUT /_ingest/pipeline/dot-expander-pipeline From 6a01fbab7b5e4174c9745ffe297d0d516c5f680c Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 11:49:46 -0700 Subject: [PATCH 24/26] Update _ingest-pipelines/processors/dot-expander.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index e0eee9ec0c..b682a2cb35 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -346,7 +346,7 @@ POST _ingest/pipeline/dot-expander-pipeline/_simulate ``` {% include copy-curl.html %} -The response confirms that the fields are merged: +The response confirms that the fields were merged: ```json { From 0d5c299f7681ecd010d8dedcc9204819388e4da8 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Thu, 18 Jan 2024 12:13:51 -0700 Subject: [PATCH 25/26] Address editorial review feedback Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index b682a2cb35..6fba80fbe1 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -185,7 +185,7 @@ PUT /_ingest/pipeline/dot-expander-pipeline ``` {% include copy-curl.html %} -Simulate the pipeline as follows: +You can simulate the pipeline as follows: ```json POST _ingest/pipeline/dot-expander-pipeline/_simulate @@ -206,7 +206,7 @@ POST _ingest/pipeline/dot-expander-pipeline/_simulate ``` {% include copy-curl.html %} -The `dot_expander` processor transforms the document into the following: +The `dot_expander` processor transforms the document into the following structure: ```json { @@ -221,7 +221,7 @@ The `dot_expander` processor transforms the document into the following: ## Field name conflicts -If a field already exists that has the same path as the path where the `dot_expander` processor should expand the value, the processor merges the two values into an array. +If a field already exists with the same path as the path to which the `dot_expander` processor should expand the value, the processor merges the two values into an array. Consider the following pipeline that expands the field `user.name`: @@ -240,7 +240,7 @@ PUT /_ingest/pipeline/dot-expander-pipeline ``` {% include copy-curl.html %} -Simulate the pipeline with a document containing two values with the exact same path `user.name`: +You can simulate the pipeline with a document containing two values with the exact same path `user.name`: ```json POST _ingest/pipeline/dot-expander-pipeline/_simulate @@ -261,7 +261,7 @@ POST _ingest/pipeline/dot-expander-pipeline/_simulate ``` {% include copy-curl.html %} -The response shows that the values were merged into an array: +The response confirms that the values were merged into an array: ```json { @@ -287,7 +287,7 @@ The response shows that the values were merged into an array: } ``` -If there is a field name with a same name but a different path field needs to be renamed. For example, the following simulate call returns a parse exception: +If a field contains the same name but a different path, then the field needs to be renamed. For example, the following simulate call returns a parse exception: ```json POST _ingest/pipeline/dot-expander-pipeline/_simulate @@ -327,7 +327,7 @@ PUT /_ingest/pipeline/dot-expander-pipeline ``` {% include copy-curl.html %} -Now simulate the pipeline: +Now you can simulate the pipeline: ```json POST _ingest/pipeline/dot-expander-pipeline/_simulate From 6729218612e067932947297c52e8a7a7c9ccfe7d Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 30 Jan 2024 10:54:50 -0700 Subject: [PATCH 26/26] Update dot-expander.md Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/dot-expander.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_ingest-pipelines/processors/dot-expander.md b/_ingest-pipelines/processors/dot-expander.md index 6fba80fbe1..5cfebba758 100644 --- a/_ingest-pipelines/processors/dot-expander.md +++ b/_ingest-pipelines/processors/dot-expander.md @@ -287,7 +287,7 @@ The response confirms that the values were merged into an array: } ``` -If a field contains the same name but a different path, then the field needs to be renamed. For example, the following simulate call returns a parse exception: +If a field contains the same name but a different path, then the field needs to be renamed. For example, the following `_simulate` call returns a parse exception: ```json POST _ingest/pipeline/dot-expander-pipeline/_simulate @@ -370,4 +370,4 @@ The response confirms that the fields were merged: } ] } -``` \ No newline at end of file +```