Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new documentation IP2Geo processor automatic updating feature #4095

Closed
wants to merge 80 commits into from
Closed
Changes from 8 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
8a49f53
Content planning
vagimeli May 17, 2023
398eae6
Content planning
vagimeli May 17, 2023
74dd61d
Writing
vagimeli May 22, 2023
dbcec4e
Writing
vagimeli May 23, 2023
daee116
Writing
vagimeli May 23, 2023
2c1bf28
Writing
vagimeli May 23, 2023
989aa7c
Writing
vagimeli May 23, 2023
0822c04
Writing
vagimeli May 23, 2023
a659296
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
0742e27
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
185179c
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
fd4b645
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
a460a6e
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
a5cf0e0
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
a5d96dc
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
9b292ea
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
9655bb8
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
a448e40
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
d225a16
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
720c244
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
aba49db
Update _api-reference/ingest-apis/geoip.md
vagimeli May 24, 2023
1affbe6
Writing
vagimeli May 24, 2023
91cd80a
Writing
vagimeli May 24, 2023
85b845b
Writing
vagimeli May 24, 2023
23d8cd4
Writing
vagimeli May 24, 2023
edbb76f
Writing
vagimeli May 24, 2023
ed5ac30
Writing
vagimeli May 25, 2023
037dff5
Writing
vagimeli May 25, 2023
0b9b5e2
Address tech review feedback
vagimeli May 26, 2023
823f77a
Update processors.md
vagimeli May 26, 2023
7c0474f
Update processors.md
vagimeli May 26, 2023
c32a6b6
Writing
vagimeli May 30, 2023
baa6a38
Add processor index page
vagimeli Jun 2, 2023
28f5b6b
Update front matter
vagimeli Jun 22, 2023
7a5bdc8
Created new file under Ingest Processors TOC
vagimeli Jun 22, 2023
bed4f18
Update ip2geo.md
vagimeli Jun 28, 2023
75572e9
Update ip2geo.md
vagimeli Jun 28, 2023
3c2c1df
Update ip2geo.md
vagimeli Jun 28, 2023
985e111
Update ip2geo.md
vagimeli Jul 5, 2023
b718810
Update ip2geo.md
vagimeli Jul 5, 2023
8a48e7b
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
7e1b3af
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
85bbaa6
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
c5e30f4
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
9c42c20
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
cc86348
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
72c2091
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
8abfc1c
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
99b0600
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
280f71b
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
0ca4e2d
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
b6f406b
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
3fbeb91
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
71ae9d7
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 3, 2023
99b117b
Update ip2geo.md
vagimeli Aug 3, 2023
a6dc976
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 18, 2023
e381d1e
Address doc review feedback
vagimeli Aug 18, 2023
c64e3cc
Update ip2geo.md
vagimeli Aug 18, 2023
008a52b
Update ip2geo.md
vagimeli Aug 18, 2023
59ef808
Add copy labels
vagimeli Aug 22, 2023
e83bbd8
Update ip2geo.md
vagimeli Aug 22, 2023
e43415a
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
512cb4f
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
0af7ee4
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
073980c
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
167468f
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
8fee903
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
9730088
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
e2c2e01
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
933a0c4
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
1a357f3
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
ff19e23
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
17cc8a6
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
c84be08
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
3e3743a
Update _api-reference/ingest-apis/ip2geo.md
vagimeli Aug 23, 2023
936b757
Copy edits
vagimeli Aug 23, 2023
48d88cf
Address editorial feedback
vagimeli Aug 23, 2023
2322ee0
Address editorial feedback
vagimeli Aug 23, 2023
2e4eac0
Update ip2geo.md
vagimeli Aug 23, 2023
42ab53a
Copy edit to align format to processors template
vagimeli Sep 5, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 172 additions & 0 deletions _api-reference/ingest-apis/geoip.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
---
layout: default
title: IP geolocation processor
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
parent: Ingest APIs
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
nav_order: 30
---

# IP geolocation processor
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Introduced 2.8
{: .label .label-purple }

Information about the geolocation of an IP address can be used for a variety of purposes:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

- **Content personalization:** You can use IP geolocation information to personalize content for your users based on their location. For example, you could show different versions of your website to users from different countries.
- **Security:** You can use GeoIP to block access to your website from certain countries. This can be helpful to protect your website from attacks or to comply with regulations.
- **Analytics:** You can use GeoIP to track the geographic location of your website visitors. This information can be used to learn more about your audience and to improve your marketing campaigns.

The OpenSearch `Ip2geo` processor adds geographical information about IP addresses based on data from the [MaxMind GeoIP2 databases](https://www.maxmind.com/en/geoip2-databases). This processor adds the geolocation information by default under the `<field_name>` and auto-updates the GeoIP2 databases based on a set interval, keeping geolocation data up-to-date and accurate.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Installing the Ip2geo processor
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

To install the `Ip2geo` processor, the `opensearch-geospatial` plugin must be installed first. Learn more in the [Installing plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) documentation.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Creating the data source
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

Once you've installed the `Ip2geo` processor, create the IP geolocation data source by defining the endpoint value to download geolocation data and specify the data update interval. The endpoint value must contain valid data formats, for example, <insert-example>. The minimum update interval is 1 day. The maximum is determined by the database provider.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

The following code example shows how to create a data source using the OpenSearch default endpoint value, which is used if the endpoint value is empty, and update interval of 3 days.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

#### Example: JSON POST request

````json
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
{
"endpoint" : "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
"update_interval_in_days" : 3
}
```

The following code example shows the JSON reponse to the preceding request. A true JSON response means the request was successful and the server was able to process the request. If you receive a false JSON reponse, check the request to make sure it is valid, check the URL to make sure it is correct, or try again.

#### Example: JSON response

```json
{
"acknowledged":true
}
```

## Sending a GET request

To request data from the specifed database you created, send a GET request.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

#### Example: GET request

```
GET https://<host>:<port>/_plugins/geospatial/ip2geo/datasource/_all
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
```

#### Example: GET response

```
<insert-response-example>
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
```

The reponse shows information for each field (for example, `name`, `endpoint`, `provider`) in the data source file, when the data source file last updated successfully or failed (for example, `last_succeeded_at_in_epoch_millis`), and fields (for example, `fields`) added to the file since you last updated it.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Updating the data source
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

To update the data source file, send a PUT request. You can continue using the current endpoint value or change it. Note that if the new endpoint value contains fields that are not in the current data source file, the update fails. You also can change the update interval.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

#### Example: PUT request

```json
{
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
"endpoint": https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json
"update_interval_in_days": 1
}
```

#### Example: Response

```json
{
"acknowledged":true
}
```

## Creating the IP2geo processor

Once the data source is created, you can create the `Ip2geo` processor. To create the processor, send a PUT request.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

#### Example: PUT request

```json
{
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
"description":"convert ip to geo",
"processors":[
{
"ip2geo":{
"field":"_ip",
"datsource"::"test1"
}
}
]
}


#### Example: Response

```json
<insert-example-response>
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
```

## Using the IP2geo processor in a pipeline

The following table describes <what>.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

| Name | Required | Default | Description |
|------|----------|---------|-------------|
| field | yes | - | The field to get the ip address from for the geographical lookup. |
| target_field | no | ip2geo | The field that will hold the geographical information looked up from the Maxmind database. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| database | no | geoip.maps.opensearch.org/v1/geolite2-city/manifest.json | The database filename referring to a database the module ships with or a custom database in the ingest-geoip config directory. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| properties | no | Default field depends on what is available in `database`. | Controls what properties are added to the target_field based on the geoip lookup. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
| ignore_missing | no | false | If `true` and `field` does not exist, the processor quietly exits without modifying the document. |
| first_only | no | true | If `true` only first found geoip data will be returned, even if field contains array. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved


The following code is an example of using <example_name> and adds the geographical information to the `geoip` field based on the `ip` field.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

### Simulating the pipeline

To simulate the pipeline, specify the pipeline in the POST request.

#### Example: POST request

```json
<insert-example-request>
```

## Deleting the Ip2geo processor
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

To delete the `IP2geo` processor, send a DELETE request.

#### Example: DELETE request

```json
{
DELETE https://<host>:<port>/_ingest/pipeline/<processor>
}
```
## Deleting the data source
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

To delete the data source, send a DELETE request. Note that if you have another processors that uses the data source, the delete fails. To delete the dat source, you must delete all processors associated with the data source.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

#### Example: DELETE request

```json
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
{
DELETE https://<host>:<port>/_plugins/geospatial/ip2geo/datasource/_all
}
```

#### Example: Failed DELETE request

```json
<insert-example-failed-request>
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
```

## Next steps

<Do we want to link to any Data Prepper processor information?>
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
<What other documentation or GitHub resources should we include?>