Skip to content

Commit

Permalink
Add documentation that supports custom log types (#4969)
Browse files Browse the repository at this point in the history
* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates

Signed-off-by: cwillum <[email protected]>

---------

Signed-off-by: cwillum <[email protected]>
  • Loading branch information
cwillum authored and vagimeli committed Sep 19, 2023
1 parent 43ad56e commit 4ad6fef
Show file tree
Hide file tree
Showing 17 changed files with 305 additions and 13 deletions.
1 change: 1 addition & 0 deletions _security-analytics/api-tools/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,5 @@ The APIs for Security Analytics are separated into the following categories:
* [Mappings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/)
* [Alerts and findings APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/alert-finding-api/)
* [Correlation engine APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/correlation-eng/)
* [Log type APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/log-type-api/)

237 changes: 237 additions & 0 deletions _security-analytics/api-tools/log-type-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,237 @@
---
layout: default
title: Log type APIs
parent: API tools
nav_order: 56
---

# Log type APIs

The log type APIs allow you to create a custom log type, search custom log types, update custom log types, and delete custom log types.


## Create log type

Creating a new custom log type involves entering a name and a description and specifying the source as `Custom`.


### Example request

```json
POST /_plugins/_security_analytics/logtype
{
"description": "custom-log-type-desc",
"name": "custom-log-type4",
"source": "Custom"
}
```
{% include copy-curl.html %}


### Example response

```json
{
"_id": "m98uk4kBlb9cbROIpEj2",
"_version": 1,
"logType": {
"name": "custom-log-type4",
"description": "custom-log-type-desc",
"source": "Custom",
"tags": {
"correlation_id": 27
}
}
}
```


## Search custom log types

This API allows you to search log types in the system.


### Example request

```json
POST /_plugins/_security_analytics/logtype/_search
{
"query": {
"match_all": {}
}
}
```
{% include copy-curl.html %}


### Example response

```json
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 26,
"relation": "eq"
},
"max_score": 2.0,
"hits": [
{
"_index": ".opensearch-sap-log-types-config",
"_id": "s3",
"_score": 2.0,
"_source": {
"name": "s3",
"description": "Windows logs",
"source": "Sigma",
"tags": {
"correlation_id": 21
}
}
},
{
"_index": ".opensearch-sap-log-types-config",
"_id": "others_compliance",
"_score": 2.0,
"_source": {
"name": "others_compliance",
"description": "Compliance logs",
"source": "Sigma",
"tags": {
"correlation_id": 4
}
}
},
{
"_index": ".opensearch-sap-log-types-config",
"_id": "github",
"_score": 2.0,
"_source": {
"name": "github",
"description": "Sys logs",
"source": "Sigma",
"tags": {
"correlation_id": 16
}
}
},
{
"_index": ".opensearch-sap-log-types-config",
"_id": "others_application",
"_score": 2.0,
"_source": {
"name": "others_application",
"description": "Application logs",
"source": "Sigma",
"tags": {
"correlation_id": 0
}
}
},
{
"_index": ".opensearch-sap-log-types-config",
"_id": "dns",
"_score": 2.0,
"_source": {
"name": "dns",
"description": "Compliance logs",
"source": "Sigma",
"tags": {
"correlation_id": 15
}
}
},
{
"_index": ".opensearch-sap-log-types-config",
"_id": "m98uk4kBlb9cbROIpEj2",
"_score": 2.0,
"_source": {
"name": "custom-log-type-updated4",
"description": "custom-log-type-updated-desc",
"source": "Custom",
"tags": null
}
}
]
}
}
```


## Update custom log type

This API allows you to update existing custom log types. Use the log type's ID in the route to specify the log type, as shown in the following example:

```json
PUT /_plugins/_security_analytics/logtype/<log_type_id>
```


### Example request

```json
PUT /_plugins/_security_analytics/logtype/m98uk4kBlb9cbROIpEj2
{
"name": "custom-log-type4",
"description": "custom-log-type-updated-desc",
"source": "Custom"
}
```
{% include copy-curl.html %}


### Example response

```json
{
"_id": "m98uk4kBlb9cbROIpEj2",
"_version": 1,
"logType": {
"name": "custom-log-type4",
"description": "custom-log-type-updated-desc",
"source": "Custom",
"tags": {
"correlation_id": 27
}
}
}
```


## Delete custom log type

This API is used to delete a custom log type. Specify the log type's ID in the route to run the operation:

```json
DELETE /_plugins/_security_analytics/logtype/<log_type_id>
```


### Example request

```json
DELETE /_plugins/_security_analytics/logtype/m98uk4kBlb9cbROIpEj2
```
{% include copy-curl.html %}


### Example response

```json
200 OK
{
"_id": "m98uk4kBlb9cbROIpEj2",
"_version": 1
}
```

Only custom log types can be deleted. Trying to delete a standard OpenSearch-defined log type results in an error.
{: .note }

2 changes: 1 addition & 1 deletion _security-analytics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ For information about configuring detectors, see [Creating detectors]({{site.url

### Log types

Log types provide the data used to evaluate events occurring in a system. OpenSearch supports several types of logs and provides out-of-the-box mappings for the most common log sources. Currently supported log sources include:
Log types provide the data used to evaluate events occurring in a system. OpenSearch supports several types of logs and provides out-of-the-box mappings for the most common log sources. Currently supported log types include the following:
* Network events
* DNS logs
* Apache access logs
Expand Down
43 changes: 43 additions & 0 deletions _security-analytics/sec-analytics-config/custom-log-type.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
---
layout: default
title: Creating custom log types
parent: Setting up Security Analytics
nav_order: 18
---


# Creating custom log types

Log types represent the different sources of data used for threat detection in Security Analytics. In addition to the standard log types supported by Security Analytics, you can create custom log types for your threat detectors. Follow the steps provided here to create a custom log type.


## The Log types page

To navigate to the **Log types** page, select **Log types** under **Detectors** in the navigation menu. The following image shows the **Log types** landing page.

<img src="{{site.url}}{{site.baseurl}}/images/Security/c-log-type.png" alt="The Log types landing page." width="85%">

The table that lists the log types provides the name of the log type, its description, and identifies whether it's a standard OpenSearch-defined log type or a custom log type. The following list describes the main features found on the **Log types** page and the actions you can take:

* Select the log type **Name** to open the log type's details page. The **Details** tab is shown by default. This tab includes the log type's ID. You can also select the **Detection rules** tab to show all detection rules associated with the log type.
* In the **Actions** column, you can select the trash can icon ({::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/alerting/trash-can-icon.png" class="inline-icon" alt="trash can icon"/>{:/}) to delete a custom log type (you cannot delete a standard OpenSearch-defined log type). Follow the prompts to confirm and safely remove the custom log type.
* Select **Create log type** in the upper-right corner of the screen to begin creating a custom log type. The **Create log type** page opens. Continue with the steps in the section that follows to create a custom log type.


## Creating a custom log type

After selecting **Create log type** on the **Log types** page, the **Create log type** page opens and provides the necessary fields for creating a new log type:

1. Enter a name for the log type.

The log type name supports characters a--z (lowercase), 0--9, hyphens, and underscores.
{: .note }

1. Enter a description for the log type.
1. Select **Create log type** in the lower-right corner of the screen. The screen returns to the **Log types** page, and the new log type appears in the list of all log types. Note that the source for the new log type indicates **Custom**.


## Log type API

To perform operations for custom log types using the REST API, see [Log type APIs]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/log-type-api/).

16 changes: 9 additions & 7 deletions _security-analytics/sec-analytics-config/detectors-config.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,16 @@ You can define a new detector by naming the detector, selecting a data source an
When multiple data sources are selected, the logs must be of the same type. We recommend creating separate detectors for different log types.
{: .note }

1. In the **Log types and rules** section, select the log type for the data source. The system automatically populates the Sigma security rules associated with the log type.
1. In the **Log types and rules** section, select the log type for the data source from the dropdown list. The system automatically populates the detection rules associated with the log type.

For information about creating your own detection rules, see [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#creating-detection-rules).
For information about creating your own detection rules, see [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#creating-detection-rules). To create your own log types, see [Creating custom log types]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/custom-log-type/).
{: .note }

The following image shows the number of associated rules populated in the **Detection rules** section.
The example in the following image shows the number of rules associated with **network** after this log type is selected.

<img src="{{site.url}}{{site.baseurl}}/images/Security/detector_rules.png" alt="Selecting threat detector type to auto-populate rules" width="85%">
<img src="{{site.url}}{{site.baseurl}}/images/Security/detector-rules.png" alt="Selecting threat detector log type to auto-populate rules" width="85%">

When you select **Network events**, **CloudTrail logs**, or **S3 access logs** as the log type, the system automatically creates a detector dashboard. The dashboard offers visualizations for the detector and can provide security-related insight into log source data. For more information about visualizations, see [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/).
When you select **network**, **cloudtrail**, or **s3** as the log type, the system automatically creates a detector dashboard. The dashboard offers visualizations for the detector and can provide security-related insight into log source data. For more information about visualizations, see [Building data visualizations]({{site.url}}{{site.baseurl}}/dashboards/visualize/viz-index/).

You can skip the next step for applying select rules if you are satisfied with those automatically populated by the system. Otherwise, go to the next step to select rules individually.
{: .note }
Expand Down Expand Up @@ -69,7 +69,7 @@ Although the ECS rule field names are largely self-explanatory, you can find pre

[Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail logs can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.

To reveal fields for a log index in either raw format or OCSF, use the [Get Mappings View]({{site.url}}{{site.baseurl}}/security-analytics/api-tools/mappings-api/#get-mappings-view) API and specify the index in the `index_name` field of the request.
{: .tip }
Expand Down Expand Up @@ -147,5 +147,7 @@ To set up an alert for a detector, continue with the following steps:
---
## What's next

If you are ready to view findings for the new detector, see the [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/) section. If you would like to import rules or set up custom rules before working with findings, see the [Working with rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/) section.
If you are ready to view findings generated by the new detector, see the [Working with findings]({{site.url}}{{site.baseurl}}/security-analytics/usage/findings/) section. If you would like to import rules or set up custom rules before working with findings, see the [Working with detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/) section.

To configure Security Analytics to identify correlations between events happening in different logs throughout your system, see [Creating correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/correlation-config/).

5 changes: 3 additions & 2 deletions _security-analytics/sec-analytics-config/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ redirect_from:

Before Security Analytics can begin generating findings and sending alerts, administrators must create detectors and make log data available to the system. Once detectors are able to generate findings, you can fine-tune your alerts to focus on specific areas of interest. The following steps outline the basic workflow for setting up components in Security Analytics.

1. Create security detectors and alerts, and ingest log data. See [Creating detectors]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/) for details.
1. Create threat detectors and alerts, and ingest log data. See [Creating detectors]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/detectors-config/) for more information.
1. Consider [creating correlation rules]({{site.url}}{{site.baseurl}}/security-analytics/sec-analytics-config/correlation-config/) to identify connections between events and possible threats occurring in different logs throughout your system.
1. Inspect findings generated from detector output and create any additional alerts.
1. If desired, create custom rules by duplicating and then modifying pre-packaged rules. See [Customizing rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#customizing-rules) for details.
1. If desired, create custom rules to better focus detectors on high-priority concerns in your system. See [Creating detection rules]({{site.url}}{{site.baseurl}}/security-analytics/usage/rules/#creating-detection-rules) for more information.

## Navigate to Security Analytics

Expand Down
14 changes: 11 additions & 3 deletions _security-analytics/usage/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ nav_order: 25

When you select **Security Analytics** from the top menu, the Overview page is displayed. The Overview page consists of five sections:
* Findings and alert count
* Top recent alerts
* Top recent findings
* Recent alerts
* Recent findings
* Most frequent detection rules
* Detectors

Expand All @@ -35,20 +35,26 @@ You can also select the **Getting started** link to expand the Get started with

The Findings and alert count section provides a graph showing data on the latest findings. Use the **Group by** dropdown list to select either **All findings** or **Log type**.

<img src="{{site.url}}{{site.baseurl}}/images/Security/count.png" alt="A graph showing counts for findings and alerts." width="75%">

---
## Recent alerts

The Recent alerts table displays recent alerts by time, trigger name, and alert severity. Select **View alerts** to go to the Alerts page.

<img src="{{site.url}}{{site.baseurl}}/images/Security/recent-alerts.png" alt="A table showing the most recent alerts." width="50%">

---
## Recent findings

The Recent findings table displays recent findings by time, rule name, rule severity, and detector. Select **View all findings** to go to the Findings page.

<img src="{{site.url}}{{site.baseurl}}/images/Security/recent-findings.png" alt="A table showing the most recent findings." width="50%">

---
## Most frequent detection rules

This section provides a graphical representation of detection rules that trigger findings most often and how they compare to others as a percentage of the whole. The rule names represented by the graph are listed to the right.
This section provides a graphical representation of detection rules that trigger findings most often and how they compare to others as a percentage of the whole. The rule names represented by the graph are listed to the right. You can hover over each color on the graph to see details about the detection rule it represents.

<img src="{{site.url}}{{site.baseurl}}/images/Security/rule_graph.png" alt="The detection rule graph on the Overview page" width="50%">

Expand All @@ -57,3 +63,5 @@ This section provides a graphical representation of detection rules that trigger

The Detectors section displays a list of available detectors by detector name, status (active/inactive), and log type. Select **View all detectors** to go to the Detectors page. Select **Create detector** to go directly to the Define detector page.

<img src="{{site.url}}{{site.baseurl}}/images/Security/detector-overview.png" alt="A table showing available detectors." width="50%">

Binary file added images/Security/c-log-type.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Security/count.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Security/detector-overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Security/detector-rules.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed images/Security/detector_rules.png
Binary file not shown.
Binary file modified images/Security/overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Security/recent-alerts.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/Security/recent-findings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified images/Security/rule_graph.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified images/Security/secanalytics-det-nav.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 4ad6fef

Please sign in to comment.