Add documentation that supports custom log types #4969

cwillum · 2023-09-06T00:27:56Z

Description

Adds documentation for custom log type enhancements.

Issues Resolved

Fixes #4741

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: cwillum <[email protected]>

cwillum · 2023-09-11T23:02:58Z

@sbcd90 @amsiglan Could you look at this documentation for custom log types? I also have a few questions about the "Search custom log types" API:

For the Search custom log types API, what kind of information goes in the "match_all" object? How are the values formatted: separated by commas? I’ve tried names and ids - neither work.
What are the other request fields that can be used with this call? (In addition to "match_all")
This appears to provide results for all log types, not only custom. Any problems just calling it "Search log types" API? Or are we intending users to only process custom log types with it?

Thank you.

Signed-off-by: cwillum <[email protected]>

cwillum · 2023-09-18T21:35:18Z

Will create a separate PR to address "Search custom log type API" questions. Pushing this to Documentation team review. (this was tech reviewed via direct communications)

_security-analytics/sec-analytics-config/custom-log-type.md

Naarcha-AWS

A couple of small questions, otherwise LGTM.

_security-analytics/sec-analytics-config/custom-log-type.md

Naarcha-AWS · 2023-09-19T16:24:25Z

_security-analytics/sec-analytics-config/custom-log-type.md

+After selecting **Create log type** in the **Log types** page, the **Create log type** page opens and provides the necessary fields to create a new log type:
+1. Enter a name for the log type.
+
+   The log type name supports characters a-z (lower case), 0--9, hyphens, and underscores.


Does this note format correctly?

Naarcha-AWS · 2023-09-19T16:26:18Z

_security-analytics/sec-analytics-config/detectors-config.md

@@ -69,7 +69,7 @@ Although the ECS rule field names are largely self-explanatory, you can find pre

 [Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

-The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
+The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.


Why is DNS lowercased and bolded in this context? Is it supposed to match a UI element?

Yes. But this representation is opposed by the PM. UX and front-end dev will eventually replace these log type names with revised names. For the time being, this is the way the log type is represented in the UI.

cwillum

Thanks. Addressed comments.

cwillum · 2023-09-19T18:01:31Z

_security-analytics/sec-analytics-config/detectors-config.md

@@ -69,7 +69,7 @@ Although the ECS rule field names are largely self-explanatory, you can find pre

 [Amazon Security Lake](https://docs.aws.amazon.com/security-lake/latest/userguide/what-is-security-lake.html) converts security log and event data to the [Open Cybersecurity Schema Framework](https://docs.aws.amazon.com/security-lake/latest/userguide/open-cybersecurity-schema-framework.html) (OCSF) to normalize combined data and facilitate its management. OpenSearch supports ingestion of log data from Security Lake in the OCSF format, and Security Analytics can automatically map fields from OCSF to ECS (the default field-mapping schema).

-The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **DNS logs** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.
+The Security Lake log types that can be used as log sources for detector creation include CloudTrail, Route 53, and VPC Flow. Given that Route 53 is a log that captures DNS activity, its log type should be specified as **dns** when [defining a detector](#step-1-define-a-detector). Furthermore, because logs such as CloudTrail can conceivably be captured in both raw format and OCSF, it's good practice to name indexes in a way that keeps these logs separate and easily identifiable. This becomes helpful when specifying an index name in any of the APIs associated with Security Analytics.


Yes. But this representation is opposed by the PM. UX and front-end dev will eventually replace these log type names with revised names. For the time being, this is the way the log type is represented in the UI.

cwillum · 2023-09-19T18:33:21Z

_security-analytics/sec-analytics-config/custom-log-type.md

+After selecting **Create log type** in the **Log types** page, the **Create log type** page opens and provides the necessary fields to create a new log type:
+1. Enter a name for the log type.
+
+   The log type name supports characters a-z (lower case), 0--9, hyphens, and underscores.


Signed-off-by: cwillum <[email protected]>

natebower

@cwillum Please see my comments and changes and let me know if you have any questions. Thanks!

_security-analytics/api-tools/log-type-api.md

_security-analytics/index.md

_security-analytics/sec-analytics-config/custom-log-type.md

_security-analytics/sec-analytics-config/detectors-config.md

_security-analytics/sec-analytics-config/index.md

_security-analytics/usage/overview.md

Signed-off-by: cwillum <[email protected]>

cwillum

All set. Thanks for the thorough review.

_security-analytics/sec-analytics-config/detectors-config.md

_security-analytics/sec-analytics-config/index.md

Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> --------- Signed-off-by: cwillum <[email protected]>

This reverts commit 4ad6fef.

…4969) * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> --------- Signed-off-by: cwillum <[email protected]>

* fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> * fix#4741 custom logtype updates Signed-off-by: cwillum <[email protected]> --------- Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

4b87dfc

Signed-off-by: cwillum <[email protected]>

cwillum added 2 - In progress Issue/PR: The issue or PR is in progress. release-notes PR: Include this PR in the automated release notes security-analytics v2.10.0 labels Sep 6, 2023

cwillum self-assigned this Sep 6, 2023

cwillum requested review from hdhalter, kolchfa-aws, Naarcha-AWS, vagimeli, ananzh, seanneumann, AMoo-Miki and natebower as code owners September 6, 2023 00:27

cwillum added 16 commits September 6, 2023 09:49

fix#4741 custom logtype updates

f49a578

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

77931c2

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

bf8020e

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

6b9912b

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

4f43a9b

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

9d4f9db

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

10a5a58

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

b17657b

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

cf15866

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

5f09eeb

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

1669fee

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

3383f9f

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

60fc156

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

461a7e8

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

3107ae6

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

0a70c29

Signed-off-by: cwillum <[email protected]>

cwillum added 8 commits September 12, 2023 16:54

fix#4741 custom logtype updates

02ef654

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

db25a9e

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

829ab68

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

f5210c3

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

d443444

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

0134802

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

9eae221

Signed-off-by: cwillum <[email protected]>

fix#4741 custom logtype updates

8969c9c

Signed-off-by: cwillum <[email protected]>

cwillum added 4 - Doc review PR: Doc review in progress and removed 2 - In progress Issue/PR: The issue or PR is in progress. labels Sep 18, 2023

Naarcha-AWS reviewed Sep 19, 2023

View reviewed changes

_security-analytics/sec-analytics-config/custom-log-type.md Outdated Show resolved Hide resolved

Naarcha-AWS approved these changes Sep 19, 2023

View reviewed changes

cwillum commented Sep 19, 2023

View reviewed changes

fix#4741 custom logtype updates

0833900

Signed-off-by: cwillum <[email protected]>

cwillum added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels Sep 19, 2023

natebower reviewed Sep 19, 2023

View reviewed changes

fix#4741 custom logtype updates

9f54189

Signed-off-by: cwillum <[email protected]>

cwillum commented Sep 19, 2023

View reviewed changes

_security-analytics/sec-analytics-config/detectors-config.md Outdated Show resolved Hide resolved

_security-analytics/sec-analytics-config/index.md Outdated Show resolved Hide resolved

fix#4741 custom logtype updates

0154048

Signed-off-by: cwillum <[email protected]>

cwillum merged commit dde7c02 into main Sep 19, 2023
4 checks passed

cwillum added 3 - Done Issue is done/complete and removed 5 - Editorial review PR: Editorial review in progress labels Sep 19, 2023

vagimeli added a commit that referenced this pull request Sep 19, 2023

Revert "Add documentation that supports custom log types (#4969)"

fab0952

This reverts commit 4ad6fef.

hdhalter deleted the fix#4741-custom-log-type branch March 28, 2024 21:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation that supports custom log types #4969

Add documentation that supports custom log types #4969

cwillum commented Sep 6, 2023

cwillum commented Sep 11, 2023

cwillum commented Sep 18, 2023 •

edited

Loading

Naarcha-AWS left a comment

Naarcha-AWS Sep 19, 2023

cwillum Sep 19, 2023

Naarcha-AWS Sep 19, 2023

cwillum Sep 19, 2023

cwillum left a comment

cwillum Sep 19, 2023

cwillum Sep 19, 2023

natebower left a comment

cwillum left a comment

Add documentation that supports custom log types #4969

Add documentation that supports custom log types #4969

Conversation

cwillum commented Sep 6, 2023

Description

Issues Resolved

Checklist

cwillum commented Sep 11, 2023

cwillum commented Sep 18, 2023 • edited Loading

Naarcha-AWS left a comment

Choose a reason for hiding this comment

Naarcha-AWS Sep 19, 2023

Choose a reason for hiding this comment

cwillum Sep 19, 2023

Choose a reason for hiding this comment

Naarcha-AWS Sep 19, 2023

Choose a reason for hiding this comment

cwillum Sep 19, 2023

Choose a reason for hiding this comment

cwillum left a comment

Choose a reason for hiding this comment

cwillum Sep 19, 2023

Choose a reason for hiding this comment

cwillum Sep 19, 2023

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

cwillum left a comment

Choose a reason for hiding this comment

cwillum commented Sep 18, 2023 •

edited

Loading