Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add APIGATEWAY as S3EventSource #805

Merged
merged 1 commit into from
Jun 12, 2024

Conversation

HannesBBR
Copy link
Contributor

@HannesBBR HannesBBR commented Jun 12, 2024

One of the supported flows for storing API Gateway Access Logs, is by sending logs to an S3 bucket via Firehose Delivery Streams (https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_apigateway.FirehoseLogDestination.html). The name of the delivery stream always needs to start with amazon-apigateway- (https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-logging-to-kinesis.html), which means the object keys in S3 will also have this string.

Currently, the forwarder doesn't support API Gateway as an S3EventSource, which means the host and service for the logs will be set to s3.

What does this PR do?

Adding API GW as a valid event source for logs stored in S3.

Motivation

Having API GW logs properly parsed if they are stored in S3 (through Kinesis Firehose)

Testing Guidelines

Built my branch locally, and pushed the changes to an existing Forwarder Lambda (using the steps described in the readme). Sending the same S3 event for an API GW log, now correctly sets the source to apigateway.

Additional Notes

Types of changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog
  • This PR passes the integration tests (ask a Datadog member to run the tests)
  • This PR passes the unit tests
  • This PR passes the installation tests (ask a Datadog member to run the tests)

One of the supported flows for storing API Gateway Access Logs, is by sending logs to an S3 bucket via Firehose Delivery Streams. 

Currently, the forwarder doesn't support API Gateway as an S3EventSource, which means the `host` and `service` for the logs will be set to `s3`.
Copy link
Contributor

@ge0Aja ge0Aja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ge0Aja ge0Aja merged commit 785c911 into DataDog:master Jun 12, 2024
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants