filter_aws: Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs #7

nathalapooja · 2024-12-30T02:18:22Z

Modified aws filter plugin to extract additional resource entity attributes
Modified cloudwatch logs output plugin to add resource entity in PLE calls

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

Example configuration file for the change

        dataplane-log.conf: |
          [INPUT]
            Name                systemd
            Tag                 dataplane.systemd.*
            Systemd_Filter      _SYSTEMD_UNIT=docker.service
            Systemd_Filter      _SYSTEMD_UNIT=containerd.service
            Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
            DB                  /var/fluent-bit/state/systemd.db
            Path                /var/log/journal
            Read_From_Tail      ${READ_FROM_TAIL}

          [INPUT]
            Name                tail
            Tag                 dataplane.tail.*
            Path                /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
            multiline.parser    docker, cri
            DB                  /var/fluent-bit/state/flb_dataplane_tail.db
            Mem_Buf_Limit       50MB
            Skip_Long_Lines     On
            Refresh_Interval    10
            Rotate_Wait         30
            storage.type        filesystem
            Read_from_Head      ${READ_FROM_HEAD}

          [FILTER]
            Name                modify
            Match               dataplane.systemd.*
            Rename              _HOSTNAME                   hostname
            Rename              _SYSTEMD_UNIT               systemd_unit
            Rename              MESSAGE                     message
            Remove_regex        ^((?!hostname|systemd_unit|message).)*$

          [FILTER]
            Name                aws
            Match               dataplane.*
            imds_version        v2
            enable_entity       true
            entity_type         resource

          [OUTPUT]
            Name                cloudwatch_logs
            Match               dataplane.*
            region              ${AWS_REGION}
            log_group_name      /aws/containerinsights/${CLUSTER_NAME}/dataplane
            log_stream_prefix   ${HOST_NAME}-
            auto_create_group   true
            extra_user_agent    container-insights
            entity_type         resource
            add_entity          true
        host-log.conf: |
          [INPUT]
            Name                tail
            Tag                 host.dmesg
            Path                /var/log/dmesg
            Key                 message
            DB                  /var/fluent-bit/state/flb_dmesg.db
            Mem_Buf_Limit       5MB
            Skip_Long_Lines     On
            Refresh_Interval    10
            Read_from_Head      ${READ_FROM_HEAD}

          [INPUT]
            Name                tail
            Tag                 host.messages
            Path                /var/log/messages
            Parser              syslog
            DB                  /var/fluent-bit/state/flb_messages.db
            Mem_Buf_Limit       5MB
            Skip_Long_Lines     On
            Refresh_Interval    10
            Read_from_Head      ${READ_FROM_HEAD}

          [INPUT]
            Name                tail
            Tag                 host.secure
            Path                /var/log/secure
            Parser              syslog
            DB                  /var/fluent-bit/state/flb_secure.db
            Mem_Buf_Limit       5MB
            Skip_Long_Lines     On
            Refresh_Interval    10
            Read_from_Head      ${READ_FROM_HEAD}

          [FILTER]
            Name                aws
            Match               host.*
            imds_version        v2
            enable_entity       true
            entity_type         resource

          [OUTPUT]
            Name                cloudwatch_logs
            Match               host.*
            region              ${AWS_REGION}
            log_group_name      /aws/containerinsights/${CLUSTER_NAME}/host
            log_stream_prefix   ${HOST_NAME}.
            auto_create_group   true
            extra_user_agent    container-insights
            entity_type         resource
            add_entity          true

Debug log output from testing the change
For Dataplane logs on EKS cluster

[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] entity platform is added eks
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] stream->entity->root_filter_count 2
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] entity_add_resource_key_attributes is called
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] stream entity resource platform eks
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] setting platform to eks eks
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] entity {"logGroupName":"/aws/containerinsights/compass-ga2/dataplane","logStreamName":"ip-192-168-2-9.ec2.internal-dataplane.systemd.kubelet.service","entity":{"keyAttributes":{"Type":"Resource","ResourceType":"AWS::EKS::Cluster","Identifier":"compass-ga2"}},"logEvents":[
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] cloudwatch:PutLogEvents: events=1, payload=900 bytes
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Sending log events to log stream ip-192-168-2-9.ec2.internal-dataplane.systemd.kubelet.service
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] data buf {"logGroupName":"/aws/containerinsights/compass-ga2/dataplane","logStreamName":"ip-192-168-2-9.ec2.internal-dataplane.systemd.kubelet.service","entity":{"keyAttributes":{"Type":"Resource","ResourceType":"AWS::EKS::Cluster","Identifier":"compass-ga2"}},"logEvents":[{"timestamp":1735314537930,"message":"{\"systemd_unit\":\"kubelet.service\",\"hostname\":\"ip-192-168-2-9.ec2.internal\",\"message\":\"E1227 15:48:57.930646    2487 pod_workers.go:965] \\\"Error syncing pod, skipping\\\" err=\\\"failed to \\\\\\\"StartContainer\\\\\\\" for \\\\\\\"otc-container\\\\\\\" with ImagePullBackOff: \\\\\\\"Back-off pulling image \\\\\\\\\\\\\\\"public.ecr.aws/cloudwatch-agent/cloudwatch-agent:1.300051.0b992\\\\\\\\\\\\\\\"\\\\\\\"\\\" pod=\\\"amazon-cloudwatch/cloudwatch-agent-9sgrn\\\" podUID=884d1655-a56a-4201-a5c5-a2934b4eee57\",\"az\":\"us-east-1d\",\"ec2_instance_id\":\"i-0cc95093249392a3f\"}"}]}
[2024/12/27 15:49:01] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] PutLogEvents http status=200

For Host logs in EKS cluster

[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] stream->entity->root_filter_count 2
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] entity platform is added eks
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] entity cluster name is added compass-ga2
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] entity platform is added eks
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] stream->entity->root_filter_count 2
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] stream->entity->root_filter_count 2
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] entity_add_resource_key_attributes is called
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] stream entity resource platform eks
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] setting platform to eks eks
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] entity {"logGroupName":"/aws/containerinsights/compass-ga2/host","logStreamName":"ip-192-168-2-9.ec2.internal.host.messages","entity":{"keyAttributes":{"Type":"Resource","ResourceType":"AWS::EKS::Cluster","Identifier":"compass-ga2"}},"logEvents":[
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] cloudwatch:PutLogEvents: events=2, payload=1206 bytes
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] Sending log events to log stream ip-192-168-2-9.ec2.internal.host.messages
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] data buf {"logGroupName":"/aws/containerinsights/compass-ga2/host","logStreamName":"ip-192-168-2-9.ec2.internal.host.messages","entity":{"keyAttributes":{"Type":"Resource","ResourceType":"AWS::EKS::Cluster","Identifier":"compass-ga2"}},"logEvents":[{"timestamp":1735314876000,"message":"{\"host\":\"ip-192-168-2-9\",\"ident\":\"kubelet\",\"message\":\"I1227 15:54:36.929262    2487 scope.go:115] \\\"RemoveContainer\\\" containerID=\\\"c2a79d41c2ef5f4dced735138985de3ddf07a9f2c70bd9f4bf80ef4aad5118fd\\\"\",\"az\":\"us-east-1d\",\"ec2_instance_id\":\"i-0cc95093249392a3f\"}"},{"timestamp":1735314876000,"message":"{\"host\":\"ip-192-168-2-9\",\"ident\":\"kubelet\",\"message\":\"E1227 15:54:36.929677    2487 pod_workers.go:965] \\\"Error syncing pod, skipping\\\" err=\\\"failed to \\\\\\\"StartContainer\\\\\\\" for \\\\\\\"aws-guardduty-agent\\\\\\\" with CrashLoopBackOff: \\\\\\\"back-off 5m0s restarting failed container=aws-guardduty-agent pod=aws-guardduty-agent-d8dlq_amazon-guardduty(9516d6bb-3ea3-4ae1-9a00-b602a0ba0ead)\\\\\\\"\\\" pod=\\\"amazon-guardduty/aws-guardduty-agent-d8dlq\\\" podUID=9516d6bb-3ea3-4ae1-9a00-b602a0ba0ead\",\"az\":\"us-east-1d\",\"ec2_instance_id\":\"i-0cc95093249392a3f\"}"}]}
[2024/12/27 15:54:39] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] entity cluster name is added compass-ga2

Attached Valgrind output that shows no leaks or memory corruption was found
For cloudwatch logs output plugin: flb-rt-out_cloudwatch

SUCCESS: All unit tests have passed.
==1052== 
==1052== HEAP SUMMARY:
==1052==     in use at exit: 0 bytes in 0 blocks
==1052==   total heap usage: 2 allocs, 2 frees, 1,168 bytes allocated
==1052== 
==1052== All heap blocks were freed -- no leaks are possible
==1052== 
==1052== For lists of detected and suppressed errors, rerun with: -s
==1052== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

Run local packaging test showing all targets (including any new ones) build.
Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

Documentation required for this feature

Backporting

Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

…lugin

nathalapooja added 8 commits December 19, 2024 10:31

added logic to retrieve cluster name and cluster type in aws filter p…

2ddebb8

…lugin

"Compiled code"

2b52bdf

added some logic to cloudwatch

56262c4

added logic in aws plugin to send required attributes

51c2e03

added logic to build entity in cloudwatch logs

ce0246c

working entity logic for dataplane end to end

fee1434

refactored aws code

96814cb

refactored cloudwatch output plugin

4781687

nathalapooja had a problem deploying to pr December 30, 2024 02:18 — with GitHub Actions Failure

nathalapooja deployed to pr December 30, 2024 02:18 — with GitHub Actions Active

nathalapooja changed the title ~~adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs~~ Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs Dec 30, 2024

nathalapooja changed the title ~~Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs~~ filter_aws: Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs Dec 30, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filter_aws: Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs #7

filter_aws: Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs #7

nathalapooja commented Dec 30, 2024 •

edited

Loading

filter_aws: Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs #7

Are you sure you want to change the base?

filter_aws: Adds resource entity to PLE calls in cloudwatch logs plugin for dataplane and host logs #7

Conversation

nathalapooja commented Dec 30, 2024 • edited Loading

nathalapooja commented Dec 30, 2024 •

edited

Loading