Utility for building a lambda layer artifact #953

srprash · 2024-11-18T19:10:28Z

Description of changes

Adding a script and utilities to build a .zip artifact that will be published as the ADOT Java Lambda Layer.
The layer packages patched version of upstream (opentelemetry-java-instrumentation) and the current source build of aws-opentelemetry-java-instrumentation, including the Application Signals logic.
The layer building logic is primarily borrowed from what we have in the aws-otel-lambda repo, and is tweaked to the build a javaagent instrumentation only layer from the current repo.

Testing

Setup

run lambda-layer/build-layer.sh
- This will produce a opentelemetry-javaagent-layer.zip artifact.
From the AWS Lambda console, created a Java 17 function. The code makes a remote call using OkHttp to www.amazon.com
From the AWS Lambda console, create a lambda layer using the opentelemetry-javaagent-layer.zip file obtained in previous step.

Set the following environment variables on the lambda function:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
OTEL_AWS_APPLICATION_SIGNALS_ENABLED=true
OTEL_LOGS_EXPORTER=none
OTEL_METRICS_EXPORTER=none
OTEL_TRACES_EXPORTER=otlp

Enable active tracing on the function
Invoke the function from lambda console

Result

Trace view

Trace timeline view

Trace raw data

{
    "Id": "1-673ccffc-3f2d31933a2b0f341ee61980",
    "Duration": 7.728,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "8aba0f96e08d63a1",
            "Document": {
                "id": "8aba0f96e08d63a1",
                "name": "java-lambda-example",
                "start_time": 1732038657.532657,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038660.474236,
                "parent_id": "31aa4a42384b852d",
                "aws": {
                    "cloudwatch_logs": [
                        {
                            "log_group": "/aws/lambda/java-lambda-example"
                        }
                    ]
                },
                "annotations": {
                    "aws:responseLatency": 2939.22,
                    "aws:runtimeOverhead": 1.565,
                    "aws:responseDuration": 0.101
                },
                "origin": "AWS::Lambda::Function",
                "subsegments": [
                    {
                        "id": "55219d8790c63176",
                        "name": "Init",
                        "start_time": 1732038653.075805,
                        "end_time": 1732038657.52772
                    },
                    {
                        "id": "8397e02d88907ef8",
                        "name": "Overhead",
                        "start_time": 1732038660.47,
                        "end_time": 1732038660.471565
                    }
                ]
            }
        },
        {
            "Id": "31aa4a42384b852d",
            "Document": {
                "id": "31aa4a42384b852d",
                "name": "java-lambda-example",
                "start_time": 1732038652.748,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038660.476,
                "http": {
                    "response": {
                        "status": 200
                    }
                },
                "aws": {
                    "request_id": "e37debae-de16-47fb-8a15-be426c90df82"
                },
                "origin": "AWS::Lambda",
                "resource_arn": "arn:aws:lambda:us-west-2:702258172533:function:java-lambda-example"
            }
        },
        {
            "Id": "0a87c356b3cbd3eb",
            "Document": {
                "id": "0a87c356b3cbd3eb",
                "name": "java-lambda-example",
                "start_time": 1732038657.649404,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038660.007994,
                "parent_id": "8aba0f96e08d63a1",
                "aws": {
                    "span.kind": "LOCAL_ROOT"
                },
                "annotations": {
                    "aws.local.service": "java-lambda-example",
                    "span.name": "java-lambda-example",
                    "aws.local.operation": "java-lambda-example/FunctionHandler",
                    "span.kind": "SERVER",
                    "aws.local.environment": "lambda:default"
                },
                "metadata": {
                    "process.command_args": [
                        "/var/lang/bin/java",
                        "-XX:MaxHeapSize=445645k",
                        "-javaagent:/var/runtime/lib/Log4jHotPatch.jar=log4jFixerVerbose=false",
                        "-XX:+UseSerialGC",
                        "-Xshare:on",
                        "-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa",
                        "-XX:+TieredCompilation",
                        "-XX:TieredStopAtLevel=1",
                        "--add-opens=java.base/java.io=ALL-UNNAMED",
                        "-Dorg.crac.Core.Compat=com.amazonaws.services.lambda.crac",
                        "-XX:+ErrorFileToStderr",
                        "-Dcom.amazonaws.services.lambda.runtime.api.client.runtimeapi.NativeClient.JNI=/var/runtime/lib/jni/libaws-lambda-jni.linux-x86_64.so",
                        "-classpath",
                        "/var/runtime/lib/aws-lambda-java-core-1.2.3.jar:/var/runtime/lib/aws-lambda-java-runtime-interface-client-2.6.0-linux-x86_64.jar:/var/runtime/lib/aws-lambda-java-serialization-1.1.5.jar",
                        "com.amazonaws.services.lambda.runtime.api.client.AWSLambda",
                        "com.example.LambdaHandler"
                    ],
                    "process.runtime.version": "17.0.13+11-LTS",
                    "os.type": "linux",
                    "thread.name": "main",
                    "process.pid": 2,
                    "telemetry.sdk.name": "opentelemetry",
                    "telemetry.sdk.language": "java",
                    "process.runtime.name": "OpenJDK Runtime Environment",
                    "os.description": "Linux 5.10.227-239.884.amzn2.x86_64",
                    "host.arch": "amd64",
                    "cloud.resource_id": "arn:aws:lambda:us-west-2:702258172533:function:java-lambda-example",
                    "host.name": "169.254.30.77",
                    "telemetry.sdk.version": "1.34.1",
                    "cloud.platform": "aws_lambda",
                    "telemetry.auto.version": "1.32.1-adot-lambda1-aws",
                    "thread.id": 1,
                    "cloud.region": "us-west-2",
                    "service.name": "java-lambda-example",
                    "faas.name": "java-lambda-example",
                    "cloud.provider": "aws",
                    "faas.invocation_id": "e37debae-de16-47fb-8a15-be426c90df82",
                    "cloud.account.id": "702258172533",
                    "process.executable.path": "/var/lang/bin/java",
                    "faas.version": "$LATEST",
                    "PlatformType": "AWS::Lambda",
                    "process.runtime.description": "Amazon.com Inc. OpenJDK 64-Bit Server VM 17.0.13+11-LTS"
                },
                "subsegments": [
                    {
                        "id": "5891a91ff13a3361",
                        "name": "www.amazon.com:443",
                        "start_time": 1732038659.8066413,
                        "end_time": 1732038659.972645,
                        "http": {
                            "request": {
                                "url": "https://www.amazon.com/",
                                "method": "GET"
                            },
                            "response": {
                                "status": 200
                            }
                        },
                        "aws": {
                            "span.kind": "CLIENT"
                        },
                        "annotations": {
                            "aws.local.service": "java-lambda-example",
                            "span.name": "GET",
                            "aws.local.operation": "java-lambda-example/FunctionHandler",
                            "span.kind": "CLIENT",
                            "aws.remote.service": "www.amazon.com:443",
                            "aws.remote.operation": "GET /",
                            "aws.local.environment": "lambda:default"
                        },
                        "metadata": {
                            "http.url": "https://www.amazon.com/",
                            "process.command_args": [
                                "/var/lang/bin/java",
                                "-XX:MaxHeapSize=445645k",
                                "-javaagent:/var/runtime/lib/Log4jHotPatch.jar=log4jFixerVerbose=false",
                                "-XX:+UseSerialGC",
                                "-Xshare:on",
                                "-XX:SharedArchiveFile=/var/lang/lib/server/runtime.jsa",
                                "-XX:+TieredCompilation",
                                "-XX:TieredStopAtLevel=1",
                                "--add-opens=java.base/java.io=ALL-UNNAMED",
                                "-Dorg.crac.Core.Compat=com.amazonaws.services.lambda.crac",
                                "-XX:+ErrorFileToStderr",
                                "-Dcom.amazonaws.services.lambda.runtime.api.client.runtimeapi.NativeClient.JNI=/var/runtime/lib/jni/libaws-lambda-jni.linux-x86_64.so",
                                "-classpath",
                                "/var/runtime/lib/aws-lambda-java-core-1.2.3.jar:/var/runtime/lib/aws-lambda-java-runtime-interface-client-2.6.0-linux-x86_64.jar:/var/runtime/lib/aws-lambda-java-serialization-1.1.5.jar",
                                "com.amazonaws.services.lambda.runtime.api.client.AWSLambda",
                                "com.example.LambdaHandler"
                            ],
                            "process.runtime.version": "17.0.13+11-LTS",
                            "os.type": "linux",
                            "thread.name": "main",
                            "process.pid": 2,
                            "telemetry.sdk.name": "opentelemetry",
                            "net.peer.name": "www.amazon.com",
                            "telemetry.sdk.language": "java",
                            "process.runtime.name": "OpenJDK Runtime Environment",
                            "os.description": "Linux 5.10.227-239.884.amzn2.x86_64",
                            "host.arch": "amd64",
                            "net.protocol.name": "http",
                            "host.name": "169.254.30.77",
                            "telemetry.sdk.version": "1.34.1",
                            "http.method": "GET",
                            "cloud.platform": "aws_lambda",
                            "telemetry.auto.version": "1.32.1-adot-lambda1-aws",
                            "thread.id": 1,
                            "net.peer.port": 443,
                            "cloud.region": "us-west-2",
                            "service.name": "java-lambda-example",
                            "faas.name": "java-lambda-example",
                            "http.status_code": 200,
                            "cloud.provider": "aws",
                            "process.executable.path": "/var/lang/bin/java",
                            "faas.version": "$LATEST",
                            "PlatformType": "AWS::Lambda",
                            "process.runtime.description": "Amazon.com Inc. OpenJDK 64-Bit Server VM 17.0.13+11-LTS",
                            "net.protocol.version": "2"
                        },
                        "namespace": "remote"
                    }
                ]
            }
        },
        {
            "Id": "2f12e028332d91f3",
            "Document": {
                "id": "2f12e028332d91f3",
                "name": "www.amazon.com:443",
                "start_time": 1732038659.8066413,
                "trace_id": "1-673ccffc-3f2d31933a2b0f341ee61980",
                "end_time": 1732038659.972645,
                "parent_id": "5891a91ff13a3361",
                "inferred": true,
                "http": {
                    "request": {
                        "url": "https://www.amazon.com/",
                        "method": "GET"
                    },
                    "response": {
                        "status": 200
                    }
                },
                "annotations": {
                    "aws.local.service": "www.amazon.com:443",
                    "aws.local.operation": "GET /"
                }
            }
        }
    ]
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This reverts commit dbc32a1.

lambda-layer/build-layer.sh

bjrara · 2024-11-19T19:12:21Z

Should we also set OTEL_AWS_APPLICATION_SIGNALS_RUNTIME_ENABLED to false on Lambda?

lambda-layer/build-layer.sh

srprash · 2024-11-19T23:17:32Z

Should we also set OTEL_AWS_APPLICATION_SIGNALS_RUNTIME_ENABLED to false on Lambda?

Ah yes. That's why my lambda logs were noisy.

codecov-commenter · 2024-11-22T07:20:58Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.66%. Comparing base (09e6487) to head (d662fe9).
Report is 371 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@              Coverage Diff              @@
##               main     #953       +/-   ##
=============================================
- Coverage     85.71%   69.66%   -16.05%     
- Complexity       19      308      +289     
=============================================
  Files             3       26       +23     
  Lines            49     1134     +1085     
  Branches          5      149      +144     
=============================================
+ Hits             42      790      +748     
- Misses            3      294      +291     
- Partials          4       50       +46

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests

srprash added 4 commits November 18, 2024 11:09

lambda layer build script and utility

1a71fb4

reverting to using adot1 after syncing with Michael He

dbc32a1

Revert "reverting to using adot1 after syncing with Michael He"

87b785f

This reverts commit dbc32a1.

cleaning up some script

c7c5378

srprash marked this pull request as ready for review November 19, 2024 18:19

srprash requested a review from a team as a code owner November 19, 2024 18:19

srprash changed the title ~~Build lambda layer~~ Utility for building a lambda layer artifact Nov 19, 2024

wangzlei reviewed Nov 19, 2024

View reviewed changes

lambda-layer/build-layer.sh Show resolved Hide resolved

bjrara previously approved these changes Nov 19, 2024

View reviewed changes

wangzlei reviewed Nov 19, 2024

View reviewed changes

lambda-layer/build-layer.sh Show resolved Hide resolved

disable runtime metrics

a602679

srprash dismissed bjrara’s stale review via a602679 November 19, 2024 23:27

get otel version from .github/patches/versions file

d662fe9

wangzlei approved these changes Nov 22, 2024

View reviewed changes

Merge branch 'main' into main

c3aa4d5

srprash merged commit 015c6db into aws-observability:main Nov 22, 2024
4 checks passed

srprash mentioned this pull request Nov 26, 2024

Renaming to otel-instrument and default configs #956

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Utility for building a lambda layer artifact #953

Utility for building a lambda layer artifact #953

srprash commented Nov 18, 2024 •

edited

Loading

bjrara commented Nov 19, 2024

srprash commented Nov 19, 2024

codecov-commenter commented Nov 22, 2024

Utility for building a lambda layer artifact #953

Utility for building a lambda layer artifact #953

Conversation

srprash commented Nov 18, 2024 • edited Loading

Description of changes

Testing

Setup

Result

Trace view

Trace timeline view

Trace raw data

bjrara commented Nov 19, 2024

srprash commented Nov 19, 2024

codecov-commenter commented Nov 22, 2024

Codecov Report

srprash commented Nov 18, 2024 •

edited

Loading