Use otel collector

[johnbley]

Prepare to use the otel collector rather than having each language instrumentation post directly to ingest. This should reduce latency for customers' apps/functions.
The splunk-otel-lambda side of this work involves modifying the scripts to unset SPLUNK_REALM and friends so they fall back to the otel default behavior of talking OTLP to localhost. Along the way, I noticed and fixed a problem where we had forked some upstream otel-lambda wrapper scripts rather than simply applying small edits to them during our build process.

NB: based on past experience, this process will likely take a few cycles of CI to iron out tiny script problems.

[dude0001]

@johnbley could you give some more explanation of what this change is doing? Are the steps to set SPLUNK_REALM and SPLUNK_ACCESS_TOKEN in the configuration section of the readme still relevant? It seems like with this change the splunk-default-config script immediately unset's these, so I'm unclear why we'd set those still. How should the token be configured after this change?

[tsloughter]

@dude0001 they are used by the collector that is included in the layer, instead of by the individual SDKs, which is why its unset in the script.

[dude0001]

Thank you @tsloughter for the further clarification. Is there a complete example of how to consume this layer after this change? I'm trying to upgrade to latest to get some other fixes. However, on latest I'm seeing 401 Unuauthorized errors assumingly because the SPLUNK_ACCESS_TOKEN I was setting up in an environment variable is getting unset now. I'm unclear how our Lambda should be authenticating now when ingesting traces directly into Splunk e.g. https://ingest.us1.signalfx.com:443/v2/trace/otlp.

[tsloughter]

Just noticed my response yesterday didn't send somehow, sorry about that.

@dude0001 what version of the layers are you using?

Are you setting any env variables besides realm and access token?

[dude0001]

@tsloughter I had to step away from this for a bit, but looking at it again @tsloughter. I am using arn:aws:lambda:us-east-2:254067382080:layer:splunk-apm:365. I am moving this conversation to an issue and opened #129. We are trying to keep the token secret, and this change broke our mechanism for this. Looking for a suggestion on how to accomplish what we need or to add a mechanism to the Splunk Lambda wrapper extension to inject the token as a secret.