How to force a DNS refresh in otlpexporter?

[bhaskarbanerjee]

Description

Our gateway endpoint is hosted on R53 with failover routing policy on top of primary and secondary records (ALBs backed by ECS).
When the primary goes down (or simulated to go down with 0 targets), we are expecting all traffic to switch to the failover in a reasonable time frame but the observation is that while some traffic does switch over, majority does not. We have run tests for over 10 hours and yet failed to see full traffic switch.

So my question is how can the otlp exporter be configured to force a DNS refresh periodically, when the target is unavailable.
Similar to #5838

Environment

• OS: RHel, ubuntu
• Architecture: x86
• Go Version: [e.g. 1.23.4]
• opentelemetry-go agent version: [v0.94.0]
• opentelemetry-collector version: [v0.104.0]

Steps To Reproduce

1. Client -----> CNAME ----> R53 (failover policy) --- Primary ---- ALB ----> Target Group || Secondary ----- ALB ------> Target Group
2. Run only 1 target per ALB on primary and secondary.
3. After running for 15 mins of steady traffic on Primary, shutdown down the target group size to 0.
4. I tried set up my collector gateway with MAX_CONNECTION_AGE as per OTLP http receiver lacking connection lifetime configuration parity with otlp grpc, prevents load balancing opentelemetry-collector#9211 but the specified version did not start up with those config params

Expected behavior

We expect the entire traffic to switch from Primary to Secondary within a reasonable or configurable time.

[dmathieu]

We do not do any caching of DNS within the OTLP exporter, nor do we handle DNS ourselves for that matter.
The HTTP exporter uses net/http, and the gRPC exporter uses google.golang.org/grpc. So if any DNS caching happens, it would be on their end, or (more likely) down within the stack.

We therefore don't have any way to flush the DNS cache.