Use native Rust support for async traits in LogExporter::export() method (11% improvement)

[lalitb]

Changes

Continuation to #2143, more to show feasibility, performance improvements, and have discussion before raising the PR for eventual review, this PR demonstrate using async traits support in Rust exporter. The support was added in Rust v1.75, while the msrv as of now is v1.70 for otel-sdk

The change is:
Existing:

#[async_trait]
pub trait LogExporter: Send + Sync + Debug {
  async fn export(&mut self, batch: LogBatch<'_>) -> LogResult<()>;
}

PR:

pub trait LogExporter: Send + Sync + Debug {
    fn export<'a>(
        &'a mut self,
        batch: &'a LogBatch<'a>,
    ) -> impl std::future::Future<Output = LogResult<()>> + Send + 'a;
}

There is 11% improvement as seen from stress test:

Before PR:

Number of threads: 16
Throughput: 40,468,186 iterations/sec
Throughput: 40,792,480 iterations/sec
Throughput: 40,456,227 iterations/sec
Throughput: 40,455,461 iterations/sec

After this PR:

Throughput: 44,793,174 iterations/sec
Throughput: 45,321,168 iterations/sec
Throughput: 45,387,969 iterations/sec
Throughput: 45,629,648 iterations/sec
Throughput: 45,607,326 iterations/sec
Throughput: 45,303,331 iterations/sec
Throughput: 45,634,108 iterations/sec

Thanks,
Lalit

Merge requirement checklist

• CONTRIBUTING guidelines followed
• Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

Attention: Patch coverage is 30.05181% with 135 lines in your changes missing coverage. Please review.

Project coverage is 76.7%. Comparing base (f911ed5) to head (e8333f5).

Files with missing lines	Patch %	Lines
opentelemetry-otlp/src/exporter/http/logs.rs	0.0%	37 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/logs.rs	0.0%	28 Missing ⚠️
opentelemetry-stdout/src/logs/exporter.rs	0.0%	25 Missing ⚠️
opentelemetry-otlp/src/logs.rs	0.0%	19 Missing ⚠️
opentelemetry-sdk/src/logs/log_processor.rs	75.9%	13 Missing ⚠️
opentelemetry-appender-tracing/src/layer.rs	0.0%	10 Missing ⚠️
opentelemetry-otlp/src/exporter/http/mod.rs	0.0%	2 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/mod.rs	0.0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #2374     +/-   ##
=======================================
- Coverage   76.9%   76.7%   -0.2%     
=======================================
  Files        122     122             
  Lines      22213   22266     +53     
=======================================
+ Hits       17082   17096     +14     
- Misses      5131    5170     +39     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lalitb]

@lalitb I think it maybe better to first check how to avoid requiring mut reference for exporting, before evaluating the results of this perf test, as that requirement has forced the need of mutex and introduced contention and hence stress tests shows way low throughput than before.

OK, have updated this PR after removing the mut self in #2380. This eliminates the need of mutex for exporter in stress test. The stress result are now similar to earlier i.e., without the inclusion of exporter. Which means, adding futures::block_on() on native async-method doesn't introduce much of latency.

Before this PR (the regression introduced by #2380 because of inclusion of exporter ~44 -> ~40):

Number of threads: 16
Throughput: 40,468,186 iterations/sec
Throughput: 40,792,480 iterations/sec
Throughput: 40,456,227 iterations/sec
Throughput: 40,455,461 iterations/sec

After this PR:

Throughput: 44,793,174 iterations/sec
Throughput: 45,321,168 iterations/sec
Throughput: 45,387,969 iterations/sec
Throughput: 45,629,648 iterations/sec
Throughput: 45,607,326 iterations/sec
Throughput: 45,303,331 iterations/sec
Throughput: 45,634,108 iterations/sec

The cons of introducing the native-async in trait would be

• Bump the msrv to 1.75
• The exporter interface is no longer object-safe. Which means, it can't be used in dynamic dispatch (e.g., via dyn Trait). This introduced slight complexity in otel-otlp, where dyn dispatch was used for http and tonic exporters.

Current export method could be simplified a bit for lifetime specifiers. I will look into that. But in general, open for discussion if we should bump the msrv.

[cijothomas]

Bump the msrv to 1.75
The exporter interface is no longer object-safe. Which means, it can't be used in dynamic dispatch (e.g., via dyn Trait). This introduced slight complexity in otel-otlp, where dyn dispatch was used for http and tonic exporters.

Based on today's SIG discussion:
1.75 bump is acceptable.
the exporter interface not being object safe is an easy to fix issue as this PR already demonstrates fixing it for OTLP.

Awaiting more feedback.

[lalitb]

This is ready for review now, with #2417 to bump msrv is merged

[lalitb]

Have updated the description now to reflect the improvement seen through stress test, which is ~11%