[question] about configuring histogram buckets

[let-skip-cake]

Component

class MeterProviderBuilderExtensions
method static MeterProviderBuilder AddView(this MeterProviderBuilder meterProviderBuilder, string instrumentName, MetricStreamConfiguration metricStreamConfiguration)

Package Version

Package Name	Version
OpenTelemetry	1.9.0
OpenTelemetry.Api	1.9.0
OpenTelemetry.Extensions.Hosting	1.9.0
OpenTelemetry.Instrumentation.Http	1.9.0
OpenTelemetry.Instrumentation.Runtime	1.9.0
OpenTelemetry.Instrumentation.AspNetCore	1.9.0
OpenTelemetry.Exporter.OpenTelemetryProtocol	1.9.0
prometheus-net	1.9.0
prometheus-net.AspNetCore	1.9.0

Runtime Version

net8.0

Description

Hello, I am encountering an issue and would appreciate your help.

I have a Docker container set up with Prometheus and Grafana. Prometheus collects metrics from an application running on the host machine. I need to integrate a project using version 1.9.0 of the library (the latest stable version). The use of stable versions is strictly required for our project.

I am trying to configure metrics for our production project, but I am struggling to understand why the bucket separation does not seem to be working.

I wrote the following example based on these references:

    https://github.com/open-telemetry/opentelemetry-dotnet/discussions/3245
    https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/customizing-the-sdk#specify-custom-boundaries-for-histogram

Based on this and the tests in the library, I wrote the following example code. However, the metric configuration does not appear in Prometheus. Furthermore, bucket separation does not seem to work when checking /metrics.

using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using Prometheus;

namespace PrometheusWebApi.Extensions;

public static class ServiceExtensions
{
    internal static IServiceCollection AddOpenTelemetry(this IServiceCollection services, string instanceName)
    {
        var bucketConfig = new ExplicitBucketHistogramConfiguration
        {
            Boundaries = [50d, 100d, 250d, 500d, 1000d, 2000d, 5000d],
            RecordMinMax = true,
        };

        services.AddOpenTelemetry()
            .ConfigureResource(resource => resource
                .AddService(serviceName: instanceName))
            .WithTracing(tracing => tracing
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddOtlpExporter())
            .WithMetrics(metrics => metrics
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddRuntimeInstrumentation()
                .AddView("*", bucketConfig)
                .AddView("*_compute*", bucketConfig)
                .AddView("*.compute*", bucketConfig)
                .AddView("*?compute*", bucketConfig)
                .AddView("*compute*", bucketConfig)
                .AddView("*test*", bucketConfig)
                .AddView("test*", bucketConfig)
                .AddView("*test", bucketConfig)
                .AddView("test?computed", bucketConfig)
                .AddView("test_computed", bucketConfig)
                .AddView("test.computed", bucketConfig)
                .AddView("test*computed", bucketConfig)
                .AddOtlpExporter()
            );

        return services;
    }

    internal static IApplicationBuilder UseOpenTelemetry(this IApplicationBuilder app)
    {
        app.UseHttpMetrics()
            .UseMetricServer();

        return app;
    }
}

Also, here is the class that defines the metric:

using System.Diagnostics.Metrics;

namespace PrometheusWebApi.Meter;

public class WeatherHistogram
{
    private readonly Histogram<double> _histogram;

    public WeatherHistogram(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("test");
        _histogram = meter.CreateHistogram<double>("computed", unit: "ms", description: "Calculate execution time when execute rules.");
    }

    public void Rec(long ms, long legalEntityId, long serviceId, long merchantId, long providerId, long? terminalId)
    {
        _histogram.Record(ms, [
            new("legal.entity.id", legalEntityId),
            new("service.id", serviceId),
            new("merchant.id", merchantId),
            new("provider.id", providerId),
            new("terminal.id", terminalId)
        ]);
    }
}

I've included the project archive in the description. Could you please help me identify what might be causing the issue?

Steps to Reproduce

1. Install or restore packages
2. Build & Run project
3. Open url https://localhost:7515/metrics. Configured in Properties/launchSettings.json
4. Find test_computed metric, and see result
   

Expected Result

See the histogram buckets into new segments

Actual Result

See the histogram buckets into default segments

Additional Context

PrometheusWebApi.zip

[github-actions]

Tagging component owner(s).

@twenzel @xiang17

[TimothyMothra]

Hi @let-skip-cake,

1. Please review this article: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/customizing-the-sdk#configuring-the-aggregation-of-a-histogram
   This section was recently rewritten.
2. I see you've created 12 Views, I'm not sure what you're trying to do here. Please test with one view at a time.

[let-skip-cake]

@TimothyMothra Apologies, but the documentation seems outdated.
https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/customizing-the-sdk#explicit-bucket-histogram-aggregation

By default, the histogram assigns values as le="2^n", where n is the metric's position index. The 12 Views were added during testing to demonstrate the issue.

[cijothomas]

@TimothyMothra Apologies, but the documentation seems outdated. https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/customizing-the-sdk#explicit-bucket-histogram-aggregation

It looks up-to-date to me. Happy to fix, if you can be specific on what is missing/outdated.

By default, the histogram assigns values as le="2^n", where n is the metric's position index. The 12 Views were added during testing to demonstrate the issue.

Please share a minimal reproducible example. You can start with https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/metrics/customizing-the-sdk/Program.cs#L32 which shows how to provide custom buckets.

[let-skip-cake]

Hello @cijothomas
I have attached a project with an example of behavior, in the first message.
I have rechecked the archive, everything is fine with it. I will duplicate it again. Please let me know if you have any problems with the example project.
PrometheusWebApi.zip

[cijothomas]

@let-skip-cake It doesn't look like you are using Prometheus Exporter from OpenTelemetry. "prometheus-net" and "prometheus-net-aspnetcore" are not related to OpenTelemetry.

    <PackageReference Include="OpenTelemetry" Version="1.9.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.9.0" />
    <PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.9.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.9.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.9.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Runtime" Version="1.9.0" />
    <PackageReference Include="prometheus-net" Version="8.2.1" />
    <PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="6.4.0" />

Please make sure you use OpenTelemetry Prometheus Exporter and then also follow my earlier suggestion:

Please share a minimal reproducible example. You can start with https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/metrics/customizing-the-sdk/Program.cs#L32 which shows how to provide custom buckets.

[let-skip-cake]

@cijothomas
Sorry, but I don’t quite understand what you’re suggesting.

As you may notice, I'm working with an ASP.NET application, and my issue is that the bucket configuration isn’t applied in the view. If I create a console application following your example, the output to the console is correct. However, this isn't relevant to what I need.

I've already reviewed and adapted this code for ASP.NET, but the bucket configuration still doesn’t apply, and I can’t figure out why.

I'm using an ASP.NET project with dependency injection, and OpenTelemetry data is exported to Prometheus. The two libraries you mentioned are indeed used to create the telemetry data endpoint for Prometheus.

internal static IApplicationBuilder UseOpenTelemetry(this IApplicationBuilder app)
{
    app.UseHttpMetrics()
        .UseMetricServer();

    return app;
}

Removing these libraries won’t affect the code above.
And as you can see, the AddOtlpExporter method is used in test project:

services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: instanceName))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddView("*", bucketConfig)
        .AddOtlpExporter()
    );

return services;

[cijothomas]

@let-skip-cake
I am suggesting to create a minimal, reproducible example, using OpenTelemtry packages. This stackoverflow post has a some good suggestions on how to do that.

It is not clear from the description what is the issue you are facing. Also, the attached repro uses non-opentelemetry based prometheus packages, so we can't really offer any help with that.

The following code snippet shared above does not even look like OpenTelemetry. (There is no UseMetricServer named method in OpenTelemetry).

internal static IApplicationBuilder UseOpenTelemetry(this IApplicationBuilder app)
{
    app.UseHttpMetrics()
        .UseMetricServer();

    return app;
}

To repeat, a minimal-reproducible-example will help us understand the issue being reported better and then we can help better.

[let-skip-cake]

@cijothomas

I think I understand now. It seems the bucket configurations are indeed generated by the Prometheus library.

Could you please advise if it's possible to work with OpenTelemetry and Prometheus without using pre-release libraries? Or would it be better to directly use Prometheus metrics along with Prometheus libraries?

This is all for ASP.NET 8, by the way. Thank you in advance

[cijothomas]

Could you please advise if it's possible to work with OpenTelemetry and Prometheus without using pre-release libraries?

See if this helps: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/metrics/getting-started-prometheus-grafana