warn: Maximum active stream limit exceeded

[marcusboden]

Bug Description

Hi,

we're getting the warning that we exceeded the max active stream limit. This can be configured, but the charm doesn't offer any way of configuring that. Could we either make that configurable or raise it by default?

To Reproduce

Not 100% sure. Maybe we have a larger number of apps here than usual.

Environment

Running on microk8s on a customer environment:

loki                             2.9.5    active      1  loki-k8s                      latest/edge    105

Relevant log output

2024-06-28T07:11:30.436Z [loki] level=warn ts=2024-06-28T07:11:30.378520587Z caller=grpc_logging.go:57 method=/logproto.Pusher/Push duration=489.418µs err="rpc error: code = Code(429) desc = Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: 'fake'" msg=gRPC

Additional context

(the user fake was a bit confusing to me, but that's just the default user if no multi-tenancy is used)

[lucabello]

We should add the possibility to configure max_global_streams_per_user (which defaults to 5000) via juju config.

[marcusboden]

Yeah, that was my thought as well. Or scale it up depending on how many apps/units are sending data.