diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 2969183..dc182ad 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,6 +1,25 @@
 # Contributing
 
-To make contributions to this charm, you'll need a working [development setup](https://juju.is/docs/sdk/dev-setup).
+## Overview
+
+This documents explains the processes and practices recommended for contributing enhancements to
+this operator.
+
+- Generally, before developing enhancements to this charm, you should consider [opening an issue
+  ](https://github.com/jnsgruk/zinc-k8s-operator/issues) explaining your use case.
+- If you would like to chat with us about your use-cases or proposed implementation, you can reach
+  us at [Canonical Mattermost public channel](https://chat.charmhub.io/charmhub/channels/charm-dev)
+  or [Discourse](https://discourse.charmhub.io/).
+- Familiarising yourself with the [Charmed Operator Framework](https://juju.is/docs/sdk) library
+  will help you a lot when working on new features or bug fixes.
+- All enhancements require review before being merged. Code review typically examines
+  - code quality
+  - test coverage
+  - user experience for Juju administrators this charm.
+- Please help us out in ensuring easy to review branches by rebasing your pull request branch onto
+  the `main` branch. This also avoids merge commits and creates a linear Git commit history.
+
+## Developing
 
 You can use the environments created by `tox` for development:
 
@@ -9,20 +28,18 @@ tox --notest -e unit
 source .tox/unit/bin/activate
 ```
 
-## Testing
-
-This project uses `tox` for managing test environments. There are some pre-configured environments
-that can be used for linting and formatting code when you're preparing contributions to the charm:
+### Testing
 
 ```shell
 tox -e fmt           # update your code according to linting rules
 tox -e lint          # code style
 tox -e unit          # unit tests
+tox -e scenario      # scenario tests
 tox -e integration   # integration tests
-tox                  # runs 'lint' and 'unit' environments
+tox                  # runs 'lint', 'scenario' and 'unit' environments
 ```
 
-## Build the charm
+## Build charm
 
 Build the charm in this git repository using:
 
@@ -30,4 +47,14 @@ Build the charm in this git repository using:
 charmcraft pack
 ```
 
-<!-- You may want to include any contribution/style guidelines in this document>
+### Deploy
+
+```bash
+# Create a model
+juju add-model dev
+# Enable DEBUG logging
+juju model-config logging-config="<root>=INFO;unit=DEBUG"
+# Deploy the charm
+juju deploy ./zinc-k8s_ubuntu-20.04-amd64.charm \
+    --resource zinc-image=public.ecr.aws/m5j1b6u0/zinc:v0.1.1 \
+```
diff --git a/HACKING.md b/HACKING.md
new file mode 100644
index 0000000..247847c
--- /dev/null
+++ b/HACKING.md
@@ -0,0 +1,47 @@
+# Deploying Tempo
+
+To manually test and develop tempo:
+
+    charmcraft pack 
+    charmcraft pack -p ./tests/integration/tester/ 
+    j deploy ./tempo-k8s_ubuntu-22.04-amd64.charm tempo --resource tempo-image=grafana/tempo:2.4.0
+
+you need to always deploy at least 2 units of tester:
+
+    j deploy ./tester_ubuntu-22.04-amd64.charm tester -n 3 --resource workload=python:slim-buster
+    j relate tempo:tracing tester:tracing
+
+# Object storage
+
+In order to test tempo with object storage (using [minio charm](https://github.com/canonical/minio-operator)), 
+you can do the following (replace YOUR_MINIO_ACCESS_KEY and YOUR_MINIO_SECRET with anything you want as long as the
+secret is more than 8 characters):
+
+Deploy minio:
+```
+juju deploy minio --channel edge --trust
+juju config minio access-key=YOUR_MINIO_ACCESS_KEY
+juju config minio secret-key=YOUR_MINIO_SECRET
+```
+
+Deploy mc and set up buckets:
+```
+sudo snap install minio-client --edge --devmode
+kubectl port-forward service/minio -n test-minio 9000:9000
+minio-client config host add local http://localhost:9000 YOUR_MINIO_ACCESS_KEY YOUR_MINIO_SECRET
+minio-client mb local/tempo
+```
+
+Deploy and configure s3-integrator:
+```
+juju deploy s3-integrator
+juju config s3-integrator endpoint=http://MINIO_IP_ADDRESS:9000
+juju config s3-integrator bucket=tempo
+juju run s3-integrator/leader sync-s3-credentials access-key=YOUR_MINIO_ACCESS_KEY secret-key=YOUR_MINIO_SECRET
+```
+
+Deploy tempo:
+```
+juju deploy ./tempo-k8s_ubuntu-22.04-amd64.charm tempo --resource tempo-image=grafana/tempo:2.4.0
+juju integrate tempo s3-integrator
+```
diff --git a/LICENSE b/LICENSE
index d26f7ef..d645695 100644
--- a/LICENSE
+++ b/LICENSE
@@ -187,7 +187,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright 2023 Ubuntu
+   Copyright [yyyy] [name of copyright owner]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
diff --git a/README.md b/README.md
index 1151ce7..c1bf712 100644
--- a/README.md
+++ b/README.md
@@ -1,13 +1,28 @@
-# Tempo Coordinator charm for Kubernetes
+# Tempo Operator
 
-[![CharmHub Badge](https://charmhub.io/tempo-coordinator-k8s/badge.svg)](https://charmhub.io/tempo-coordinator-k8s)
-[![Release](https://github.com/canonical/tempo-coordinator-k8s-operator/actions/workflows/release.yaml/badge.svg)](https://github.com/canonical/tempo-coordinator-k8s-operator/actions/workflows/release.yaml)
+[![CharmHub Badge](https://charmhub.io/tempo-k8s/badge.svg)](https://charmhub.io/tempo-k8s)
+[![Release](https://github.com/canonical/tempo-k8s-operator/actions/workflows/release.yaml/badge.svg)](https://github.com/canonical/tempo-k8s-operator/actions/workflows/release.yaml)
 [![Discourse Status](https://img.shields.io/discourse/status?server=https%3A%2F%2Fdiscourse.charmhub.io&style=flat&label=CharmHub%20Discourse)](https://discourse.charmhub.io)
 
-## Description
+This repository contains the source code for a Charmed Operator that drives [Tempo] on Kubernetes.
 
-This charm serves as a coordinator for a Tempo HA deployment, together with the [tempo-worker-k8s](https://github.com/canonical/tempo-worker-k8s-operator) charm.  
+## Usage
+
+Assuming you have access to a bootstrapped Juju controller on Kubernetes, you can:
+
+```bash
+$ juju deploy tempo-k8s # --trust (use when cluster has RBAC enabled)
+```
+
+## OCI Images
+
+This charm, by default, deploys `grafana/tempo:2.4.0`.
 
 ## Contributing
 
-See the [Juju SDK documentation](https://juju.is/docs/sdk) for more information about developing and improving charms.
+Please see the [Juju SDK docs](https://juju.is/docs/sdk) for guidelines
+on enhancements to this charm following best practice guidelines, and the
+[contributing] doc for developer guidance.
+
+[Tempo]: https://grafana.com/traces/
+[contributing]: https://github.com/PietroPasotti/tempo-k8s-operator/blob/main/CONTRIBUTING.md
diff --git a/charmcraft.yaml b/charmcraft.yaml
index 95abeba..6832689 100644
--- a/charmcraft.yaml
+++ b/charmcraft.yaml
@@ -1,29 +1,136 @@
-# This file configures Charmcraft.
-# See https://juju.is/docs/sdk/charmcraft-config for guidance.
-
+# Copyright 2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+name: tempo-k8s
 type: charm
+
+assumes:
+  - k8s-api
+
+  # Juju 3.4.0 needed for pebble notify
+  - juju >= 3.4.0
+
+description: |
+  Tempo is a distributed tracing backend by Grafana, supporting Jaeger,
+  Zipkin, and OpenTelemetry protocols.
+
+summary: |
+  Tempo is a distributed tracing backend by Grafana.
+
+links:
+  documentation: https://discourse.charmhub.io/t/tempo-k8s-docs-index/14005
+  website:
+    - https://charmhub.io/tempo-k8s
+  source:
+    - https://github.com/canonical/tempo-k8s-operator
+  issues:
+    - https://github.com/canonical/tempo-k8s-operator/issues
+
+containers:
+  tempo:
+    resource: tempo-image
+    mounts:
+      - storage: data
+        location: /tmp/tempo
+
+resources:
+  tempo-image:
+    type: oci-image
+    description: OCI image for Tempo
+    # Included for simplicity in integration tests
+    # see https://hub.docker.com/r/grafana/tempo/tags
+    upstream-source: grafana/tempo:2.4.0
+
+provides:
+  tempo-cluster:
+    interface: tempo_cluster
+    description: |
+      Connect any number of tempo-worker-k8s charms to cluster the Tempo components off to different nodes.
+  grafana-dashboard:
+    interface: grafana_dashboard
+    description: |
+      Forwards the built-in grafana dashboard(s) for monitoring Tempo.
+  grafana-source:
+    interface: grafana_datasource
+    description: |
+      Configures Grafana to be able to use this Tempo instance as a datasource.
+  metrics-endpoint:
+    interface: prometheus_scrape
+    description: |
+      Exposes the Prometheus metrics endpoint providing telemetry about the
+      Tempo instance.
+  tracing:
+    interface: tracing
+    description: |
+      Integration to offer other charms the possibility to send traces to Tempo.
+
+
+requires:
+  s3:
+    interface: s3
+    limit: 1
+    description: |
+      Obtains access to the s3 bucket for data storage.
+  logging:
+    interface: loki_push_api
+    description: |
+      Integration with Loki to push Tempo logs to the observability stack.
+  ingress:
+    interface: traefik_route
+    description: |
+      Ingress integration for Tempo server and Tempo receiver endpoints,
+      so that cross-model workloads can send their traces to Tempo through the ingress.
+      Uses `traefik_route` to open ports on Traefik host for tracing ingesters.
+  certificates:
+    interface: tls-certificates
+    limit: 1
+    description: |
+      Certificate and key files for securing Tempo internal and external 
+      communications with TLS.
+
+storage:
+  data:
+    type: filesystem
+    location: /tempo-data
+
+actions:
+  list-receivers:
+    description: |
+      Returns a list of all enabled receiver endpoints.
+
+peers:
+  tempo-peers:
+    interface: tempo_peers
+    description: |
+      peer relation for internal coordination
+
+config:
+  options:
+    coordinator_runs_workload_when_clustered:
+      type: boolean
+      default: true
+      description: |
+        Whether this charm should also run a worker node when related
+        to any number of specialized worker applications.
+        
+        Set it to ``false`` if you want this charm to stop running `Tempo` 
+        as soon as you integrate it with a tempo-worker-k8s-charm instance. 
+        In this case, tempo-k8s will only act as coordinator 
+        (and reverse proxy) for the tempo cluster.
+
+
 bases:
   - build-on:
-    - name: ubuntu
-      channel: "22.04"
+      - name: "ubuntu"
+        channel: "22.04"
     run-on:
-    - name: ubuntu
-      channel: "22.04"
+      - name: "ubuntu"
+        channel: "22.04"
 
 parts:
   charm:
     charm-binary-python-packages:
-      - pydantic>2.0
-
-      # For v2.tls_certificates
-      - cryptography
-      - jsonschema
-
-  cos-tool:
-    plugin: dump
-    source: .
-    build-packages:
-      - curl
-    override-pull: |
-      curl -L -O https://github.com/canonical/cos-tool/releases/latest/download/cos-tool-${CRAFT_TARGET_ARCH}
-      chmod +x cos-tool-*
+      - "pydantic>=2"
+      - "cryptography"
+      - "jsonschema"
+      - "opentelemetry-exporter-otlp-proto-http==1.21.0"
+
diff --git a/icon.svg b/icon.svg
new file mode 100644
index 0000000..c1f0ea7
--- /dev/null
+++ b/icon.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 121.85 99.17"><defs><style>.cls-1{fill:url(#linear-gradient);}</style><linearGradient id="linear-gradient" x1="168.55" y1="13.4" x2="27.2" y2="57" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="#fff100"/><stop offset="1" stop-color="#f05a28"/></linearGradient></defs><g id="Layer_6" data-name="Layer 6"><path class="cls-1" d="M4.66,25H2.37a2.37,2.37,0,0,0,0,4.74H4.66a2.37,2.37,0,1,0,0-4.74ZM48.48,59.57H46.19a2.37,2.37,0,1,0,0,4.74h2.29a2.37,2.37,0,1,0,0-4.74ZM46.56,37a2.37,2.37,0,0,0-2.37-2.37H36.27a2.37,2.37,0,0,0,0,4.74h7.92A2.37,2.37,0,0,0,46.56,37ZM121.73,22.1,119.32,8.56A9.88,9.88,0,0,0,109.07,0H16.24A6.28,6.28,0,0,0,9.9,7.7l2.54,14.4a3.38,3.38,0,0,0,.08.34v0c.3,1.76-.59,2.47-1.39,2.73h0a2.37,2.37,0,0,0,.79,4.6H115.39A6.28,6.28,0,0,0,121.73,22.1ZM90.15,76.42c-1-5.25-4-7.2-7.39-7.2H58.24a2.39,2.39,0,0,0-2.37,2.4A2.37,2.37,0,0,0,58,74h0c.78.14,1.62,1.16,2.15,3.68l2.52,14a9.59,9.59,0,0,0,9.19,7.54l14.63-.07a6.28,6.28,0,0,0,6.44-7.61ZM57.73,64.48H84.34a2.27,2.27,0,0,0,.59-.09c2.46-.52,2.58-2.52,2.26-4.51L83.8,41.27c-.93-4.84-3.74-6.75-7.43-6.75H52a2.37,2.37,0,0,0-.28,4.72h0c.81.15,1.7,1.24,2.22,4l2.57,14.24v0A1.92,1.92,0,0,1,55,59.87h0a2.36,2.36,0,0,0,.79,4.59h1.9Z"/></g></svg>
\ No newline at end of file
diff --git a/lib/charms/data_platform_libs/v0/s3.py b/lib/charms/data_platform_libs/v0/s3.py
index 7beb113..f5614aa 100644
--- a/lib/charms/data_platform_libs/v0/s3.py
+++ b/lib/charms/data_platform_libs/v0/s3.py
@@ -137,7 +137,7 @@ def _on_credential_gone(self, event: CredentialsGoneEvent):
 
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
-LIBPATCH = 4
+LIBPATCH = 5
 
 logger = logging.getLogger(__name__)
 
@@ -212,7 +212,7 @@ class S3CredentialEvents(CharmEvents):
 class S3Provider(Object):
     """A provider handler for communicating S3 credentials to consumers."""
 
-    on = S3CredentialEvents()  # pyright: ignore [reportGeneralTypeIssues]
+    on = S3CredentialEvents()  # pyright: ignore [reportAssignmentType]
 
     def __init__(
         self,
@@ -481,6 +481,18 @@ def set_s3_api_version(self, relation_id: int, s3_api_version: str) -> None:
         """
         self.update_connection_info(relation_id, {"s3-api-version": s3_api_version})
 
+    def set_delete_older_than_days(self, relation_id: int, days: int) -> None:
+        """Sets the retention days for full backups in application databag.
+
+        This function writes in the application data bag, therefore,
+        only the leader unit can call it.
+
+        Args:
+            relation_id: the identifier for a particular relation.
+            days: the value.
+        """
+        self.update_connection_info(relation_id, {"delete-older-than-days": str(days)})
+
     def set_attributes(self, relation_id: int, attributes: List[str]) -> None:
         """Sets the connection attributes in application databag.
 
@@ -580,6 +592,17 @@ def s3_api_version(self) -> Optional[str]:
 
         return self.relation.data[self.relation.app].get("s3-api-version")
 
+    @property
+    def delete_older_than_days(self) -> Optional[int]:
+        """Returns the retention days for full backups."""
+        if not self.relation.app:
+            return None
+
+        days = self.relation.data[self.relation.app].get("delete-older-than-days")
+        if days is None:
+            return None
+        return int(days)
+
     @property
     def attributes(self) -> Optional[List[str]]:
         """Returns the attributes."""
@@ -613,7 +636,7 @@ class S3CredentialRequiresEvents(ObjectEvents):
 class S3Requirer(Object):
     """Requires-side of the s3 relation."""
 
-    on = S3CredentialRequiresEvents()  # pyright: ignore[reportGeneralTypeIssues]
+    on = S3CredentialRequiresEvents()  # pyright: ignore[reportAssignmentType]
 
     def __init__(
         self, charm: ops.charm.CharmBase, relation_name: str, bucket_name: Optional[str] = None
diff --git a/lib/charms/grafana_k8s/v0/grafana_dashboard.py b/lib/charms/grafana_k8s/v0/grafana_dashboard.py
index 1f1bc4f..dfc32dd 100644
--- a/lib/charms/grafana_k8s/v0/grafana_dashboard.py
+++ b/lib/charms/grafana_k8s/v0/grafana_dashboard.py
@@ -219,7 +219,7 @@ def __init__(self, *args):
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
 
-LIBPATCH = 35
+LIBPATCH = 36
 
 logger = logging.getLogger(__name__)
 
@@ -1050,6 +1050,7 @@ def __init__(
 
         self.framework.observe(self._charm.on.leader_elected, self._update_all_dashboards_from_dir)
         self.framework.observe(self._charm.on.upgrade_charm, self._update_all_dashboards_from_dir)
+        self.framework.observe(self._charm.on.config_changed, self._update_all_dashboards_from_dir)
 
         self.framework.observe(
             self._charm.on[self._relation_name].relation_created,
diff --git a/lib/charms/loki_k8s/v0/loki_push_api.py b/lib/charms/loki_k8s/v0/loki_push_api.py
new file mode 100644
index 0000000..16e1294
--- /dev/null
+++ b/lib/charms/loki_k8s/v0/loki_push_api.py
@@ -0,0 +1,2510 @@
+#!/usr/bin/env python3
+# Copyright 2021 Canonical Ltd.
+# See LICENSE file for licensing details.
+#
+# Learn more at: https://juju.is/docs/sdk
+
+r"""## Overview.
+
+This document explains how to use the two principal objects this library provides:
+
+- `LokiPushApiProvider`: This object is meant to be used by any Charmed Operator that needs to
+implement the provider side of the `loki_push_api` relation interface. For instance, a Loki charm.
+The provider side of the relation represents the server side, to which logs are being pushed.
+
+- `LokiPushApiConsumer`: Used to obtain the loki api endpoint. This is useful for configuring
+  applications such as pebble, or charmed operators of workloads such as grafana-agent or promtail,
+  that can communicate with loki directly.
+
+- `LogProxyConsumer`: DEPRECATED.
+This object can be used by any Charmed Operator which needs to send telemetry, such as logs, to
+Loki through a Log Proxy by implementing the consumer side of the `loki_push_api` relation
+interface.
+
+Filtering logs in Loki is largely performed on the basis of labels. In the Juju ecosystem, Juju
+topology labels are used to uniquely identify the workload which generates telemetry like logs.
+
+In order to be able to control the labels on the logs pushed this object adds a Pebble layer
+that runs Promtail in the workload container, injecting Juju topology labels into the
+logs on the fly.
+
+## LokiPushApiProvider Library Usage
+
+This object may be used by any Charmed Operator which implements the `loki_push_api` interface.
+For instance, Loki or Grafana Agent.
+
+For this purpose a charm needs to instantiate the `LokiPushApiProvider` object with one mandatory
+and three optional arguments.
+
+- `charm`: A reference to the parent (Loki) charm.
+
+- `relation_name`: The name of the relation that the charm uses to interact
+  with its clients, which implement `LokiPushApiConsumer` or `LogProxyConsumer`
+  (note that LogProxyConsumer is deprecated).
+
+  If provided, this relation name must match a provided relation in metadata.yaml with the
+  `loki_push_api` interface.
+
+  The default relation name is "logging" for `LokiPushApiConsumer` and "log-proxy" for
+  `LogProxyConsumer` (note that LogProxyConsumer is deprecated).
+
+  For example, a provider's `metadata.yaml` file may look as follows:
+
+  ```yaml
+  provides:
+    logging:
+      interface: loki_push_api
+  ```
+
+  Subsequently, a Loki charm may instantiate the `LokiPushApiProvider` in its constructor as
+  follows:
+
+      from charms.loki_k8s.v0.loki_push_api import LokiPushApiProvider
+      from loki_server import LokiServer
+      ...
+
+      class LokiOperatorCharm(CharmBase):
+          ...
+
+          def __init__(self, *args):
+              super().__init__(*args)
+              ...
+              external_url = urlparse(self._external_url)
+              self.loki_provider = LokiPushApiProvider(
+                  self,
+                  address=external_url.hostname or self.hostname,
+                  port=external_url.port or 80,
+                  scheme=external_url.scheme,
+                  path=f"{external_url.path}/loki/api/v1/push",
+              )
+              ...
+
+  - `port`: Loki Push Api endpoint port. Default value: `3100`.
+  - `scheme`: Loki Push Api endpoint scheme (`HTTP` or `HTTPS`). Default value: `HTTP`
+  - `address`: Loki Push Api endpoint address. Default value: `localhost`
+  - `path`: Loki Push Api endpoint path. Default value: `loki/api/v1/push`
+
+
+The `LokiPushApiProvider` object has several responsibilities:
+
+1. Set the URL of the Loki Push API in the relation application data bag; the URL
+   must be unique to all instances (e.g. using a load balancer).
+
+2. Set the Promtail binary URL (`promtail_binary_zip_url`) so clients that use
+   `LogProxyConsumer` object could download and configure it.
+
+3. Process the metadata of the consumer application, provided via the
+   "metadata" field of the consumer data bag, which are used to annotate the
+   alert rules (see next point). An example for "metadata" is the following:
+
+    {'model': 'loki',
+     'model_uuid': '0b7d1071-ded2-4bf5-80a3-10a81aeb1386',
+     'application': 'promtail-k8s'
+    }
+
+4. Process alert rules set into the relation by the `LokiPushApiConsumer`
+   objects, e.g.:
+
+    '{
+         "groups": [{
+             "name": "loki_0b7d1071-ded2-4bf5-80a3-10a81aeb1386_promtail-k8s_alerts",
+             "rules": [{
+                 "alert": "HighPercentageError",
+                 "expr": "sum(rate({app=\\"foo\\", env=\\"production\\"} |= \\"error\\" [5m]))
+                          by (job) \\n  /\\nsum(rate({app=\\"foo\\", env=\\"production\\"}[5m]))
+                          by (job)\\n  > 0.05
+                          \\n", "for": "10m",
+                 "labels": {
+                     "severity": "page",
+                     "juju_model": "loki",
+                     "juju_model_uuid": "0b7d1071-ded2-4bf5-80a3-10a81aeb1386",
+                     "juju_application": "promtail-k8s"
+                },
+                "annotations": {
+                    "summary": "High request latency"
+               }
+             }]
+         }]
+     }'
+
+
+Once these alert rules are sent over relation data, the `LokiPushApiProvider` object
+stores these files in the directory `/loki/rules` inside the Loki charm container. After
+storing alert rules files, the object will check alert rules by querying Loki API
+endpoint: [`loki/api/v1/rules`](https://grafana.com/docs/loki/latest/api/#list-rule-groups).
+If there are changes in the alert rules a `loki_push_api_alert_rules_changed` event will
+be emitted with details about the `RelationEvent` which triggered it.
+
+This events should be observed in the charm that uses `LokiPushApiProvider`:
+
+```python
+    def __init__(self, *args):
+        super().__init__(*args)
+        ...
+        self.loki_provider = LokiPushApiProvider(self)
+        self.framework.observe(
+            self.loki_provider.on.loki_push_api_alert_rules_changed,
+            self._loki_push_api_alert_rules_changed,
+        )
+```
+
+
+## LokiPushApiConsumer Library Usage
+
+This Loki charm interacts with its clients using the Loki charm library. Charms
+seeking to send log to Loki, must do so using the `LokiPushApiConsumer` object from
+this charm library.
+
+> **NOTE**: `LokiPushApiConsumer` also depends on an additional charm library.
+>
+> Ensure sure you `charmcraft fetch-lib charms.observability_libs.v0.juju_topology`
+> when using this library.
+
+For the simplest use cases, using the `LokiPushApiConsumer` object only requires
+instantiating it, typically in the constructor of your charm (the one which
+sends logs).
+
+```python
+from charms.loki_k8s.v0.loki_push_api import LokiPushApiConsumer
+
+class LokiClientCharm(CharmBase):
+
+    def __init__(self, *args):
+        super().__init__(*args)
+        ...
+        self._loki_consumer = LokiPushApiConsumer(self)
+```
+
+The `LokiPushApiConsumer` constructor requires two things:
+
+- A reference to the parent (LokiClientCharm) charm.
+
+- Optionally, the name of the relation that the Loki charm uses to interact
+  with its clients. If provided, this relation name must match a required
+  relation in metadata.yaml with the `loki_push_api` interface.
+
+  This argument is not required if your metadata.yaml has precisely one
+  required relation in metadata.yaml with the `loki_push_api` interface, as the
+  lib will automatically resolve the relation name inspecting the using the
+  meta information of the charm
+
+Any time the relation between a Loki provider charm and a Loki consumer charm is
+established, a `LokiPushApiEndpointJoined` event is fired. In the consumer side
+is it possible to observe this event with:
+
+```python
+
+self.framework.observe(
+    self._loki_consumer.on.loki_push_api_endpoint_joined,
+    self._on_loki_push_api_endpoint_joined,
+)
+```
+
+Any time there are departures in relations between the consumer charm and Loki
+the consumer charm is informed, through a `LokiPushApiEndpointDeparted` event, for instance:
+
+```python
+self.framework.observe(
+    self._loki_consumer.on.loki_push_api_endpoint_departed,
+    self._on_loki_push_api_endpoint_departed,
+)
+```
+
+The consumer charm can then choose to update its configuration in both situations.
+
+Note that LokiPushApiConsumer does not add any labels automatically on its own. In
+order to better integrate with the Canonical Observability Stack, you may want to configure your
+software to add Juju topology labels. The
+[observability-libs](https://charmhub.io/observability-libs) library can be used to get topology
+labels in charm code. See :func:`LogProxyConsumer._scrape_configs` for an example of how
+to do this with promtail.
+
+## LogProxyConsumer Library Usage
+
+> Note: This object is deprecated. Consider migrating to LogForwarder (see v1/loki_push_api) with
+> the release of Juju 3.6 LTS.
+
+Let's say that we have a workload charm that produces logs, and we need to send those logs to a
+workload implementing the `loki_push_api` interface, such as `Loki` or `Grafana Agent`.
+
+Adopting this object in a Charmed Operator consist of two steps:
+
+1. Use the `LogProxyConsumer` class by instantiating it in the `__init__` method of the charmed
+   operator. There are two ways to get logs in to promtail. You can give it a list of files to
+   read, or you can write to it using the syslog protocol.
+
+   For example:
+
+   ```python
+   from charms.loki_k8s.v0.loki_push_api import LogProxyConsumer
+
+   ...
+
+       def __init__(self, *args):
+           ...
+           self._log_proxy = LogProxyConsumer(
+               charm=self, log_files=LOG_FILES, container_name=PEER, enable_syslog=True
+           )
+
+           self.framework.observe(
+               self._log_proxy.on.promtail_digest_error,
+               self._promtail_error,
+           )
+
+       def _promtail_error(self, event):
+           logger.error(event.message)
+           self.unit.status = BlockedStatus(event.message)
+   ```
+
+   Any time the relation between a provider charm and a LogProxy consumer charm is
+   established, a `LogProxyEndpointJoined` event is fired. In the consumer side is it
+   possible to observe this event with:
+
+   ```python
+
+   self.framework.observe(
+       self._log_proxy.on.log_proxy_endpoint_joined,
+       self._on_log_proxy_endpoint_joined,
+   )
+   ```
+
+   Any time there are departures in relations between the consumer charm and the provider
+   the consumer charm is informed, through a `LogProxyEndpointDeparted` event, for instance:
+
+   ```python
+   self.framework.observe(
+       self._log_proxy.on.log_proxy_endpoint_departed,
+       self._on_log_proxy_endpoint_departed,
+   )
+   ```
+
+   The consumer charm can then choose to update its configuration in both situations.
+
+   Note that:
+
+   - `LOG_FILES` is a `list` containing the log files we want to send to `Loki` or
+   `Grafana Agent`, for instance:
+
+   ```python
+   LOG_FILES = [
+       "/var/log/apache2/access.log",
+       "/var/log/alternatives.log",
+   ]
+   ```
+
+   - `container_name` is the name of the container in which the application is running.
+      If in the Pod there is only one container, this argument can be omitted.
+
+   - You can configure your syslog software using `localhost` as the address and the method
+     `LogProxyConsumer.syslog_port` to get the port, or, alternatively, if you are using rsyslog
+     you may use the method `LogProxyConsumer.rsyslog_config()`.
+
+2. Modify the `metadata.yaml` file to add:
+
+   - The `log-proxy` relation in the `requires` section:
+     ```yaml
+     requires:
+       log-proxy:
+         interface: loki_push_api
+         optional: true
+     ```
+
+Once the library is implemented in a Charmed Operator and a relation is established with
+the charm that implements the `loki_push_api` interface, the library will inject a
+Pebble layer that runs Promtail in the workload container to send logs.
+
+By default, the promtail binary injected into the container will be downloaded from the internet.
+If, for any reason, the container has limited network access, you may allow charm administrators
+to provide their own promtail binary at runtime by adding the following snippet to your charm
+metadata:
+
+```yaml
+resources:
+  promtail-bin:
+      type: file
+      description: Promtail binary for logging
+      filename: promtail-linux
+```
+
+Which would then allow operators to deploy the charm this way:
+
+```
+juju deploy \
+    ./your_charm.charm \
+    --resource promtail-bin=/tmp/promtail-linux-amd64
+```
+
+If a different resource name is used, it can be specified with the `promtail_resource_name`
+argument to the `LogProxyConsumer` constructor.
+
+The object can emit a `PromtailDigestError` event:
+
+- Promtail binary cannot be downloaded.
+- The sha256 sum mismatch for promtail binary.
+
+The object can raise a `ContainerNotFoundError` event:
+
+- No `container_name` parameter has been specified and the Pod has more than 1 container.
+
+These can be monitored via the PromtailDigestError events via:
+
+```python
+   self.framework.observe(
+       self._loki_consumer.on.promtail_digest_error,
+       self._promtail_error,
+   )
+
+   def _promtail_error(self, event):
+       logger.error(msg)
+       self.unit.status = BlockedStatus(event.message)
+    )
+```
+
+## Alerting Rules
+
+This charm library also supports gathering alerting rules from all related Loki client
+charms and enabling corresponding alerts within the Loki charm. Alert rules are
+automatically gathered by `LokiPushApiConsumer` object from a directory conventionally
+named `loki_alert_rules`.
+
+This directory must reside at the top level in the `src` folder of the
+consumer charm. Each file in this directory is assumed to be a single alert rule
+in YAML format. The file name must have the `.rule` extension.
+The format of this alert rule conforms to the
+[Loki docs](https://grafana.com/docs/loki/latest/rules/#alerting-rules).
+
+An example of the contents of one such file is shown below.
+
+```yaml
+alert: HighPercentageError
+expr: |
+  sum(rate({%%juju_topology%%} |= "error" [5m])) by (job)
+    /
+  sum(rate({%%juju_topology%%}[5m])) by (job)
+    > 0.05
+for: 10m
+labels:
+    severity: page
+annotations:
+    summary: High request latency
+
+```
+
+It is **critical** to use the `%%juju_topology%%` filter in the expression for the alert
+rule shown above. This filter is a stub that is automatically replaced by the
+`LokiPushApiConsumer` following Loki Client's Juju topology (application, model and its
+UUID). Such a topology filter is essential to ensure that alert rules submitted by one
+provider charm generates alerts only for that same charm.
+
+The Loki charm may be related to multiple Loki client charms. Without this, filter
+rules submitted by one provider charm will also result in corresponding alerts for other
+provider charms. Hence, every alert rule expression must include such a topology filter stub.
+
+Gathering alert rules and generating rule files within the Loki charm is easily done using
+the `alerts()` method of `LokiPushApiProvider`. Alerts generated by Loki will automatically
+include Juju topology labels in the alerts. These labels indicate the source of the alert.
+
+The following labels are automatically added to every alert
+
+- `juju_model`
+- `juju_model_uuid`
+- `juju_application`
+
+
+Whether alert rules files does not contain the keys `alert` or `expr` or there is no alert
+rules file in `alert_rules_path` a `loki_push_api_alert_rules_error` event is emitted.
+
+To handle these situations the event must be observed in the `LokiClientCharm` charm.py file:
+
+```python
+class LokiClientCharm(CharmBase):
+
+    def __init__(self, *args):
+        super().__init__(*args)
+        ...
+        self._loki_consumer = LokiPushApiConsumer(self)
+
+        self.framework.observe(
+            self._loki_consumer.on.loki_push_api_alert_rules_error,
+            self._alert_rules_error
+        )
+
+    def _alert_rules_error(self, event):
+        self.unit.status = BlockedStatus(event.message)
+```
+
+## Relation Data
+
+The Loki charm uses both application and unit relation data to obtain information regarding
+Loki Push API and alert rules.
+
+Units of consumer charm send their alert rules over app relation data using the `alert_rules`
+key.
+"""
+
+import json
+import logging
+import os
+import platform
+import re
+import socket
+import subprocess
+import tempfile
+import typing
+from copy import deepcopy
+from gzip import GzipFile
+from hashlib import sha256
+from io import BytesIO
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple, Union, cast
+from urllib import request
+from urllib.error import HTTPError
+
+import yaml
+from cosl import JujuTopology
+from ops.charm import (
+    CharmBase,
+    HookEvent,
+    RelationBrokenEvent,
+    RelationCreatedEvent,
+    RelationDepartedEvent,
+    RelationEvent,
+    RelationJoinedEvent,
+    RelationRole,
+    WorkloadEvent,
+)
+from ops.framework import EventBase, EventSource, Object, ObjectEvents
+from ops.model import Container, ModelError, Relation
+from ops.pebble import APIError, ChangeError, PathError, ProtocolError
+
+# The unique Charmhub library identifier, never change it
+LIBID = "bf76f23cdd03464b877c52bd1d2f563e"
+
+# Increment this major API version when introducing breaking changes
+LIBAPI = 0
+
+# Increment this PATCH version before using `charmcraft publish-lib` or reset
+# to 0 if you are raising the major API version
+LIBPATCH = 30
+
+PYDEPS = ["cosl"]
+
+logger = logging.getLogger(__name__)
+
+RELATION_INTERFACE_NAME = "loki_push_api"
+DEFAULT_RELATION_NAME = "logging"
+DEFAULT_ALERT_RULES_RELATIVE_PATH = "./src/loki_alert_rules"
+DEFAULT_LOG_PROXY_RELATION_NAME = "log-proxy"
+
+PROMTAIL_BASE_URL = "https://github.com/canonical/loki-k8s-operator/releases/download"
+# To update Promtail version you only need to change the PROMTAIL_VERSION and
+# update all sha256 sums in PROMTAIL_BINARIES. To support a new architecture
+# you only need to add a new key value pair for the architecture in PROMTAIL_BINARIES.
+PROMTAIL_VERSION = "v2.5.0"
+PROMTAIL_BINARIES = {
+    "amd64": {
+        "filename": "promtail-static-amd64",
+        "zipsha": "543e333b0184e14015a42c3c9e9e66d2464aaa66eca48b29e185a6a18f67ab6d",
+        "binsha": "17e2e271e65f793a9fbe81eab887b941e9d680abe82d5a0602888c50f5e0cac9",
+    },
+}
+
+# Paths in `charm` container
+BINARY_DIR = "/tmp"
+
+# Paths in `workload` container
+WORKLOAD_BINARY_DIR = "/opt/promtail"
+WORKLOAD_CONFIG_DIR = "/etc/promtail"
+WORKLOAD_CONFIG_FILE_NAME = "promtail_config.yaml"
+WORKLOAD_CONFIG_PATH = "{}/{}".format(WORKLOAD_CONFIG_DIR, WORKLOAD_CONFIG_FILE_NAME)
+WORKLOAD_POSITIONS_PATH = "{}/positions.yaml".format(WORKLOAD_BINARY_DIR)
+WORKLOAD_SERVICE_NAME = "promtail"
+
+HTTP_LISTEN_PORT = 9080
+GRPC_LISTEN_PORT = 9095
+
+
+class RelationNotFoundError(ValueError):
+    """Raised if there is no relation with the given name."""
+
+    def __init__(self, relation_name: str):
+        self.relation_name = relation_name
+        self.message = "No relation named '{}' found".format(relation_name)
+
+        super().__init__(self.message)
+
+
+class RelationInterfaceMismatchError(Exception):
+    """Raised if the relation with the given name has a different interface."""
+
+    def __init__(
+        self,
+        relation_name: str,
+        expected_relation_interface: str,
+        actual_relation_interface: str,
+    ):
+        self.relation_name = relation_name
+        self.expected_relation_interface = expected_relation_interface
+        self.actual_relation_interface = actual_relation_interface
+        self.message = (
+            "The '{}' relation has '{}' as interface rather than the expected '{}'".format(
+                relation_name, actual_relation_interface, expected_relation_interface
+            )
+        )
+        super().__init__(self.message)
+
+
+class RelationRoleMismatchError(Exception):
+    """Raised if the relation with the given name has a different direction."""
+
+    def __init__(
+        self,
+        relation_name: str,
+        expected_relation_role: RelationRole,
+        actual_relation_role: RelationRole,
+    ):
+        self.relation_name = relation_name
+        self.expected_relation_interface = expected_relation_role
+        self.actual_relation_role = actual_relation_role
+        self.message = "The '{}' relation has role '{}' rather than the expected '{}'".format(
+            relation_name, repr(actual_relation_role), repr(expected_relation_role)
+        )
+        super().__init__(self.message)
+
+
+def _validate_relation_by_interface_and_direction(
+    charm: CharmBase,
+    relation_name: str,
+    expected_relation_interface: str,
+    expected_relation_role: RelationRole,
+):
+    """Verifies that a relation has the necessary characteristics.
+
+    Verifies that the `relation_name` provided: (1) exists in metadata.yaml,
+    (2) declares as interface the interface name passed as `relation_interface`
+    and (3) has the right "direction", i.e., it is a relation that `charm`
+    provides or requires.
+
+    Args:
+        charm: a `CharmBase` object to scan for the matching relation.
+        relation_name: the name of the relation to be verified.
+        expected_relation_interface: the interface name to be matched by the
+            relation named `relation_name`.
+        expected_relation_role: whether the `relation_name` must be either
+            provided or required by `charm`.
+
+    Raises:
+        RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+            with the same name as provided via `relation_name` argument.
+        RelationInterfaceMismatchError: The relation with the same name as provided
+            via `relation_name` argument does not have the same relation interface
+            as specified via the `expected_relation_interface` argument.
+        RelationRoleMismatchError: If the relation with the same name as provided
+            via `relation_name` argument does not have the same role as specified
+            via the `expected_relation_role` argument.
+    """
+    if relation_name not in charm.meta.relations:
+        raise RelationNotFoundError(relation_name)
+
+    relation = charm.meta.relations[relation_name]
+
+    actual_relation_interface = relation.interface_name
+    if actual_relation_interface != expected_relation_interface:
+        raise RelationInterfaceMismatchError(
+            relation_name,
+            expected_relation_interface,
+            actual_relation_interface,  # pyright: ignore
+        )
+
+    if expected_relation_role == RelationRole.provides:
+        if relation_name not in charm.meta.provides:
+            raise RelationRoleMismatchError(
+                relation_name, RelationRole.provides, RelationRole.requires
+            )
+    elif expected_relation_role == RelationRole.requires:
+        if relation_name not in charm.meta.requires:
+            raise RelationRoleMismatchError(
+                relation_name, RelationRole.requires, RelationRole.provides
+            )
+    else:
+        raise Exception("Unexpected RelationDirection: {}".format(expected_relation_role))
+
+
+class InvalidAlertRulePathError(Exception):
+    """Raised if the alert rules folder cannot be found or is otherwise invalid."""
+
+    def __init__(
+        self,
+        alert_rules_absolute_path: Path,
+        message: str,
+    ):
+        self.alert_rules_absolute_path = alert_rules_absolute_path
+        self.message = message
+
+        super().__init__(self.message)
+
+
+def _is_official_alert_rule_format(rules_dict: dict) -> bool:
+    """Are alert rules in the upstream format as supported by Loki.
+
+    Alert rules in dictionary format are in "official" form if they
+    contain a "groups" key, since this implies they contain a list of
+    alert rule groups.
+
+    Args:
+        rules_dict: a set of alert rules in Python dictionary format
+
+    Returns:
+        True if alert rules are in official Loki file format.
+    """
+    return "groups" in rules_dict
+
+
+def _is_single_alert_rule_format(rules_dict: dict) -> bool:
+    """Are alert rules in single rule format.
+
+    The Loki charm library supports reading of alert rules in a
+    custom format that consists of a single alert rule per file. This
+    does not conform to the official Loki alert rule file format
+    which requires that each alert rules file consists of a list of
+    alert rule groups and each group consists of a list of alert
+    rules.
+
+    Alert rules in dictionary form are considered to be in single rule
+    format if in the least it contains two keys corresponding to the
+    alert rule name and alert expression.
+
+    Returns:
+        True if alert rule is in single rule file format.
+    """
+    # one alert rule per file
+    return set(rules_dict) >= {"alert", "expr"}
+
+
+class AlertRules:
+    """Utility class for amalgamating Loki alert rule files and injecting juju topology.
+
+    An `AlertRules` object supports aggregating alert rules from files and directories in both
+    official and single rule file formats using the `add_path()` method. All the alert rules
+    read are annotated with Juju topology labels and amalgamated into a single data structure
+    in the form of a Python dictionary using the `as_dict()` method. Such a dictionary can be
+    easily dumped into JSON format and exchanged over relation data. The dictionary can also
+    be dumped into YAML format and written directly into an alert rules file that is read by
+    Loki. Note that multiple `AlertRules` objects must not be written into the same file,
+    since Loki allows only a single list of alert rule groups per alert rules file.
+
+    The official Loki format is a YAML file conforming to the Loki documentation
+    (https://grafana.com/docs/loki/latest/api/#list-rule-groups).
+    The custom single rule format is a subsection of the official YAML, having a single alert
+    rule, effectively "one alert per file".
+    """
+
+    # This class uses the following terminology for the various parts of a rule file:
+    # - alert rules file: the entire groups[] yaml, including the "groups:" key.
+    # - alert groups (plural): the list of groups[] (a list, i.e. no "groups:" key) - it is a list
+    #   of dictionaries that have the "name" and "rules" keys.
+    # - alert group (singular): a single dictionary that has the "name" and "rules" keys.
+    # - alert rules (plural): all the alerts in a given alert group - a list of dictionaries with
+    #   the "alert" and "expr" keys.
+    # - alert rule (singular): a single dictionary that has the "alert" and "expr" keys.
+
+    def __init__(self, topology: Optional[JujuTopology] = None):
+        """Build and alert rule object.
+
+        Args:
+            topology: a `JujuTopology` instance that is used to annotate all alert rules.
+        """
+        self.topology = topology
+        self.tool = CosTool(None)
+        self.alert_groups = []  # type: List[dict]
+
+    def _from_file(self, root_path: Path, file_path: Path) -> List[dict]:
+        """Read a rules file from path, injecting juju topology.
+
+        Args:
+            root_path: full path to the root rules folder (used only for generating group name)
+            file_path: full path to a *.rule file.
+
+        Returns:
+            A list of dictionaries representing the rules file, if file is valid (the structure is
+            formed by `yaml.safe_load` of the file); an empty list otherwise.
+        """
+        with file_path.open() as rf:
+            # Load a list of rules from file then add labels and filters
+            try:
+                rule_file = yaml.safe_load(rf) or {}
+
+            except Exception as e:
+                logger.error("Failed to read alert rules from %s: %s", file_path.name, e)
+                return []
+
+            if _is_official_alert_rule_format(rule_file):
+                alert_groups = rule_file["groups"]
+            elif _is_single_alert_rule_format(rule_file):
+                # convert to list of alert groups
+                # group name is made up from the file name
+                alert_groups = [{"name": file_path.stem, "rules": [rule_file]}]
+            else:
+                # invalid/unsupported
+                reason = "file is empty" if not rule_file else "unexpected file structure"
+                logger.error("Invalid rules file (%s): %s", reason, file_path.name)
+                return []
+
+            # update rules with additional metadata
+            for alert_group in alert_groups:
+                # update group name with topology and sub-path
+                alert_group["name"] = self._group_name(
+                    str(root_path),
+                    str(file_path),
+                    alert_group["name"],
+                )
+
+                # add "juju_" topology labels
+                for alert_rule in alert_group["rules"]:
+                    if "labels" not in alert_rule:
+                        alert_rule["labels"] = {}
+
+                    if self.topology:
+                        alert_rule["labels"].update(self.topology.label_matcher_dict)
+                        # insert juju topology filters into a prometheus alert rule
+                        # logql doesn't like empty matchers, so add a job matcher which hits
+                        # any string as a "wildcard" which the topology labels will
+                        # filter down
+                        alert_rule["expr"] = self.tool.inject_label_matchers(
+                            re.sub(r"%%juju_topology%%", r'job=~".+"', alert_rule["expr"]),
+                            self.topology.label_matcher_dict,
+                        )
+
+            return alert_groups
+
+    def _group_name(
+        self,
+        root_path: typing.Union[Path, str],
+        file_path: typing.Union[Path, str],
+        group_name: str,
+    ) -> str:
+        """Generate group name from path and topology.
+
+        The group name is made up of the relative path between the root dir_path, the file path,
+        and topology identifier.
+
+        Args:
+            root_path: path to the root rules dir.
+            file_path: path to rule file.
+            group_name: original group name to keep as part of the new augmented group name
+
+        Returns:
+            New group name, augmented by juju topology and relative path.
+        """
+        file_path = Path(file_path) if not isinstance(file_path, Path) else file_path
+        root_path = Path(root_path) if not isinstance(root_path, Path) else root_path
+        rel_path = file_path.parent.relative_to(root_path.as_posix())
+
+        # We should account for both absolute paths and Windows paths. Convert it to a POSIX
+        # string, strip off any leading /, then join it
+
+        path_str = ""
+        if not rel_path == Path("."):
+            # Get rid of leading / and optionally drive letters so they don't muck up
+            # the template later, since Path.parts returns them. The 'if relpath.is_absolute ...'
+            # isn't even needed since re.sub doesn't throw exceptions if it doesn't match, so it's
+            # optional, but it makes it clear what we're doing.
+
+            # Note that Path doesn't actually care whether the path is valid just to instantiate
+            # the object, so we can happily strip that stuff out to make templating nicer
+            rel_path = Path(
+                re.sub(r"^([A-Za-z]+:)?/", "", rel_path.as_posix())
+                if rel_path.is_absolute()
+                else str(rel_path)
+            )
+
+            # Get rid of relative path characters in the middle which both os.path and pathlib
+            # leave hanging around. We could use path.resolve(), but that would lead to very
+            # long template strings when rules come from pods and/or other deeply nested charm
+            # paths
+            path_str = "_".join(filter(lambda x: x not in ["..", "/"], rel_path.parts))
+
+        # Generate group name:
+        #  - name, from juju topology
+        #  - suffix, from the relative path of the rule file;
+        group_name_parts = [self.topology.identifier] if self.topology else []
+        group_name_parts.extend([path_str, group_name, "alerts"])
+        # filter to remove empty strings
+        return "_".join(filter(lambda x: x, group_name_parts))
+
+    @classmethod
+    def _multi_suffix_glob(
+        cls, dir_path: Path, suffixes: List[str], recursive: bool = True
+    ) -> list:
+        """Helper function for getting all files in a directory that have a matching suffix.
+
+        Args:
+            dir_path: path to the directory to glob from.
+            suffixes: list of suffixes to include in the glob (items should begin with a period).
+            recursive: a flag indicating whether a glob is recursive (nested) or not.
+
+        Returns:
+            List of files in `dir_path` that have one of the suffixes specified in `suffixes`.
+        """
+        all_files_in_dir = dir_path.glob("**/*" if recursive else "*")
+        return list(filter(lambda f: f.is_file() and f.suffix in suffixes, all_files_in_dir))
+
+    def _from_dir(self, dir_path: Path, recursive: bool) -> List[dict]:
+        """Read all rule files in a directory.
+
+        All rules from files for the same directory are loaded into a single
+        group. The generated name of this group includes juju topology.
+        By default, only the top directory is scanned; for nested scanning, pass `recursive=True`.
+
+        Args:
+            dir_path: directory containing *.rule files (alert rules without groups).
+            recursive: flag indicating whether to scan for rule files recursively.
+
+        Returns:
+            a list of dictionaries representing prometheus alert rule groups, each dictionary
+            representing an alert group (structure determined by `yaml.safe_load`).
+        """
+        alert_groups = []  # type: List[dict]
+
+        # Gather all alerts into a list of groups
+        for file_path in self._multi_suffix_glob(dir_path, [".rule", ".rules"], recursive):
+            alert_groups_from_file = self._from_file(dir_path, file_path)
+            if alert_groups_from_file:
+                logger.debug("Reading alert rule from %s", file_path)
+                alert_groups.extend(alert_groups_from_file)
+
+        return alert_groups
+
+    def add_path(self, path_str: str, *, recursive: bool = False):
+        """Add rules from a dir path.
+
+        All rules from files are aggregated into a data structure representing a single rule file.
+        All group names are augmented with juju topology.
+
+        Args:
+            path_str: either a rules file or a dir of rules files.
+            recursive: whether to read files recursively or not (no impact if `path` is a file).
+
+        Raises:
+            InvalidAlertRulePathError: if the provided path is invalid.
+        """
+        path = Path(path_str)  # type: Path
+        if path.is_dir():
+            self.alert_groups.extend(self._from_dir(path, recursive))
+        elif path.is_file():
+            self.alert_groups.extend(self._from_file(path.parent, path))
+        else:
+            logger.debug("The alerts file does not exist: %s", path)
+
+    def as_dict(self) -> dict:
+        """Return standard alert rules file in dict representation.
+
+        Returns:
+            a dictionary containing a single list of alert rule groups.
+            The list of alert rule groups is provided as value of the
+            "groups" dictionary key.
+        """
+        return {"groups": self.alert_groups} if self.alert_groups else {}
+
+
+def _resolve_dir_against_charm_path(charm: CharmBase, *path_elements: str) -> str:
+    """Resolve the provided path items against the directory of the main file.
+
+    Look up the directory of the `main.py` file being executed. This is normally
+    going to be the charm.py file of the charm including this library. Then, resolve
+    the provided path elements and, if the result path exists and is a directory,
+    return its absolute path; otherwise, raise en exception.
+
+    Raises:
+        InvalidAlertRulePathError, if the path does not exist or is not a directory.
+    """
+    charm_dir = Path(str(charm.charm_dir))
+    if not charm_dir.exists() or not charm_dir.is_dir():
+        # Operator Framework does not currently expose a robust
+        # way to determine the top level charm source directory
+        # that is consistent across deployed charms and unit tests
+        # Hence for unit tests the current working directory is used
+        # TODO: updated this logic when the following ticket is resolved
+        # https://github.com/canonical/operator/issues/643
+        charm_dir = Path(os.getcwd())
+
+    alerts_dir_path = charm_dir.absolute().joinpath(*path_elements)
+
+    if not alerts_dir_path.exists():
+        raise InvalidAlertRulePathError(alerts_dir_path, "directory does not exist")
+    if not alerts_dir_path.is_dir():
+        raise InvalidAlertRulePathError(alerts_dir_path, "is not a directory")
+
+    return str(alerts_dir_path)
+
+
+class NoRelationWithInterfaceFoundError(Exception):
+    """No relations with the given interface are found in the charm meta."""
+
+    def __init__(self, charm: CharmBase, relation_interface: Optional[str] = None):
+        self.charm = charm
+        self.relation_interface = relation_interface
+        self.message = (
+            "No relations with interface '{}' found in the meta of the '{}' charm".format(
+                relation_interface, charm.meta.name
+            )
+        )
+
+        super().__init__(self.message)
+
+
+class MultipleRelationsWithInterfaceFoundError(Exception):
+    """Multiple relations with the given interface are found in the charm meta."""
+
+    def __init__(self, charm: CharmBase, relation_interface: str, relations: list):
+        self.charm = charm
+        self.relation_interface = relation_interface
+        self.relations = relations
+        self.message = (
+            "Multiple relations with interface '{}' found in the meta of the '{}' charm.".format(
+                relation_interface, charm.meta.name
+            )
+        )
+        super().__init__(self.message)
+
+
+class LokiPushApiEndpointDeparted(EventBase):
+    """Event emitted when Loki departed."""
+
+
+class LokiPushApiEndpointJoined(EventBase):
+    """Event emitted when Loki joined."""
+
+
+class LokiPushApiAlertRulesChanged(EventBase):
+    """Event emitted if there is a change in the alert rules."""
+
+    def __init__(self, handle, relation, relation_id, app=None, unit=None):
+        """Pretend we are almost like a RelationEvent.
+
+        Fields to serialize:
+            {
+                "relation_name": <a relation name as a string>,
+                "relation_id": <a relation id, optional>,
+                "app_name": <app name as a string>,
+                "unit_name": <unit name as a string>
+            }
+
+        In this way, we can transparently use `RelationEvent.snapshot()` to pass
+        it back if we need to log it.
+        """
+        super().__init__(handle)
+        self.relation = relation
+        self.relation_id = relation_id
+        self.app = app
+        self.unit = unit
+
+    def snapshot(self) -> Dict:
+        """Save event information."""
+        if not self.relation:
+            return {}
+        snapshot = {"relation_name": self.relation.name, "relation_id": self.relation.id}
+        if self.app:
+            snapshot["app_name"] = self.app.name
+        if self.unit:
+            snapshot["unit_name"] = self.unit.name
+        return snapshot
+
+    def restore(self, snapshot: dict):
+        """Restore event information."""
+        self.relation = self.framework.model.get_relation(
+            snapshot["relation_name"], snapshot["relation_id"]
+        )
+        app_name = snapshot.get("app_name")
+        if app_name:
+            self.app = self.framework.model.get_app(app_name)
+        else:
+            self.app = None
+        unit_name = snapshot.get("unit_name")
+        if unit_name:
+            self.unit = self.framework.model.get_unit(unit_name)
+        else:
+            self.unit = None
+
+
+class InvalidAlertRuleEvent(EventBase):
+    """Event emitted when alert rule files are not parsable.
+
+    Enables us to set a clear status on the provider.
+    """
+
+    def __init__(self, handle, errors: str = "", valid: bool = False):
+        super().__init__(handle)
+        self.errors = errors
+        self.valid = valid
+
+    def snapshot(self) -> Dict:
+        """Save alert rule information."""
+        return {
+            "valid": self.valid,
+            "errors": self.errors,
+        }
+
+    def restore(self, snapshot):
+        """Restore alert rule information."""
+        self.valid = snapshot["valid"]
+        self.errors = snapshot["errors"]
+
+
+class LokiPushApiEvents(ObjectEvents):
+    """Event descriptor for events raised by `LokiPushApiProvider`."""
+
+    loki_push_api_endpoint_departed = EventSource(LokiPushApiEndpointDeparted)
+    loki_push_api_endpoint_joined = EventSource(LokiPushApiEndpointJoined)
+    loki_push_api_alert_rules_changed = EventSource(LokiPushApiAlertRulesChanged)
+    alert_rule_status_changed = EventSource(InvalidAlertRuleEvent)
+
+
+class LokiPushApiProvider(Object):
+    """A LokiPushApiProvider class."""
+
+    on = LokiPushApiEvents()  # pyright: ignore
+
+    def __init__(
+        self,
+        charm,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        *,
+        port: Union[str, int] = 3100,
+        scheme: str = "http",
+        address: str = "localhost",
+        path: str = "loki/api/v1/push",
+    ):
+        """A Loki service provider.
+
+        Args:
+            charm: a `CharmBase` instance that manages this
+                instance of the Loki service.
+            relation_name: an optional string name of the relation between `charm`
+                and the Loki charmed service. The default is "logging".
+                It is strongly advised not to change the default, so that people
+                deploying your charm will have a consistent experience with all
+                other charms that consume metrics endpoints.
+            port: an optional port of the Loki service (default is "3100").
+            scheme: an optional scheme of the Loki API URL (default is "http").
+            address: an optional address of the Loki service (default is "localhost").
+            path: an optional path of the Loki API URL (default is "loki/api/v1/push")
+
+        Raises:
+            RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+                with the same name as provided via `relation_name` argument.
+            RelationInterfaceMismatchError: The relation with the same name as provided
+                via `relation_name` argument does not have the `loki_push_api` relation
+                interface.
+            RelationRoleMismatchError: If the relation with the same name as provided
+                via `relation_name` argument does not have the `RelationRole.requires`
+                role.
+        """
+        _validate_relation_by_interface_and_direction(
+            charm, relation_name, RELATION_INTERFACE_NAME, RelationRole.provides
+        )
+        super().__init__(charm, relation_name)
+        self._charm = charm
+        self._relation_name = relation_name
+        self._tool = CosTool(self)
+        self.port = int(port)
+        self.scheme = scheme
+        self.address = address
+        self.path = path
+
+        events = self._charm.on[relation_name]
+        self.framework.observe(self._charm.on.upgrade_charm, self._on_lifecycle_event)
+        self.framework.observe(events.relation_joined, self._on_logging_relation_joined)
+        self.framework.observe(events.relation_changed, self._on_logging_relation_changed)
+        self.framework.observe(events.relation_departed, self._on_logging_relation_departed)
+        self.framework.observe(events.relation_broken, self._on_logging_relation_broken)
+
+    def _on_lifecycle_event(self, _):
+        # Upgrade event or other charm-level event
+        should_update = False
+        for relation in self._charm.model.relations[self._relation_name]:
+            # Don't accidentally flip a True result back.
+            should_update = should_update or self._process_logging_relation_changed(relation)
+        if should_update:
+            # We don't have a RelationEvent, so build it up by hand
+            first_rel = self._charm.model.relations[self._relation_name][0]
+            self.on.loki_push_api_alert_rules_changed.emit(
+                relation=first_rel,
+                relation_id=first_rel.id,
+            )
+
+    def _on_logging_relation_joined(self, event: RelationJoinedEvent):
+        """Set basic data on relation joins.
+
+        Set the promtail binary URL location, which will not change, and anything
+        else which may be required, but is static..
+
+        Args:
+            event: a `CharmEvent` in response to which the consumer
+                charm must set its relation data.
+        """
+        if self._charm.unit.is_leader():
+            event.relation.data[self._charm.app].update(self._promtail_binary_url)
+            logger.debug("Saved promtail binary url: %s", self._promtail_binary_url)
+
+    def _on_logging_relation_changed(self, event: HookEvent):
+        """Handle changes in related consumers.
+
+        Anytime there are changes in the relation between Loki
+        and its consumers charms.
+
+        Args:
+            event: a `CharmEvent` in response to which the consumer
+                charm must update its relation data.
+        """
+        should_update = self._process_logging_relation_changed(event.relation)  # pyright: ignore
+        if should_update:
+            self.on.loki_push_api_alert_rules_changed.emit(
+                relation=event.relation,  # pyright: ignore
+                relation_id=event.relation.id,  # pyright: ignore
+                app=self._charm.app,
+                unit=self._charm.unit,
+            )
+
+    def _on_logging_relation_broken(self, event: RelationBrokenEvent):
+        """Removes alert rules files when consumer charms left the relation with Loki.
+
+        Args:
+            event: a `CharmEvent` in response to which the Loki
+                charm must update its relation data.
+        """
+        self.on.loki_push_api_alert_rules_changed.emit(
+            relation=event.relation,
+            relation_id=event.relation.id,
+            app=self._charm.app,
+            unit=self._charm.unit,
+        )
+
+    def _on_logging_relation_departed(self, event: RelationDepartedEvent):
+        """Removes alert rules files when consumer charms left the relation with Loki.
+
+        Args:
+            event: a `CharmEvent` in response to which the Loki
+                charm must update its relation data.
+        """
+        self.on.loki_push_api_alert_rules_changed.emit(
+            relation=event.relation,
+            relation_id=event.relation.id,
+            app=self._charm.app,
+            unit=self._charm.unit,
+        )
+
+    def _should_update_alert_rules(self, relation) -> bool:
+        """Determine whether alert rules should be regenerated.
+
+        If there are alert rules in the relation data bag, tell the charm
+        whether to regenerate them based on the boolean returned here.
+        """
+        if relation.data.get(relation.app).get("alert_rules", None) is not None:
+            return True
+        return False
+
+    def _process_logging_relation_changed(self, relation: Relation) -> bool:
+        """Handle changes in related consumers.
+
+        Anytime there are changes in relations between Loki
+        and its consumers charms, Loki set the `loki_push_api`
+        into the relation data. Set the endpoint building
+        appropriately, and if there are alert rules present in
+        the relation, let the caller know.
+        Besides Loki generates alert rules files based what
+        consumer charms forwards,
+
+        Args:
+            relation: the `Relation` instance to update.
+
+        Returns:
+            A boolean indicating whether an event should be emitted, so we
+            only emit one on lifecycle events
+        """
+        relation.data[self._charm.unit]["public_address"] = socket.getfqdn() or ""
+        self.update_endpoint(relation=relation)
+        return self._should_update_alert_rules(relation)
+
+    @property
+    def _promtail_binary_url(self) -> dict:
+        """URL from which Promtail binary can be downloaded."""
+        # construct promtail binary url paths from parts
+        promtail_binaries = {}
+        for arch, info in PROMTAIL_BINARIES.items():
+            info["url"] = "{}/promtail-{}/{}.gz".format(
+                PROMTAIL_BASE_URL, PROMTAIL_VERSION, info["filename"]
+            )
+            promtail_binaries[arch] = info
+
+        return {"promtail_binary_zip_url": json.dumps(promtail_binaries)}
+
+    def update_endpoint(self, url: str = "", relation: Optional[Relation] = None) -> None:
+        """Triggers programmatically the update of endpoint in unit relation data.
+
+        This method should be used when the charm relying on this library needs
+        to update the relation data in response to something occurring outside
+        the `logging` relation lifecycle, e.g., in case of a
+        host address change because the charmed operator becomes connected to an
+        Ingress after the `logging` relation is established.
+
+        Args:
+            url: An optional url value to update relation data.
+            relation: An optional instance of `class:ops.model.Relation` to update.
+        """
+        # if no relation is specified update all of them
+        if not relation:
+            if not self._charm.model.relations.get(self._relation_name):
+                return
+
+            relations_list = self._charm.model.relations.get(self._relation_name)
+        else:
+            relations_list = [relation]
+
+        endpoint = self._endpoint(url or self._url)
+
+        for relation in relations_list:
+            relation.data[self._charm.unit].update({"endpoint": json.dumps(endpoint)})
+
+        logger.debug("Saved endpoint in unit relation data")
+
+    @property
+    def _url(self) -> str:
+        """Get local Loki Push API url.
+
+        Return url to loki, including port number, but without the endpoint subpath.
+        """
+        return "http://{}:{}".format(socket.getfqdn(), self.port)
+
+    def _endpoint(self, url) -> dict:
+        """Get Loki push API endpoint for a given url.
+
+        Args:
+            url: A loki unit URL.
+
+        Returns: str
+        """
+        endpoint = "/loki/api/v1/push"
+        return {"url": url.rstrip("/") + endpoint}
+
+    @property
+    def alerts(self) -> dict:  # noqa: C901
+        """Fetch alerts for all relations.
+
+        A Loki alert rules file consists of a list of "groups". Each
+        group consists of a list of alerts (`rules`) that are sequentially
+        executed. This method returns all the alert rules provided by each
+        related metrics provider charm. These rules may be used to generate a
+        separate alert rules file for each relation since the returned list
+        of alert groups are indexed by relation ID. Also for each relation ID
+        associated scrape metadata such as Juju model, UUID and application
+        name are provided so a unique name may be generated for the rules
+        file. For each relation the structure of data returned is a dictionary
+        with four keys
+
+        - groups
+        - model
+        - model_uuid
+        - application
+
+        The value of the `groups` key is such that it may be used to generate
+        a Loki alert rules file directly using `yaml.dump` but the
+        `groups` key itself must be included as this is required by Loki,
+        for example as in `yaml.dump({"groups": alerts["groups"]})`.
+
+        Currently only accepts a list of rules and these
+        rules are all placed into a single group, even though Loki itself
+        allows for multiple groups within a single alert rules file.
+
+        Returns:
+            a dictionary of alert rule groups and associated scrape
+            metadata indexed by relation ID.
+        """
+        alerts = {}  # type: Dict[str, dict] # mapping b/w juju identifiers and alert rule files
+        for relation in self._charm.model.relations[self._relation_name]:
+            if not relation.units or not relation.app:
+                continue
+
+            alert_rules = json.loads(relation.data[relation.app].get("alert_rules", "{}"))
+            if not alert_rules:
+                continue
+
+            alert_rules = self._inject_alert_expr_labels(alert_rules)
+
+            identifier, topology = self._get_identifier_by_alert_rules(alert_rules)
+            if not topology:
+                try:
+                    metadata = json.loads(relation.data[relation.app]["metadata"])
+                    identifier = JujuTopology.from_dict(metadata).identifier
+                    alerts[identifier] = self._tool.apply_label_matchers(alert_rules)  # type: ignore
+
+                except KeyError as e:
+                    logger.debug(
+                        "Relation %s has no 'metadata': %s",
+                        relation.id,
+                        e,
+                    )
+
+            if not identifier:
+                logger.error(
+                    "Alert rules were found but no usable group or identifier was present."
+                )
+                continue
+
+            _, errmsg = self._tool.validate_alert_rules(alert_rules)
+            if errmsg:
+                relation.data[self._charm.app]["event"] = json.dumps({"errors": errmsg})
+                continue
+
+            alerts[identifier] = alert_rules
+
+        return alerts
+
+    def _get_identifier_by_alert_rules(
+        self, rules: dict
+    ) -> Tuple[Union[str, None], Union[JujuTopology, None]]:
+        """Determine an appropriate dict key for alert rules.
+
+        The key is used as the filename when writing alerts to disk, so the structure
+        and uniqueness is important.
+
+        Args:
+            rules: a dict of alert rules
+        Returns:
+            A tuple containing an identifier, if found, and a JujuTopology, if it could
+            be constructed.
+        """
+        if "groups" not in rules:
+            logger.debug("No alert groups were found in relation data")
+            return None, None
+
+        # Construct an ID based on what's in the alert rules if they have labels
+        for group in rules["groups"]:
+            try:
+                labels = group["rules"][0]["labels"]
+                topology = JujuTopology(
+                    # Don't try to safely get required constructor fields. There's already
+                    # a handler for KeyErrors
+                    model_uuid=labels["juju_model_uuid"],
+                    model=labels["juju_model"],
+                    application=labels["juju_application"],
+                    unit=labels.get("juju_unit", ""),
+                    charm_name=labels.get("juju_charm", ""),
+                )
+                return topology.identifier, topology
+            except KeyError:
+                logger.debug("Alert rules were found but no usable labels were present")
+                continue
+
+        logger.warning(
+            "No labeled alert rules were found, and no 'scrape_metadata' "
+            "was available. Using the alert group name as filename."
+        )
+        try:
+            for group in rules["groups"]:
+                return group["name"], None
+        except KeyError:
+            logger.debug("No group name was found to use as identifier")
+
+        return None, None
+
+    def _inject_alert_expr_labels(self, rules: Dict[str, Any]) -> Dict[str, Any]:
+        """Iterate through alert rules and inject topology into expressions.
+
+        Args:
+            rules: a dict of alert rules
+        """
+        if "groups" not in rules:
+            return rules
+
+        modified_groups = []
+        for group in rules["groups"]:
+            # Copy off rules, so we don't modify an object we're iterating over
+            rules_copy = group["rules"]
+            for idx, rule in enumerate(rules_copy):
+                labels = rule.get("labels")
+
+                if labels:
+                    try:
+                        topology = JujuTopology(
+                            # Don't try to safely get required constructor fields. There's already
+                            # a handler for KeyErrors
+                            model_uuid=labels["juju_model_uuid"],
+                            model=labels["juju_model"],
+                            application=labels["juju_application"],
+                            unit=labels.get("juju_unit", ""),
+                            charm_name=labels.get("juju_charm", ""),
+                        )
+
+                        # Inject topology and put it back in the list
+                        rule["expr"] = self._tool.inject_label_matchers(
+                            re.sub(r"%%juju_topology%%,?", "", rule["expr"]),
+                            topology.label_matcher_dict,
+                        )
+                    except KeyError:
+                        # Some required JujuTopology key is missing. Just move on.
+                        pass
+
+                    group["rules"][idx] = rule
+
+            modified_groups.append(group)
+
+        rules["groups"] = modified_groups
+        return rules
+
+
+class ConsumerBase(Object):
+    """Consumer's base class."""
+
+    def __init__(
+        self,
+        charm: CharmBase,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        alert_rules_path: str = DEFAULT_ALERT_RULES_RELATIVE_PATH,
+        recursive: bool = False,
+        skip_alert_topology_labeling: bool = False,
+    ):
+        super().__init__(charm, relation_name)
+        self._charm = charm
+        self._relation_name = relation_name
+        self.topology = JujuTopology.from_charm(charm)
+
+        try:
+            alert_rules_path = _resolve_dir_against_charm_path(charm, alert_rules_path)
+        except InvalidAlertRulePathError as e:
+            logger.debug(
+                "Invalid Loki alert rules folder at %s: %s",
+                e.alert_rules_absolute_path,
+                e.message,
+            )
+        self._alert_rules_path = alert_rules_path
+        self._skip_alert_topology_labeling = skip_alert_topology_labeling
+
+        self._recursive = recursive
+
+    def _handle_alert_rules(self, relation):
+        if not self._charm.unit.is_leader():
+            return
+
+        alert_rules = (
+            AlertRules(None) if self._skip_alert_topology_labeling else AlertRules(self.topology)
+        )
+        alert_rules.add_path(self._alert_rules_path, recursive=self._recursive)
+        alert_rules_as_dict = alert_rules.as_dict()
+
+        relation.data[self._charm.app]["metadata"] = json.dumps(self.topology.as_dict())
+        relation.data[self._charm.app]["alert_rules"] = json.dumps(
+            alert_rules_as_dict,
+            sort_keys=True,  # sort, to prevent unnecessary relation_changed events
+        )
+
+    @property
+    def loki_endpoints(self) -> List[dict]:
+        """Fetch Loki Push API endpoints sent from LokiPushApiProvider through relation data.
+
+        Returns:
+            A list of dictionaries with Loki Push API endpoints, for instance:
+            [
+                {"url": "http://loki1:3100/loki/api/v1/push"},
+                {"url": "http://loki2:3100/loki/api/v1/push"},
+            ]
+        """
+        endpoints = []  # type: list
+
+        for relation in self._charm.model.relations[self._relation_name]:
+            for unit in relation.units:
+                if unit.app == self._charm.app:
+                    # This is a peer unit
+                    continue
+
+                endpoint = relation.data[unit].get("endpoint")
+                if endpoint:
+                    deserialized_endpoint = json.loads(endpoint)
+                    endpoints.append(deserialized_endpoint)
+
+        return endpoints
+
+
+class LokiPushApiConsumer(ConsumerBase):
+    """Loki Consumer class."""
+
+    on = LokiPushApiEvents()  # pyright: ignore
+
+    def __init__(
+        self,
+        charm: CharmBase,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        alert_rules_path: str = DEFAULT_ALERT_RULES_RELATIVE_PATH,
+        recursive: bool = True,
+        skip_alert_topology_labeling: bool = False,
+    ):
+        """Construct a Loki charm client.
+
+        The `LokiPushApiConsumer` object provides configurations to a Loki client charm, such as
+        the Loki API endpoint to push logs. It is intended for workloads that can speak
+        loki_push_api (https://grafana.com/docs/loki/latest/api/#push-log-entries-to-loki), such
+        as grafana-agent.
+        (If you need to forward workload stdout logs, then use v1/loki_push_api.LogForwarder; if
+        you need to forward log files, then use LogProxyConsumer.)
+
+        `LokiPushApiConsumer` can be instantiated as follows:
+
+            self._loki_consumer = LokiPushApiConsumer(self)
+
+        Args:
+            charm: a `CharmBase` object that manages this `LokiPushApiConsumer` object.
+                Typically, this is `self` in the instantiating class.
+            relation_name: the string name of the relation interface to look up.
+                If `charm` has exactly one relation with this interface, the relation's
+                name is returned. If none or multiple relations with the provided interface
+                are found, this method will raise either a NoRelationWithInterfaceFoundError or
+                MultipleRelationsWithInterfaceFoundError exception, respectively.
+            alert_rules_path: a string indicating a path where alert rules can be found
+            recursive: Whether to scan for rule files recursively.
+            skip_alert_topology_labeling: whether to skip the alert topology labeling.
+
+        Raises:
+            RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+                with the same name as provided via `relation_name` argument.
+            RelationInterfaceMismatchError: The relation with the same name as provided
+                via `relation_name` argument does not have the `loki_push_api` relation
+                interface.
+            RelationRoleMismatchError: If the relation with the same name as provided
+                via `relation_name` argument does not have the `RelationRole.provides`
+                role.
+
+        Emits:
+            loki_push_api_endpoint_joined: This event is emitted when the relation between the
+                Charmed Operator that instantiates `LokiPushApiProvider` (Loki charm for instance)
+                and the Charmed Operator that instantiates `LokiPushApiConsumer` is established.
+            loki_push_api_endpoint_departed: This event is emitted when the relation between the
+                Charmed Operator that implements `LokiPushApiProvider` (Loki charm for instance)
+                and the Charmed Operator that implements `LokiPushApiConsumer` is removed.
+            loki_push_api_alert_rules_error: This event is emitted when an invalid alert rules
+                file is encountered or if `alert_rules_path` is empty.
+        """
+        _validate_relation_by_interface_and_direction(
+            charm, relation_name, RELATION_INTERFACE_NAME, RelationRole.requires
+        )
+        super().__init__(
+            charm, relation_name, alert_rules_path, recursive, skip_alert_topology_labeling
+        )
+        events = self._charm.on[relation_name]
+        self.framework.observe(self._charm.on.upgrade_charm, self._on_lifecycle_event)
+        self.framework.observe(events.relation_joined, self._on_logging_relation_joined)
+        self.framework.observe(events.relation_changed, self._on_logging_relation_changed)
+        self.framework.observe(events.relation_departed, self._on_logging_relation_departed)
+
+    def _on_lifecycle_event(self, _: HookEvent):
+        """Update require relation data on charm upgrades and other lifecycle events.
+
+        Args:
+            event: a `CharmEvent` in response to which the consumer
+                charm must update its relation data.
+        """
+        # Upgrade event or other charm-level event
+        self._reinitialize_alert_rules()
+        self.on.loki_push_api_endpoint_joined.emit()
+
+    def _on_logging_relation_joined(self, event: RelationJoinedEvent):
+        """Handle changes in related consumers.
+
+        Update relation data and emit events when a relation is established.
+
+        Args:
+            event: a `CharmEvent` in response to which the consumer
+                charm must update its relation data.
+
+        Emits:
+            loki_push_api_endpoint_joined: Once the relation is established, this event is emitted.
+            loki_push_api_alert_rules_error: This event is emitted when an invalid alert rules
+                file is encountered or if `alert_rules_path` is empty.
+        """
+        # Alert rules will not change over the lifecycle of a charm, and do not need to be
+        # constantly set on every relation_changed event. Leave them here.
+        self._handle_alert_rules(event.relation)
+        self.on.loki_push_api_endpoint_joined.emit()
+
+    def _on_logging_relation_changed(self, event: RelationEvent):
+        """Handle changes in related consumers.
+
+        Anytime there are changes in the relation between Loki
+        and its consumers charms.
+
+        Args:
+            event: a `CharmEvent` in response to which the consumer
+                charm must update its relation data.
+
+        Emits:
+            loki_push_api_endpoint_joined: Once the relation is established, this event is emitted.
+            loki_push_api_alert_rules_error: This event is emitted when an invalid alert rules
+                file is encountered or if `alert_rules_path` is empty.
+        """
+        if self._charm.unit.is_leader():
+            ev = json.loads(event.relation.data[event.app].get("event", "{}"))
+
+            if ev:
+                valid = bool(ev.get("valid", True))
+                errors = ev.get("errors", "")
+
+                if valid and not errors:
+                    self.on.alert_rule_status_changed.emit(valid=valid)
+                else:
+                    self.on.alert_rule_status_changed.emit(valid=valid, errors=errors)
+
+        self.on.loki_push_api_endpoint_joined.emit()
+
+    def _reinitialize_alert_rules(self):
+        """Reloads alert rules and updates all relations."""
+        for relation in self._charm.model.relations[self._relation_name]:
+            self._handle_alert_rules(relation)
+
+    def _process_logging_relation_changed(self, relation: Relation):
+        self._handle_alert_rules(relation)
+        self.on.loki_push_api_endpoint_joined.emit()
+
+    def _on_logging_relation_departed(self, _: RelationEvent):
+        """Handle departures in related providers.
+
+        Anytime there are departures in relations between the consumer charm and Loki
+        the consumer charm is informed, through a `LokiPushApiEndpointDeparted` event.
+        The consumer charm can then choose to update its configuration.
+        """
+        # Provide default to avoid throwing, as in some complicated scenarios with
+        # upgrades and hook failures we might not have data in the storage
+        self.on.loki_push_api_endpoint_departed.emit()
+
+
+class ContainerNotFoundError(Exception):
+    """Raised if the specified container does not exist."""
+
+    def __init__(self):
+        msg = "The specified container does not exist."
+        self.message = msg
+
+        super().__init__(self.message)
+
+
+class MultipleContainersFoundError(Exception):
+    """Raised if no container name is passed but multiple containers are present."""
+
+    def __init__(self):
+        msg = (
+            "No 'container_name' parameter has been specified; since this Charmed Operator"
+            " is has multiple containers, container_name must be specified for the container"
+            " to get logs from."
+        )
+        self.message = msg
+
+        super().__init__(self.message)
+
+
+class PromtailDigestError(EventBase):
+    """Event emitted when there is an error with Promtail initialization."""
+
+    def __init__(self, handle, message):
+        super().__init__(handle)
+        self.message = message
+
+    def snapshot(self):
+        """Save message information."""
+        return {"message": self.message}
+
+    def restore(self, snapshot):
+        """Restore message information."""
+        self.message = snapshot["message"]
+
+
+class LogProxyEndpointDeparted(EventBase):
+    """Event emitted when a Log Proxy has departed."""
+
+
+class LogProxyEndpointJoined(EventBase):
+    """Event emitted when a Log Proxy joins."""
+
+
+class LogProxyEvents(ObjectEvents):
+    """Event descriptor for events raised by `LogProxyConsumer`."""
+
+    promtail_digest_error = EventSource(PromtailDigestError)
+    log_proxy_endpoint_departed = EventSource(LogProxyEndpointDeparted)
+    log_proxy_endpoint_joined = EventSource(LogProxyEndpointJoined)
+
+
+class LogProxyConsumer(ConsumerBase):
+    """LogProxyConsumer class.
+
+    > Note: This object is deprecated. Consider migrating to v1/loki_push_api.LogForwarder with the
+    > release of Juju 3.6 LTS.
+
+    The `LogProxyConsumer` object provides a method for attaching `promtail` to
+    a workload in order to generate structured logging data from applications
+    which traditionally log to syslog or do not have native Loki integration.
+    The `LogProxyConsumer` can be instantiated as follows:
+
+        self._log_proxy_consumer = LogProxyConsumer(self, log_files=["/var/log/messages"])
+
+    Args:
+        charm: a `CharmBase` object that manages this `LokiPushApiConsumer` object.
+            Typically, this is `self` in the instantiating class.
+        log_files: a list of log files to monitor with Promtail.
+        relation_name: the string name of the relation interface to look up.
+            If `charm` has exactly one relation with this interface, the relation's
+            name is returned. If none or multiple relations with the provided interface
+            are found, this method will raise either a NoRelationWithInterfaceFoundError or
+            MultipleRelationsWithInterfaceFoundError exception, respectively.
+        enable_syslog: Whether to enable syslog integration.
+        syslog_port: The port syslog is attached to.
+        alert_rules_path: an optional path for the location of alert rules
+            files. Defaults to "./src/loki_alert_rules",
+            resolved from the directory hosting the charm entry file.
+            The alert rules are automatically updated on charm upgrade.
+        recursive: Whether to scan for rule files recursively.
+        container_name: An optional container name to inject the payload into.
+        promtail_resource_name: An optional promtail resource name from metadata
+            if it has been modified and attached
+
+    Raises:
+        RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+            with the same name as provided via `relation_name` argument.
+        RelationInterfaceMismatchError: The relation with the same name as provided
+            via `relation_name` argument does not have the `loki_push_api` relation
+            interface.
+        RelationRoleMismatchError: If the relation with the same name as provided
+            via `relation_name` argument does not have the `RelationRole.provides`
+            role.
+    """
+
+    on = LogProxyEvents()  # pyright: ignore
+
+    def __init__(
+        self,
+        charm,
+        log_files: Optional[Union[List[str], str]] = None,
+        relation_name: str = DEFAULT_LOG_PROXY_RELATION_NAME,
+        enable_syslog: bool = False,
+        syslog_port: int = 1514,
+        alert_rules_path: str = DEFAULT_ALERT_RULES_RELATIVE_PATH,
+        recursive: bool = False,
+        container_name: str = "",
+        promtail_resource_name: Optional[str] = None,
+        *,  # TODO: In v1, move the star up so everything after 'charm' is a kwarg
+        insecure_skip_verify: bool = False,
+    ):
+        super().__init__(charm, relation_name, alert_rules_path, recursive)
+        self._charm = charm
+        self._relation_name = relation_name
+        self._container = self._get_container(container_name)
+        self._container_name = self._get_container_name(container_name)
+
+        if not log_files:
+            log_files = []
+        elif isinstance(log_files, str):
+            log_files = [log_files]
+        elif not isinstance(log_files, list) or not all((isinstance(x, str) for x in log_files)):
+            raise TypeError("The 'log_files' argument must be a list of strings.")
+        self._log_files = log_files
+
+        self._syslog_port = syslog_port
+        self._is_syslog = enable_syslog
+        self.topology = JujuTopology.from_charm(charm)
+        self._promtail_resource_name = promtail_resource_name or "promtail-bin"
+        self.insecure_skip_verify = insecure_skip_verify
+
+        # architecture used for promtail binary
+        arch = platform.processor()
+        self._arch = "amd64" if arch == "x86_64" else arch
+
+        events = self._charm.on[relation_name]
+        self.framework.observe(events.relation_created, self._on_relation_created)
+        self.framework.observe(events.relation_changed, self._on_relation_changed)
+        self.framework.observe(events.relation_departed, self._on_relation_departed)
+        # turn the container name to a valid Python identifier
+        snake_case_container_name = self._container_name.replace("-", "_")
+        self.framework.observe(
+            getattr(self._charm.on, "{}_pebble_ready".format(snake_case_container_name)),
+            self._on_pebble_ready,
+        )
+
+    def _on_pebble_ready(self, _: WorkloadEvent):
+        """Event handler for `pebble_ready`."""
+        if self.model.relations[self._relation_name]:
+            self._setup_promtail()
+
+    def _on_relation_created(self, _: RelationCreatedEvent) -> None:
+        """Event handler for `relation_created`."""
+        if not self._container.can_connect():
+            return
+        self._setup_promtail()
+
+    def _on_relation_changed(self, event: RelationEvent) -> None:
+        """Event handler for `relation_changed`.
+
+        Args:
+            event: The event object `RelationChangedEvent`.
+        """
+        self._handle_alert_rules(event.relation)
+
+        if self._charm.unit.is_leader():
+            ev = json.loads(event.relation.data[event.app].get("event", "{}"))
+
+            if ev:
+                valid = bool(ev.get("valid", True))
+                errors = ev.get("errors", "")
+
+                if valid and not errors:
+                    self.on.alert_rule_status_changed.emit(valid=valid)
+                else:
+                    self.on.alert_rule_status_changed.emit(valid=valid, errors=errors)
+
+        if not self._container.can_connect():
+            return
+        if self.model.relations[self._relation_name]:
+            if "promtail" not in self._container.get_plan().services:
+                self._setup_promtail()
+                return
+
+            new_config = self._promtail_config
+            if new_config != self._current_config:
+                self._container.push(
+                    WORKLOAD_CONFIG_PATH, yaml.safe_dump(new_config), make_dirs=True
+                )
+
+            # Loki may send endpoints late. Don't necessarily start, there may be
+            # no clients
+            if new_config["clients"]:
+                self._container.restart(WORKLOAD_SERVICE_NAME)
+                self.on.log_proxy_endpoint_joined.emit()
+            else:
+                self.on.promtail_digest_error.emit("No promtail client endpoints available!")
+
+    def _on_relation_departed(self, _: RelationEvent) -> None:
+        """Event handler for `relation_departed`.
+
+        Args:
+            event: The event object `RelationDepartedEvent`.
+        """
+        if not self._container.can_connect():
+            return
+        if not self._charm.model.relations[self._relation_name]:
+            self._container.stop(WORKLOAD_SERVICE_NAME)
+            return
+
+        new_config = self._promtail_config
+        if new_config != self._current_config:
+            self._container.push(WORKLOAD_CONFIG_PATH, yaml.safe_dump(new_config), make_dirs=True)
+
+        if new_config["clients"]:
+            self._container.restart(WORKLOAD_SERVICE_NAME)
+        else:
+            self._container.stop(WORKLOAD_SERVICE_NAME)
+        self.on.log_proxy_endpoint_departed.emit()
+
+    def _get_container(self, container_name: str = "") -> Container:  # pyright: ignore
+        """Gets a single container by name or using the only container running in the Pod.
+
+        If there is more than one container in the Pod a `PromtailDigestError` is emitted.
+
+        Args:
+            container_name: The container name.
+
+        Returns:
+            A `ops.model.Container` object representing the container.
+
+        Emits:
+            PromtailDigestError, if there was a problem obtaining a container.
+        """
+        try:
+            container_name = self._get_container_name(container_name)
+            return self._charm.unit.get_container(container_name)
+        except (MultipleContainersFoundError, ContainerNotFoundError, ModelError) as e:
+            msg = str(e)
+            logger.warning(msg)
+            self.on.promtail_digest_error.emit(msg)
+
+    def _get_container_name(self, container_name: str = "") -> str:
+        """Helper function for getting/validating a container name.
+
+        Args:
+            container_name: The container name to be validated (optional).
+
+        Returns:
+            container_name: The same container_name that was passed (if it exists) or the only
+            container name that is present (if no container_name was passed).
+
+        Raises:
+            ContainerNotFoundError, if container_name does not exist.
+            MultipleContainersFoundError, if container_name was not provided but multiple
+            containers are present.
+        """
+        containers = dict(self._charm.model.unit.containers)
+        if len(containers) == 0:
+            raise ContainerNotFoundError
+
+        if not container_name:
+            # container_name was not provided - will get it ourselves, if it is the only one
+            if len(containers) > 1:
+                raise MultipleContainersFoundError
+
+            # Get the first key in the containers' dict.
+            # Need to "cast", otherwise:
+            # error: Incompatible return value type (got "Optional[str]", expected "str")
+            container_name = cast(str, next(iter(containers.keys())))
+
+        elif container_name not in containers:
+            raise ContainerNotFoundError
+
+        return container_name
+
+    def _add_pebble_layer(self, workload_binary_path: str) -> None:
+        """Adds Pebble layer that manages Promtail service in Workload container.
+
+        Args:
+            workload_binary_path: string providing path to promtail binary in workload container.
+        """
+        pebble_layer = {
+            "summary": "promtail layer",
+            "description": "pebble config layer for promtail",
+            "services": {
+                WORKLOAD_SERVICE_NAME: {
+                    "override": "replace",
+                    "summary": WORKLOAD_SERVICE_NAME,
+                    "command": "{} {}".format(workload_binary_path, self._cli_args),
+                    "startup": "disabled",
+                }
+            },
+        }
+        self._container.add_layer(
+            self._container_name, pebble_layer, combine=True  # pyright: ignore
+        )
+
+    def _create_directories(self) -> None:
+        """Creates the directories for Promtail binary and config file."""
+        self._container.make_dir(path=WORKLOAD_BINARY_DIR, make_parents=True)
+        self._container.make_dir(path=WORKLOAD_CONFIG_DIR, make_parents=True)
+
+    def _obtain_promtail(self, promtail_info: dict) -> None:
+        """Obtain promtail binary from an attached resource or download it.
+
+        Args:
+            promtail_info: dictionary containing information about promtail binary
+               that must be used. The dictionary must have three keys
+               - "filename": filename of promtail binary
+               - "zipsha": sha256 sum of zip file of promtail binary
+               - "binsha": sha256 sum of unpacked promtail binary
+        """
+        workload_binary_path = os.path.join(WORKLOAD_BINARY_DIR, promtail_info["filename"])
+        if self._promtail_attached_as_resource:
+            self._push_promtail_if_attached(workload_binary_path)
+            return
+
+        if self._promtail_must_be_downloaded(promtail_info):
+            self._download_and_push_promtail_to_workload(promtail_info)
+        else:
+            binary_path = os.path.join(BINARY_DIR, promtail_info["filename"])
+            self._push_binary_to_workload(binary_path, workload_binary_path)
+
+    def _push_binary_to_workload(self, binary_path: str, workload_binary_path: str) -> None:
+        """Push promtail binary into workload container.
+
+        Args:
+            binary_path: path in charm container from which promtail binary is read.
+            workload_binary_path: path in workload container to which promtail binary is pushed.
+        """
+        with open(binary_path, "rb") as f:
+            self._container.push(
+                workload_binary_path,
+                f,
+                permissions=0o755,
+                encoding=None,  # pyright: ignore
+                make_dirs=True,
+            )
+            logger.debug("The promtail binary file has been pushed to the workload container.")
+
+    @property
+    def _promtail_attached_as_resource(self) -> bool:
+        """Checks whether Promtail binary is attached to the charm or not.
+
+        Returns:
+            a boolean representing whether Promtail binary is attached as a resource or not.
+        """
+        try:
+            self._charm.model.resources.fetch(self._promtail_resource_name)
+            return True
+        except ModelError:
+            return False
+        except NameError as e:
+            if "invalid resource name" in str(e):
+                return False
+            raise
+
+    def _push_promtail_if_attached(self, workload_binary_path: str) -> bool:
+        """Checks whether Promtail binary is attached to the charm or not.
+
+        Args:
+            workload_binary_path: string specifying expected path of promtail
+                in workload container
+
+        Returns:
+            a boolean representing whether Promtail binary is attached or not.
+        """
+        logger.info("Promtail binary file has been obtained from an attached resource.")
+        resource_path = self._charm.model.resources.fetch(self._promtail_resource_name)
+        self._push_binary_to_workload(resource_path, workload_binary_path)
+        return True
+
+    def _promtail_must_be_downloaded(self, promtail_info: dict) -> bool:
+        """Checks whether promtail binary must be downloaded or not.
+
+        Args:
+            promtail_info: dictionary containing information about promtail binary
+               that must be used. The dictionary must have three keys
+               - "filename": filename of promtail binary
+               - "zipsha": sha256 sum of zip file of promtail binary
+               - "binsha": sha256 sum of unpacked promtail binary
+
+        Returns:
+            a boolean representing whether Promtail binary must be downloaded or not.
+        """
+        binary_path = os.path.join(BINARY_DIR, promtail_info["filename"])
+        if not self._is_promtail_binary_in_charm(binary_path):
+            return True
+
+        if not self._sha256sums_matches(binary_path, promtail_info["binsha"]):
+            return True
+
+        logger.debug("Promtail binary file is already in the the charm container.")
+        return False
+
+    def _sha256sums_matches(self, file_path: str, sha256sum: str) -> bool:
+        """Checks whether a file's sha256sum matches or not with a specific sha256sum.
+
+        Args:
+            file_path: A string representing the files' patch.
+            sha256sum: The sha256sum against which we want to verify.
+
+        Returns:
+            a boolean representing whether a file's sha256sum matches or not with
+            a specific sha256sum.
+        """
+        try:
+            with open(file_path, "rb") as f:
+                file_bytes = f.read()
+                result = sha256(file_bytes).hexdigest()
+
+                if result != sha256sum:
+                    msg = "File sha256sum mismatch, expected:'{}' but got '{}'".format(
+                        sha256sum, result
+                    )
+                    logger.debug(msg)
+                    return False
+
+                return True
+        except (APIError, FileNotFoundError):
+            msg = "File: '{}' could not be opened".format(file_path)
+            logger.error(msg)
+            return False
+
+    def _is_promtail_binary_in_charm(self, binary_path: str) -> bool:
+        """Check if Promtail binary is already stored in charm container.
+
+        Args:
+            binary_path: string path of promtail binary to check
+
+        Returns:
+            a boolean representing whether Promtail is present or not.
+        """
+        return True if Path(binary_path).is_file() else False
+
+    def _download_and_push_promtail_to_workload(self, promtail_info: dict) -> None:
+        """Downloads a Promtail zip file and pushes the binary to the workload.
+
+        Args:
+            promtail_info: dictionary containing information about promtail binary
+               that must be used. The dictionary must have three keys
+               - "filename": filename of promtail binary
+               - "zipsha": sha256 sum of zip file of promtail binary
+               - "binsha": sha256 sum of unpacked promtail binary
+        """
+        # Check for Juju proxy variables and fall back to standard ones if not set
+        # If no Juju proxy variable was set, we set proxies to None to let the ProxyHandler get
+        # the proxy env variables from the environment
+        proxies = {
+            # The ProxyHandler uses only the protocol names as keys
+            # https://docs.python.org/3/library/urllib.request.html#urllib.request.ProxyHandler
+            "https": os.environ.get("JUJU_CHARM_HTTPS_PROXY", ""),
+            "http": os.environ.get("JUJU_CHARM_HTTP_PROXY", ""),
+            # The ProxyHandler uses `no` for the no_proxy key
+            # https://github.com/python/cpython/blob/3.12/Lib/urllib/request.py#L2553
+            "no": os.environ.get("JUJU_CHARM_NO_PROXY", ""),
+        }
+        proxies = {k: v for k, v in proxies.items() if v != ""} or None
+
+        proxy_handler = request.ProxyHandler(proxies)
+        opener = request.build_opener(proxy_handler)
+
+        with opener.open(promtail_info["url"]) as r:
+            file_bytes = r.read()
+            file_path = os.path.join(BINARY_DIR, promtail_info["filename"] + ".gz")
+            with open(file_path, "wb") as f:
+                f.write(file_bytes)
+                logger.info(
+                    "Promtail binary zip file has been downloaded and stored in: %s",
+                    file_path,
+                )
+
+            decompressed_file = GzipFile(fileobj=BytesIO(file_bytes))
+            binary_path = os.path.join(BINARY_DIR, promtail_info["filename"])
+            with open(binary_path, "wb") as outfile:
+                outfile.write(decompressed_file.read())
+                logger.debug("Promtail binary file has been downloaded.")
+
+        workload_binary_path = os.path.join(WORKLOAD_BINARY_DIR, promtail_info["filename"])
+        self._push_binary_to_workload(binary_path, workload_binary_path)
+
+    @property
+    def _cli_args(self) -> str:
+        """Return the cli arguments to pass to promtail.
+
+        Returns:
+            The arguments as a string
+        """
+        return "-config.file={}".format(WORKLOAD_CONFIG_PATH)
+
+    @property
+    def _current_config(self) -> dict:
+        """Property that returns the current Promtail configuration.
+
+        Returns:
+            A dict containing Promtail configuration.
+        """
+        if not self._container.can_connect():
+            logger.debug("Could not connect to promtail container!")
+            return {}
+        try:
+            raw_current = self._container.pull(WORKLOAD_CONFIG_PATH).read()
+            return yaml.safe_load(raw_current)
+        except (ProtocolError, PathError) as e:
+            logger.warning(
+                "Could not check the current promtail configuration due to "
+                "a failure in retrieving the file: %s",
+                e,
+            )
+            return {}
+
+    @property
+    def _promtail_config(self) -> dict:
+        """Generates the config file for Promtail.
+
+        Reference: https://grafana.com/docs/loki/latest/send-data/promtail/configuration
+        """
+        config = {"clients": self._clients_list()}
+        if self.insecure_skip_verify:
+            for client in config["clients"]:
+                client["tls_config"] = {"insecure_skip_verify": True}
+
+        config.update(self._server_config())
+        config.update(self._positions())
+        config.update(self._scrape_configs())
+        return config
+
+    def _clients_list(self) -> list:
+        """Generates a list of clients for use in the promtail config.
+
+        Returns:
+            A list of endpoints
+        """
+        return self.loki_endpoints
+
+    def _server_config(self) -> dict:
+        """Generates the server section of the Promtail config file.
+
+        Returns:
+            A dict representing the `server` section.
+        """
+        return {
+            "server": {
+                "http_listen_port": HTTP_LISTEN_PORT,
+                "grpc_listen_port": GRPC_LISTEN_PORT,
+            }
+        }
+
+    def _positions(self) -> dict:
+        """Generates the positions section of the Promtail config file.
+
+        Returns:
+            A dict representing the `positions` section.
+        """
+        return {"positions": {"filename": WORKLOAD_POSITIONS_PATH}}
+
+    def _scrape_configs(self) -> dict:
+        """Generates the scrape_configs section of the Promtail config file.
+
+        Returns:
+            A dict representing the `scrape_configs` section.
+        """
+        job_name = "juju_{}".format(self.topology.identifier)
+
+        # The new JujuTopology doesn't include unit, but LogProxyConsumer should have it
+        common_labels = {
+            "juju_{}".format(k): v
+            for k, v in self.topology.as_dict(remapped_keys={"charm_name": "charm"}).items()
+        }
+        scrape_configs = []
+
+        # Files config
+        labels = common_labels.copy()
+        labels.update(
+            {
+                "job": job_name,
+                "__path__": "",
+            }
+        )
+        config = {"targets": ["localhost"], "labels": labels}
+        scrape_config = {
+            "job_name": "system",
+            "static_configs": self._generate_static_configs(config),
+        }
+        scrape_configs.append(scrape_config)
+
+        # Syslog config
+        if self._is_syslog:
+            relabel_mappings = [
+                "severity",
+                "facility",
+                "hostname",
+                "app_name",
+                "proc_id",
+                "msg_id",
+            ]
+            syslog_labels = common_labels.copy()
+            syslog_labels.update({"job": "{}_syslog".format(job_name)})
+            syslog_config = {
+                "job_name": "syslog",
+                "syslog": {
+                    "listen_address": "127.0.0.1:{}".format(self._syslog_port),
+                    "label_structured_data": True,
+                    "labels": syslog_labels,
+                },
+                "relabel_configs": [
+                    {"source_labels": ["__syslog_message_{}".format(val)], "target_label": val}
+                    for val in relabel_mappings
+                ]
+                + [{"action": "labelmap", "regex": "__syslog_message_sd_(.+)"}],
+            }
+            scrape_configs.append(syslog_config)  # type: ignore
+
+        return {"scrape_configs": scrape_configs}
+
+    def _generate_static_configs(self, config: dict) -> list:
+        """Generates static_configs section.
+
+        Returns:
+            - a list of dictionaries representing static_configs section
+        """
+        static_configs = []
+
+        for _file in self._log_files:
+            conf = deepcopy(config)
+            conf["labels"]["__path__"] = _file
+            static_configs.append(conf)
+
+        return static_configs
+
+    def _setup_promtail(self) -> None:
+        # Use the first
+        relations = self._charm.model.relations[self._relation_name]
+        if len(relations) > 1:
+            logger.debug(
+                "Multiple log_proxy relations. Getting Promtail from application {}".format(
+                    relations[0].app.name
+                )
+            )
+        relation = relations[0]
+        promtail_binaries = json.loads(
+            relation.data[relation.app].get("promtail_binary_zip_url", "{}")
+        )
+        if not promtail_binaries:
+            return
+
+        if not self._is_promtail_installed(promtail_binaries[self._arch]):
+            try:
+                self._obtain_promtail(promtail_binaries[self._arch])
+            except HTTPError as e:
+                msg = "Promtail binary couldn't be downloaded - {}".format(str(e))
+                logger.warning(msg)
+                self.on.promtail_digest_error.emit(msg)
+                return
+
+        workload_binary_path = os.path.join(
+            WORKLOAD_BINARY_DIR, promtail_binaries[self._arch]["filename"]
+        )
+
+        self._create_directories()
+        self._container.push(
+            WORKLOAD_CONFIG_PATH, yaml.safe_dump(self._promtail_config), make_dirs=True
+        )
+
+        self._add_pebble_layer(workload_binary_path)
+
+        if self._current_config.get("clients"):
+            try:
+                self._container.restart(WORKLOAD_SERVICE_NAME)
+            except ChangeError as e:
+                self.on.promtail_digest_error.emit(str(e))
+            else:
+                self.on.log_proxy_endpoint_joined.emit()
+        else:
+            self.on.promtail_digest_error.emit("No promtail client endpoints available!")
+
+    def _is_promtail_installed(self, promtail_info: dict) -> bool:
+        """Determine if promtail has already been installed to the container.
+
+        Args:
+            promtail_info: dictionary containing information about promtail binary
+               that must be used. The dictionary must at least contain a key
+               "filename" giving the name of promtail binary
+        """
+        workload_binary_path = "{}/{}".format(WORKLOAD_BINARY_DIR, promtail_info["filename"])
+        try:
+            self._container.list_files(workload_binary_path)
+        except (APIError, FileNotFoundError):
+            return False
+        return True
+
+    @property
+    def syslog_port(self) -> str:
+        """Gets the port on which promtail is listening for syslog.
+
+        Returns:
+            A str representing the port
+        """
+        return str(self._syslog_port)
+
+    @property
+    def rsyslog_config(self) -> str:
+        """Generates a config line for use with rsyslog.
+
+        Returns:
+            The rsyslog config line as a string
+        """
+        return 'action(type="omfwd" protocol="tcp" target="127.0.0.1" port="{}" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted")'.format(
+            self._syslog_port
+        )
+
+
+class CosTool:
+    """Uses cos-tool to inject label matchers into alert rule expressions and validate rules."""
+
+    _path = None
+    _disabled = False
+
+    def __init__(self, charm):
+        self._charm = charm
+
+    @property
+    def path(self):
+        """Lazy lookup of the path of cos-tool."""
+        if self._disabled:
+            return None
+        if not self._path:
+            self._path = self._get_tool_path()
+            if not self._path:
+                logger.debug("Skipping injection of juju topology as label matchers")
+                self._disabled = True
+        return self._path
+
+    def apply_label_matchers(self, rules) -> dict:
+        """Will apply label matchers to the expression of all alerts in all supplied groups."""
+        if not self.path:
+            return rules
+        for group in rules["groups"]:
+            rules_in_group = group.get("rules", [])
+            for rule in rules_in_group:
+                topology = {}
+                # if the user for some reason has provided juju_unit, we'll need to honor it
+                # in most cases, however, this will be empty
+                for label in [
+                    "juju_model",
+                    "juju_model_uuid",
+                    "juju_application",
+                    "juju_charm",
+                    "juju_unit",
+                ]:
+                    if label in rule["labels"]:
+                        topology[label] = rule["labels"][label]
+
+                rule["expr"] = self.inject_label_matchers(rule["expr"], topology)
+        return rules
+
+    def validate_alert_rules(self, rules: dict) -> Tuple[bool, str]:
+        """Will validate correctness of alert rules, returning a boolean and any errors."""
+        if not self.path:
+            logger.debug("`cos-tool` unavailable. Not validating alert correctness.")
+            return True, ""
+
+        with tempfile.TemporaryDirectory() as tmpdir:
+            rule_path = Path(tmpdir + "/validate_rule.yaml")
+
+            # Smash "our" rules format into what upstream actually uses, which is more like:
+            #
+            # groups:
+            #   - name: foo
+            #     rules:
+            #       - alert: SomeAlert
+            #         expr: up
+            #       - alert: OtherAlert
+            #         expr: up
+            transformed_rules = {"groups": []}  # type: ignore
+            for rule in rules["groups"]:
+                transformed_rules["groups"].append(rule)
+
+            rule_path.write_text(yaml.dump(transformed_rules))
+            args = [str(self.path), "--format", "logql", "validate", str(rule_path)]
+            # noinspection PyBroadException
+            try:
+                self._exec(args)
+                return True, ""
+            except subprocess.CalledProcessError as e:
+                logger.debug("Validating the rules failed: %s", e.output)
+                return False, ", ".join([line for line in e.output if "error validating" in line])
+
+    def inject_label_matchers(self, expression, topology) -> str:
+        """Add label matchers to an expression."""
+        if not topology:
+            return expression
+        if not self.path:
+            logger.debug("`cos-tool` unavailable. Leaving expression unchanged: %s", expression)
+            return expression
+        args = [str(self.path), "--format", "logql", "transform"]
+        args.extend(
+            ["--label-matcher={}={}".format(key, value) for key, value in topology.items()]
+        )
+
+        args.extend(["{}".format(expression)])
+        # noinspection PyBroadException
+        try:
+            return self._exec(args)
+        except subprocess.CalledProcessError as e:
+            logger.debug('Applying the expression failed: "%s", falling back to the original', e)
+            print('Applying the expression failed: "{}", falling back to the original'.format(e))
+            return expression
+
+    def _get_tool_path(self) -> Optional[Path]:
+        arch = platform.processor()
+        arch = "amd64" if arch == "x86_64" else arch
+        res = "cos-tool-{}".format(arch)
+        try:
+            path = Path(res).resolve()
+            path.chmod(0o777)
+            return path
+        except NotImplementedError:
+            logger.debug("System lacks support for chmod")
+        except FileNotFoundError:
+            logger.debug('Could not locate cos-tool at: "{}"'.format(res))
+        return None
+
+    def _exec(self, cmd) -> str:
+        result = subprocess.run(cmd, check=True, stdout=subprocess.PIPE)
+        output = result.stdout.decode("utf-8").strip()
+        return output
diff --git a/lib/charms/observability_libs/v0/kubernetes_service_patch.py b/lib/charms/observability_libs/v0/kubernetes_service_patch.py
new file mode 100644
index 0000000..a3fb910
--- /dev/null
+++ b/lib/charms/observability_libs/v0/kubernetes_service_patch.py
@@ -0,0 +1,280 @@
+# Copyright 2021 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+"""# KubernetesServicePatch Library.
+
+This library is designed to enable developers to more simply patch the Kubernetes Service created
+by Juju during the deployment of a sidecar charm. When sidecar charms are deployed, Juju creates a
+service named after the application in the namespace (named after the Juju model). This service by
+default contains a "placeholder" port, which is 65536/TCP.
+
+When modifying the default set of resources managed by Juju, one must consider the lifecycle of the
+charm. In this case, any modifications to the default service (created during deployment), will
+be overwritten during a charm upgrade.
+
+When initialised, this library binds a handler to the parent charm's `install` and `upgrade_charm`
+events which applies the patch to the cluster. This should ensure that the service ports are
+correct throughout the charm's life.
+
+The constructor simply takes a reference to the parent charm, and a list of tuples that each define
+a port for the service, where each tuple contains:
+
+- a name for the port
+- port for the service to listen on
+- optionally: a targetPort for the service (the port in the container!)
+- optionally: a nodePort for the service (for NodePort or LoadBalancer services only!)
+- optionally: a name of the service (in case service name needs to be patched as well)
+
+## Getting Started
+
+To get started using the library, you just need to fetch the library using `charmcraft`. **Note
+that you also need to add `lightkube` and `lightkube-models` to your charm's `requirements.txt`.**
+
+```shell
+cd some-charm
+charmcraft fetch-lib charms.observability_libs.v0.kubernetes_service_patch
+echo <<-EOF >> requirements.txt
+lightkube
+lightkube-models
+EOF
+```
+
+Then, to initialise the library:
+
+For ClusterIP services:
+```python
+# ...
+from charms.observability_libs.v0.kubernetes_service_patch import KubernetesServicePatch
+
+class SomeCharm(CharmBase):
+  def __init__(self, *args):
+    # ...
+    self.service_patcher = KubernetesServicePatch(self, [(f"{self.app.name}", 8080)])
+    # ...
+```
+
+For LoadBalancer/NodePort services:
+```python
+# ...
+from charms.observability_libs.v0.kubernetes_service_patch import KubernetesServicePatch
+
+class SomeCharm(CharmBase):
+  def __init__(self, *args):
+    # ...
+    self.service_patcher = KubernetesServicePatch(
+        self, [(f"{self.app.name}", 443, 443, 30666)], "LoadBalancer"
+    )
+    # ...
+```
+
+Additionally, you may wish to use mocks in your charm's unit testing to ensure that the library
+does not try to make any API calls, or open any files during testing that are unlikely to be
+present, and could break your tests. The easiest way to do this is during your test `setUp`:
+
+```python
+# ...
+
+@patch("charm.KubernetesServicePatch", lambda x, y: None)
+def setUp(self, *unused):
+    self.harness = Harness(SomeCharm)
+    # ...
+```
+"""
+
+import logging
+from types import MethodType
+from typing import Literal, Sequence, Tuple, Union
+
+from lightkube import ApiError, Client
+from lightkube.models.core_v1 import ServicePort, ServiceSpec
+from lightkube.models.meta_v1 import ObjectMeta
+from lightkube.resources.core_v1 import Service
+from lightkube.types import PatchType
+from ops.charm import CharmBase
+from ops.framework import Object
+
+logger = logging.getLogger(__name__)
+
+# The unique Charmhub library identifier, never change it
+LIBID = "0042f86d0a874435adef581806cddbbb"
+
+# Increment this major API version when introducing breaking changes
+LIBAPI = 0
+
+# Increment this PATCH version before using `charmcraft publish-lib` or reset
+# to 0 if you are raising the major API version
+LIBPATCH = 6
+
+PortDefinition = Union[Tuple[str, int], Tuple[str, int, int], Tuple[str, int, int, int]]
+ServiceType = Literal["ClusterIP", "LoadBalancer"]
+
+
+class KubernetesServicePatch(Object):
+    """A utility for patching the Kubernetes service set up by Juju."""
+
+    def __init__(
+        self,
+        charm: CharmBase,
+        ports: Sequence[PortDefinition],
+        service_name: str = None,
+        service_type: ServiceType = "ClusterIP",
+        additional_labels: dict = None,
+        additional_selectors: dict = None,
+        additional_annotations: dict = None,
+    ):
+        """Constructor for KubernetesServicePatch.
+
+        Args:
+            charm: the charm that is instantiating the library.
+            ports: a list of tuples (name, port, targetPort, nodePort) for every service port.
+            service_name: allows setting custom name to the patched service. If none given,
+                application name will be used.
+            service_type: desired type of K8s service. Default value is in line with ServiceSpec's
+                default value.
+            additional_labels: Labels to be added to the kubernetes service (by default only
+                "app.kubernetes.io/name" is set to the service name)
+            additional_selectors: Selectors to be added to the kubernetes service (by default only
+                "app.kubernetes.io/name" is set to the service name)
+            additional_annotations: Annotations to be added to the kubernetes service.
+        """
+        super().__init__(charm, "kubernetes-service-patch")
+        self.charm = charm
+        self.service_name = service_name if service_name else self._app
+        self.service = self._service_object(
+            ports,
+            service_name,
+            service_type,
+            additional_labels,
+            additional_selectors,
+            additional_annotations,
+        )
+
+        # Make mypy type checking happy that self._patch is a method
+        assert isinstance(self._patch, MethodType)
+        # Ensure this patch is applied during the 'install' and 'upgrade-charm' events
+        self.framework.observe(charm.on.install, self._patch)
+        self.framework.observe(charm.on.upgrade_charm, self._patch)
+
+    def _service_object(
+        self,
+        ports: Sequence[PortDefinition],
+        service_name: str = None,
+        service_type: ServiceType = "ClusterIP",
+        additional_labels: dict = None,
+        additional_selectors: dict = None,
+        additional_annotations: dict = None,
+    ) -> Service:
+        """Creates a valid Service representation.
+
+        Args:
+            ports: a list of tuples of the form (name, port) or (name, port, targetPort)
+                or (name, port, targetPort, nodePort) for every service port. If the 'targetPort'
+                is omitted, it is assumed to be equal to 'port', with the exception of NodePort
+                and LoadBalancer services, where all port numbers have to be specified.
+            service_name: allows setting custom name to the patched service. If none given,
+                application name will be used.
+            service_type: desired type of K8s service. Default value is in line with ServiceSpec's
+                default value.
+            additional_labels: Labels to be added to the kubernetes service (by default only
+                "app.kubernetes.io/name" is set to the service name)
+            additional_selectors: Selectors to be added to the kubernetes service (by default only
+                "app.kubernetes.io/name" is set to the service name)
+            additional_annotations: Annotations to be added to the kubernetes service.
+
+        Returns:
+            Service: A valid representation of a Kubernetes Service with the correct ports.
+        """
+        if not service_name:
+            service_name = self._app
+        labels = {"app.kubernetes.io/name": self._app}
+        if additional_labels:
+            labels.update(additional_labels)
+        selector = {"app.kubernetes.io/name": self._app}
+        if additional_selectors:
+            selector.update(additional_selectors)
+        return Service(
+            apiVersion="v1",
+            kind="Service",
+            metadata=ObjectMeta(
+                namespace=self._namespace,
+                name=service_name,
+                labels=labels,
+                annotations=additional_annotations,  # type: ignore[arg-type]
+            ),
+            spec=ServiceSpec(
+                selector=selector,
+                ports=[
+                    ServicePort(
+                        name=p[0],
+                        port=p[1],
+                        targetPort=p[2] if len(p) > 2 else p[1],  # type: ignore[misc]
+                        nodePort=p[3] if len(p) > 3 else None,  # type: ignore[arg-type, misc]
+                    )
+                    for p in ports
+                ],
+                type=service_type,
+            ),
+        )
+
+    def _patch(self, _) -> None:
+        """Patch the Kubernetes service created by Juju to map the correct port.
+
+        Raises:
+            PatchFailed: if patching fails due to lack of permissions, or otherwise.
+        """
+        if not self.charm.unit.is_leader():
+            return
+
+        client = Client()
+        try:
+            if self.service_name != self._app:
+                self._delete_and_create_service(client)
+            client.patch(Service, self.service_name, self.service, patch_type=PatchType.MERGE)
+        except ApiError as e:
+            if e.status.code == 403:
+                logger.error("Kubernetes service patch failed: `juju trust` this application.")
+            else:
+                logger.error("Kubernetes service patch failed: %s", str(e))
+        else:
+            logger.info("Kubernetes service '%s' patched successfully", self._app)
+
+    def _delete_and_create_service(self, client: Client):
+        service = client.get(Service, self._app, namespace=self._namespace)
+        service.metadata.name = self.service_name  # type: ignore[attr-defined]
+        service.metadata.resourceVersion = service.metadata.uid = None  # type: ignore[attr-defined]   # noqa: E501
+        client.delete(Service, self._app, namespace=self._namespace)
+        client.create(service)
+
+    def is_patched(self) -> bool:
+        """Reports if the service patch has been applied.
+
+        Returns:
+            bool: A boolean indicating if the service patch has been applied.
+        """
+        client = Client()
+        # Get the relevant service from the cluster
+        service = client.get(Service, name=self.service_name, namespace=self._namespace)
+        # Construct a list of expected ports, should the patch be applied
+        expected_ports = [(p.port, p.targetPort) for p in self.service.spec.ports]
+        # Construct a list in the same manner, using the fetched service
+        fetched_ports = [(p.port, p.targetPort) for p in service.spec.ports]  # type: ignore[attr-defined]  # noqa: E501
+        return expected_ports == fetched_ports
+
+    @property
+    def _app(self) -> str:
+        """Name of the current Juju application.
+
+        Returns:
+            str: A string containing the name of the current Juju application.
+        """
+        return self.charm.app.name
+
+    @property
+    def _namespace(self) -> str:
+        """The Kubernetes namespace we're running in.
+
+        Returns:
+            str: A string containing the name of the current Kubernetes namespace.
+        """
+        with open("/var/run/secrets/kubernetes.io/serviceaccount/namespace", "r") as f:
+            return f.read().strip()
diff --git a/lib/charms/observability_libs/v1/cert_handler.py b/lib/charms/observability_libs/v1/cert_handler.py
index 1375550..f6a3eda 100644
--- a/lib/charms/observability_libs/v1/cert_handler.py
+++ b/lib/charms/observability_libs/v1/cert_handler.py
@@ -31,10 +31,12 @@
 
 Since this library uses [Juju Secrets](https://juju.is/docs/juju/secret) it requires Juju >= 3.0.3.
 """
+import abc
 import ipaddress
+import json
 import socket
 from itertools import filterfalse
-from typing import List, Optional, Union
+from typing import Dict, List, Optional, Union
 
 try:
     from charms.tls_certificates_interface.v3.tls_certificates import (  # type: ignore
@@ -42,13 +44,14 @@
         CertificateAvailableEvent,
         CertificateExpiringEvent,
         CertificateInvalidatedEvent,
+        ProviderCertificate,
         TLSCertificatesRequiresV3,
         generate_csr,
         generate_private_key,
     )
 except ImportError as e:
     raise ImportError(
-        "failed to import charms.tls_certificates_interface.v2.tls_certificates; "
+        "failed to import charms.tls_certificates_interface.v3.tls_certificates; "
         "Either the library itself is missing (please get it through charmcraft fetch-lib) "
         "or one of its dependencies is unmet."
     ) from e
@@ -58,14 +61,15 @@
 from ops.charm import CharmBase, RelationBrokenEvent
 from ops.framework import EventBase, EventSource, Object, ObjectEvents
 from ops.jujuversion import JujuVersion
-from ops.model import SecretNotFoundError
+from ops.model import Relation, Secret, SecretNotFoundError
 
 logger = logging.getLogger(__name__)
 
-
 LIBID = "b5cd5cd580f3428fa5f59a8876dcbe6a"
 LIBAPI = 1
-LIBPATCH = 4
+LIBPATCH = 9
+
+VAULT_SECRET_LABEL = "cert-handler-private-vault"
 
 
 def is_ip_address(value: str) -> bool:
@@ -87,6 +91,178 @@ class CertHandlerEvents(ObjectEvents):
     cert_changed = EventSource(CertChanged)
 
 
+class _VaultBackend(abc.ABC):
+    """Base class for a single secret manager.
+
+    Assumptions:
+    - A single secret (label) is managed by a single instance.
+    - Secret is per-unit (not per-app, i.e. may differ from unit to unit).
+    """
+
+    def store(self, contents: Dict[str, str], clear: bool = False): ...
+
+    def get_value(self, key: str) -> Optional[str]: ...
+
+    def retrieve(self) -> Dict[str, str]: ...
+
+    def clear(self): ...
+
+
+class _RelationVaultBackend(_VaultBackend):
+    """Relation backend for Vault.
+
+    Use it to store data in a relation databag.
+    Assumes that a single relation exists and its data is readable.
+    If not, it will raise RuntimeErrors as soon as you try to read/write.
+    It will store the data, in plaintext (json-dumped) nested under a configurable
+    key in the **unit databag** of this relation.
+
+    Typically, you'll use this with peer relations.
+
+    Note: it is assumed that this object has exclusive access to the data, even though in practice it does not.
+      Modifying relation data yourself would go unnoticed and disrupt consistency.
+    """
+
+    _NEST_UNDER = "lib.charms.observability_libs.v1.cert_handler::vault"
+    # This key needs to be relation-unique. If someone ever creates multiple Vault(_RelationVaultBackend)
+    # instances backed by the same (peer) relation, they'll need to set different _NEST_UNDERs
+    # for each _RelationVaultBackend instance or they'll be fighting over it.
+
+    def __init__(self, charm: CharmBase, relation_name: str):
+        self.charm = charm
+        self.relation_name = relation_name
+
+    def _check_ready(self):
+        relation = self.charm.model.get_relation(self.relation_name)
+        if not relation or not relation.data:
+            # if something goes wrong here, the peer-backed vault is not ready to operate
+            # it can be because you are trying to use it too soon, i.e. before the (peer)
+            # relation has been created (or has joined).
+            raise RuntimeError("Relation backend not ready.")
+
+    @property
+    def _relation(self) -> Optional[Relation]:
+        self._check_ready()
+        return self.charm.model.get_relation(self.relation_name)
+
+    @property
+    def _databag(self):
+        self._check_ready()
+        # _check_ready verifies that there is a relation
+        return self._relation.data[self.charm.unit]  # type: ignore
+
+    def _read(self) -> Dict[str, str]:
+        value = self._databag.get(self._NEST_UNDER)
+        if value:
+            return json.loads(value)
+        return {}
+
+    def _write(self, value: Dict[str, str]):
+        if not all(isinstance(x, str) for x in value.values()):
+            # the caller has to take care of encoding
+            raise TypeError("You can only store strings in Vault.")
+
+        self._databag[self._NEST_UNDER] = json.dumps(value)
+
+    def store(self, contents: Dict[str, str], clear: bool = False):
+        """Create a new revision by updating the previous one with ``contents``."""
+        current = self._read()
+
+        if clear:
+            current.clear()
+
+        current.update(contents)
+        self._write(current)
+
+    def get_value(self, key: str) -> Optional[str]:
+        """Like retrieve, but single-value."""
+        return self._read().get(key)
+
+    def retrieve(self):
+        """Return the full vault content."""
+        return self._read()
+
+    def clear(self):
+        del self._databag[self._NEST_UNDER]
+
+
+class _SecretVaultBackend(_VaultBackend):
+    """Relation backend for Vault.
+
+    Use it to store data in a Juju secret.
+    Assumes that Juju supports secrets.
+    If not, it will raise some exception as soon as you try to read/write.
+
+    Note: it is assumed that this object has exclusive access to the data, even though in practice it does not.
+      Modifying secret's data yourself would go unnoticed and disrupt consistency.
+    """
+
+    _uninitialized_key = "uninitialized-secret-key"
+
+    def __init__(self, charm: CharmBase, secret_label: str):
+        self.charm = charm
+        self.secret_label = secret_label  # needs to be charm-unique.
+
+    @property
+    def _secret(self) -> Secret:
+        try:
+            # we are owners, so we don't need to grant it to ourselves
+            return self.charm.model.get_secret(label=self.secret_label)
+        except SecretNotFoundError:
+            # we need to set SOME contents when we're creating the secret, so we do it.
+            return self.charm.unit.add_secret(
+                {self._uninitialized_key: "42"}, label=self.secret_label
+            )
+
+    def store(self, contents: Dict[str, str], clear: bool = False):
+        """Create a new revision by updating the previous one with ``contents``."""
+        secret = self._secret
+        current = secret.get_content(refresh=True)
+
+        if clear:
+            current.clear()
+        elif current.get(self._uninitialized_key):
+            # is this the first revision? clean up the mock contents we created instants ago.
+            del current[self._uninitialized_key]
+
+        current.update(contents)
+        secret.set_content(current)
+
+    def get_value(self, key: str) -> Optional[str]:
+        """Like retrieve, but single-value."""
+        return self._secret.get_content(refresh=True).get(key)
+
+    def retrieve(self):
+        """Return the full vault content."""
+        return self._secret.get_content(refresh=True)
+
+    def clear(self):
+        self._secret.remove_all_revisions()
+
+
+class Vault:
+    """Simple application secret wrapper for local usage."""
+
+    def __init__(self, backend: _VaultBackend):
+        self._backend = backend
+
+    def store(self, contents: Dict[str, str], clear: bool = False):
+        """Store these contents in the vault overriding whatever is there."""
+        self._backend.store(contents, clear=clear)
+
+    def get_value(self, key: str):
+        """Like retrieve, but single-value."""
+        return self._backend.get_value(key)
+
+    def retrieve(self) -> Dict[str, str]:
+        """Return the full vault content."""
+        return self._backend.retrieve()
+
+    def clear(self):
+        """Clear the vault."""
+        self._backend.clear()
+
+
 class CertHandler(Object):
     """A wrapper for the requirer side of the TLS Certificates charm library."""
 
@@ -114,9 +290,8 @@ def __init__(
             sans: DNS names. If none are given, use FQDN.
         """
         super().__init__(charm, key)
-        self._check_juju_supports_secrets()
-
         self.charm = charm
+
         # We need to sanitize the unit name, otherwise route53 complains:
         # "urn:ietf:params:acme:error:malformed" :: Domain name contains an invalid character
         self.cert_subject = charm.unit.name.replace("/", "-") if not cert_subject else cert_subject
@@ -126,6 +301,17 @@ def __init__(
         self.sans_ip = list(filter(is_ip_address, sans))
         self.sans_dns = list(filterfalse(is_ip_address, sans))
 
+        if self._check_juju_supports_secrets():
+            vault_backend = _SecretVaultBackend(charm, secret_label=VAULT_SECRET_LABEL)
+
+            # TODO: gracefully handle situations where the
+            #  secret is gone because the admin has removed it manually
+            # self.framework.observe(self.charm.on.secret_remove, self._rotate_csr)
+
+        else:
+            vault_backend = _RelationVaultBackend(charm, relation_name="peers")
+        self.vault = Vault(vault_backend)
+
         self.certificates_relation_name = certificates_relation_name
         self.certificates = TLSCertificatesRequiresV3(self.charm, self.certificates_relation_name)
 
@@ -157,58 +343,97 @@ def __init__(
             self.charm.on[self.certificates_relation_name].relation_broken,  # pyright: ignore
             self._on_certificates_relation_broken,
         )
+        self.framework.observe(
+            self.charm.on.upgrade_charm,  # pyright: ignore
+            self._on_upgrade_charm,
+        )
+
+    def _on_upgrade_charm(self, _):
+        has_privkey = self.vault.get_value("private-key")
+
+        self._migrate_vault()
+
+        # If we already have a csr, but the pre-migration vault has no privkey stored,
+        # the csr must have been signed with a privkey that is now outdated and utterly lost.
+        # So we throw away the csr and generate a new one (and a new privkey along with it).
+        if not has_privkey and self._csr:
+            logger.debug("CSR and privkey out of sync after charm upgrade. Renewing CSR.")
+            # this will call `self.private_key` which will generate a new privkey.
+            self._generate_csr(renew=True)
+
+    def _migrate_vault(self):
+        peer_backend = _RelationVaultBackend(self.charm, relation_name="peers")
+
+        if self._check_juju_supports_secrets():
+            # we are on recent juju
+            if self.vault.retrieve():
+                # we already were on recent juju: nothing to migrate
+                logger.debug(
+                    "Private key is already stored as a juju secret. Skipping private key migration."
+                )
+                return
+
+            # we used to be on old juju: our secret stuff is in peer data
+            if contents := peer_backend.retrieve():
+                logger.debug(
+                    "Private key found in relation data. "
+                    "Migrating private key to a juju secret."
+                )
+                # move over to secret-backed storage
+                self.vault.store(contents)
+
+                # clear the peer storage
+                peer_backend.clear()
+                return
+
+        # if we are downgrading, i.e. from juju with secrets to juju without,
+        # we have lost all that was in the secrets backend.
 
     @property
     def enabled(self) -> bool:
-        """Boolean indicating whether the charm has a tls_certificates relation."""
+        """Boolean indicating whether the charm has a tls_certificates relation.
+
+        See also the `available` property.
+        """
         # We need to check for units as a temporary workaround because of https://bugs.launchpad.net/juju/+bug/2024583
         # This could in theory not work correctly on scale down to 0 but it is necessary for the moment.
 
-        if not self.charm.model.get_relation(self.certificates_relation_name):
+        if not self.relation:
             return False
 
-        if not self.charm.model.get_relation(
-            self.certificates_relation_name
-        ).units:  # pyright: ignore
+        if not self.relation.units:  # pyright: ignore
             return False
 
-        if not self.charm.model.get_relation(
-            self.certificates_relation_name
-        ).app:  # pyright: ignore
+        if not self.relation.app:  # pyright: ignore
             return False
 
-        if not self.charm.model.get_relation(
-            self.certificates_relation_name
-        ).data:  # pyright: ignore
+        if not self.relation.data:  # pyright: ignore
             return False
 
         return True
 
+    @property
+    def available(self) -> bool:
+        """Return True if all certs are available in relation data; False otherwise."""
+        return (
+            self.enabled
+            and self.server_cert is not None
+            and self.private_key is not None
+            and self.ca_cert is not None
+        )
+
     def _on_certificates_relation_joined(self, _) -> None:
-        self._generate_privkey()
+        # this will only generate a csr if we don't have one already
         self._generate_csr()
 
-    def _generate_privkey(self):
-        # Generate priv key unless done already
-        # TODO figure out how to go about key rotation.
-
-        if not (relation := self.charm.model.get_relation(self.certificates_relation_name)):
-            return
-
-        if not self.private_key:
-            private_key = generate_private_key()
-            secret = self.charm.unit.add_secret({"private-key": private_key.decode()})
-            secret.grant(relation)
-            relation.data[self.charm.unit]["private-key-secret-id"] = secret.id  # pyright: ignore
-
     def _on_config_changed(self, _):
-        relation = self.charm.model.get_relation(self.certificates_relation_name)
-
-        if not relation:
-            return
+        # this will only generate a csr if we don't have one already
+        self._generate_csr()
 
-        self._generate_privkey()
-        self._generate_csr(renew=True)
+    @property
+    def relation(self):
+        """The "certificates" relation."""
+        return self.charm.model.get_relation(self.certificates_relation_name)
 
     def _generate_csr(
         self, overwrite: bool = False, renew: bool = False, clear_cert: bool = False
@@ -221,15 +446,16 @@ def _generate_csr(
 
         This method intentionally does not emit any events, leave it for caller's responsibility.
         """
+        # if we are in a relation-broken hook, we might not have a relation to publish the csr to.
+        if not self.relation:
+            logger.warning(
+                f"No {self.certificates_relation_name!r} relation found. " f"Cannot generate csr."
+            )
+            return
+
         # In case we already have a csr, do not overwrite it by default.
         if overwrite or renew or not self._csr:
             private_key = self.private_key
-            if private_key is None:
-                # FIXME: raise this in a less nested scope by
-                #  generating privkey and csr in the same method.
-                raise RuntimeError(
-                    "private key unset. call _generate_privkey() before you call this method."
-                )
             csr = generate_csr(
                 private_key=private_key.encode(),
                 subject=self.cert_subject,
@@ -251,119 +477,73 @@ def _generate_csr(
                 )
                 self.certificates.request_certificate_creation(certificate_signing_request=csr)
 
-            # Note: CSR is being replaced with a new one, so until we get the new cert, we'd have
-            # a mismatch between the CSR and the cert.
-            # For some reason the csr contains a trailing '\n'. TODO figure out why
-            self._csr = csr.decode().strip()
-
         if clear_cert:
-            try:
-                secret = self.model.get_secret(label="ca-certificate-chain")
-                secret.remove_all_revisions()
-            except SecretNotFoundError:
-                logger.debug("Secret with label: 'ca-certificate-chain' not found")
+            self.vault.clear()
 
     def _on_certificate_available(self, event: CertificateAvailableEvent) -> None:
-        """Get the certificate from the event and store it in a peer relation.
-
-        Note: assuming "limit: 1" in metadata
-        """
-        event_csr = (
-            event.certificate_signing_request.strip()
-            if event.certificate_signing_request
-            else None
-        )
-        if event_csr == self._csr:
-            content = {
-                "ca-cert": event.ca,
-                "server-cert": event.certificate,
-                "chain": event.chain_as_pem(),
-                "csr": event_csr,
-            }
-            try:
-                secret = self.model.get_secret(label="ca-certificate-chain")
-            except SecretNotFoundError:
-                if not (
-                    relation := self.charm.model.get_relation(self.certificates_relation_name)
-                ):
-                    logger.error("Relation %s not found", self.certificates_relation_name)
-                    return
-
-                secret = self.charm.unit.add_secret(content, label="ca-certificate-chain")
-                secret.grant(relation)
-                relation.data[self.charm.unit]["secret-id"] = secret.id  # pyright: ignore
-                self.on.cert_changed.emit()  # pyright: ignore
-
-    def _retrieve_secret_id(self, secret_id_name: str) -> Optional[str]:
-        if not (relation := self.charm.model.get_relation(self.certificates_relation_name)):
-            return None
-
-        if not (secret_id := relation.data[self.charm.unit].get(secret_id_name)):
-            return None
-
-        return secret_id
-
-    def _retrieve_from_secret(self, value: str, secret_id_name: str) -> Optional[str]:
-        if not (secret_id := self._retrieve_secret_id(secret_id_name)):
-            return None
-
-        if not (secret := self.model.get_secret(id=secret_id)):
-            return None
-
-        content = secret.get_content()
-        return content.get(value)
+        """Emit cert-changed."""
+        self.on.cert_changed.emit()  # pyright: ignore
 
     @property
-    def private_key(self) -> Optional[str]:
-        """Private key."""
-        return self._retrieve_from_secret("private-key", "private-key-secret-id")
+    def private_key(self) -> str:
+        """Private key.
 
-    @property
-    def private_key_secret_id(self) -> Optional[str]:
-        """ID of the Juju Secret for the Private key."""
-        return self._retrieve_secret_id("private-key-secret-id")
+        BEWARE: if the vault misbehaves, the backing secret is removed, the peer relation dies
+        or whatever, we might be calling generate_private_key() again and cause a desync
+        with the CSR because it's going to be signed with an outdated key we have no way of retrieving.
+        The caller needs to ensure that if the vault backend gets reset, then so does the csr.
+
+        TODO: we could consider adding a way to verify if the csr was signed by our privkey,
+            and do that on collect_unit_status as a consistency check
+        """
+        private_key = self.vault.get_value("private-key")
+        if private_key is None:
+            private_key = generate_private_key().decode()
+            self.vault.store({"private-key": private_key})
+        return private_key
 
     @property
     def _csr(self) -> Optional[str]:
-        return self._retrieve_from_secret("csr", "csr-secret-id")
+        csrs = self.certificates.get_requirer_csrs()
+        if not csrs:
+            return None
 
-    @_csr.setter
-    def _csr(self, value: str):
-        if not (relation := self.charm.model.get_relation(self.certificates_relation_name)):
-            return
+        # in principle we only ever need one cert.
+        # we might want to complicate this a bit once we get into cert rotations: during the rotation, we may need to
+        # keep the old one around for a little while. If there's multiple certs, at the moment we're
+        # ignoring all but the last one.
+        if len(csrs) > 1:
+            logger.warning(
+                "Multiple CSRs found in `certificates` relation. "
+                "cert_handler is not ready to expect it."
+            )
 
-        if not (secret_id := relation.data[self.charm.unit].get("csr-secret-id", None)):
-            secret = self.charm.unit.add_secret({"csr": value})
-            secret.grant(relation)
-            relation.data[self.charm.unit]["csr-secret-id"] = secret.id  # pyright: ignore
-            return
+        return csrs[-1].csr
 
-        secret = self.model.get_secret(id=secret_id)
-        secret.set_content({"csr": value})
+    def get_cert(self) -> Optional[ProviderCertificate]:
+        """Get the certificate from relation data."""
+        all_certs = self.certificates.get_provider_certificates()
+        # search for the cert matching our csr.
+        matching_cert = [c for c in all_certs if c.csr == self._csr]
+        return matching_cert[0] if matching_cert else None
 
     @property
     def ca_cert(self) -> Optional[str]:
         """CA Certificate."""
-        return self._retrieve_from_secret("ca-cert", "secret-id")
-
-    @property
-    def ca_server_cert_secret_id(self) -> Optional[str]:
-        """CA server cert secret id."""
-        return self._retrieve_secret_id("secret-id")
+        cert = self.get_cert()
+        return cert.ca if cert else None
 
     @property
     def server_cert(self) -> Optional[str]:
         """Server Certificate."""
-        return self._retrieve_from_secret("server-cert", "secret-id")
-
-    @property
-    def _chain(self) -> Optional[str]:
-        return self._retrieve_from_secret("chain", "secret-id")
+        cert = self.get_cert()
+        return cert.certificate if cert else None
 
     @property
     def chain(self) -> Optional[str]:
-        """Return the ca chain."""
-        return self._chain
+        """Return the ca chain bundled as a single PEM string."""
+        cert = self.get_cert()
+        return cert.chain_as_pem() if cert else None
 
     def _on_certificate_expiring(
         self, event: Union[CertificateExpiringEvent, CertificateInvalidatedEvent]
@@ -371,6 +551,7 @@ def _on_certificate_expiring(
         """Generate a new CSR and request certificate renewal."""
         if event.certificate == self.server_cert:
             self._generate_csr(renew=True)
+            # FIXME why are we not emitting cert_changed here?
 
     def _certificate_revoked(self, event) -> None:
         """Remove the certificate and generate a new CSR."""
@@ -381,13 +562,11 @@ def _certificate_revoked(self, event) -> None:
 
     def _on_certificate_invalidated(self, event: CertificateInvalidatedEvent) -> None:
         """Deal with certificate revocation and expiration."""
-        if event.certificate != self.server_cert:
-            return
-
-        # if event.reason in ("revoked", "expired"):
-        # Currently, the reason does not matter to us because the action is the same.
-        self._generate_csr(overwrite=True, clear_cert=True)
-        self.on.cert_changed.emit()  # pyright: ignore
+        if event.certificate == self.server_cert:
+            # if event.reason in ("revoked", "expired"):
+            # Currently, the reason does not matter to us because the action is the same.
+            self._generate_csr(overwrite=True, clear_cert=True)
+            self.on.cert_changed.emit()  # pyright: ignore
 
     def _on_all_certificates_invalidated(self, _: AllCertificatesInvalidatedEvent) -> None:
         # Do what you want with this information, probably remove all certificates
@@ -396,18 +575,14 @@ def _on_all_certificates_invalidated(self, _: AllCertificatesInvalidatedEvent) -
         self.on.cert_changed.emit()  # pyright: ignore
 
     def _on_certificates_relation_broken(self, _: RelationBrokenEvent) -> None:
-        """Clear the certificates data when removing the relation."""
-        try:
-            secret = self.model.get_secret(label="csr-secret-id")
-            secret.remove_all_revisions()
-        except SecretNotFoundError:
-            logger.debug("Secret 'csr-scret-id' not found")
+        """Clear all secrets data when removing the relation."""
+        self.vault.clear()
         self.on.cert_changed.emit()  # pyright: ignore
 
-    def _check_juju_supports_secrets(self) -> None:
+    def _check_juju_supports_secrets(self) -> bool:
         version = JujuVersion.from_environ()
-
         if not JujuVersion(version=str(version)).has_secrets:
             msg = f"Juju version {version} does not supports Secrets. Juju >= 3.0.3 is needed"
-            logger.error(msg)
-            raise RuntimeError(msg)
+            logger.debug(msg)
+            return False
+        return True
diff --git a/lib/charms/parca/v0/parca_scrape.py b/lib/charms/parca/v0/parca_scrape.py
new file mode 100644
index 0000000..82801f6
--- /dev/null
+++ b/lib/charms/parca/v0/parca_scrape.py
@@ -0,0 +1,809 @@
+# Copyright 2022 Jon Seager
+# See LICENSE file for licensing details.
+"""## Overview.
+
+This document explains how to integrate with the Parca charm for the purpose of providing a
+profiling endpoint to Parca. It also explains how alternative implementations of the Parca charms
+may maintain the same interface and be backward compatible with all currently integrated charms.
+
+## Provider Library Usage
+
+This Parca charm interacts with its scrape targets using its charm library. Charms seeking to
+expose profiling endpoints for the Parca charm, must do so using the `ProfilingEndpointProvider`
+object from this charm library. For the simplest use cases, using the `ProfilingEndpointProvider`
+object only requires instantiating it, typically in the constructor of your charm. The
+`ProfilingEndpointProvider` constructor requires the name of the relation over which a scrape
+target (profiling endpoint) is exposed to the Parca charm. This relation must use the
+`parca_scrape` interface. By default address of the profiling endpoint is set to the unit IP
+address, by each unit of the `ProfilingEndpointProvider` charm. These units set their address in
+response to the `PebbleReady` event of each container in the unit, since container restarts of
+Kubernetes charms can result in change of IP addresses. The default name for the profiling endpoint
+relation is `profiling-endpoint`. It is strongly recommended to use the same relation name for
+consistency across charms and doing so obviates the need for an additional constructor argument.
+The `ProfilingEndpointProvider` object may be instantiated as follows
+
+    from charms.parca.v0.parca_scrape import ProfilingEndpointProvider
+
+    def __init__(self, *args):
+        super().__init__(*args)
+        # ...
+        self.profiling_endpoint = ProfilingEndpointProvider(self)
+        # ...
+
+Note that the first argument (`self`) to `ProfilingEndpointProvider` is always a reference to the
+parent (scrape target) charm.
+
+An instantiated `ProfilingEndpointProvider` object will ensure that each unit of its parent charm,
+is a scrape target for the `ProfilingEndpointConsumer` (Parca) charm. By default
+`ProfilingEndpointProvider` assumes each unit of the consumer charm exports its profiles on port
+80. These defaults may be changed by providing the `ProfilingEndpointProvider` constructor an
+optional argument (`jobs`) that represents a Parca scrape job specification using Python standard
+data structures. This job specification is a subset of Parca's own [scrape
+configuration](https://www.parca.dev/docs/configuration) format but represented using Python data
+structures. More than one job may be provided using the `jobs` argument. Hence `jobs` accepts a
+list of dictionaries where each dictionary represents one `<scrape_config>` object as described in
+the Parca documentation. The currently supported configuration subset is: `job_name`,
+`static_configs`
+
+Suppose it is required to change the port on which scraped profiles are exposed to 8000. This may be
+done by providing the following data structure as the value of `jobs`.
+
+```python
+[{"static_configs": [{"targets": ["*:8000"]}]}]
+```
+
+The wildcard ("*") host specification implies that the scrape targets will automatically be set to
+the host addresses advertised by each unit of the consumer charm.
+
+It is also possible to change the profile path and scrape multiple ports, for example
+
+```
+[{"static_configs": [{"targets": ["*:8000", "*:8081"]}]}]
+```
+
+More complex scrape configurations are possible. For example
+
+```
+[{
+    "static_configs": [{
+        "targets": ["10.1.32.215:7000", "*:8000"],
+        "labels": {
+            "some-key": "some-value"
+        }
+    }]
+}]
+```
+
+This example scrapes the target "10.1.32.215" at port 7000 in addition to scraping each unit at
+port 8000. There is however one difference between wildcard targets (specified using "*") and fully
+qualified targets (such as "10.1.32.215"). The Parca charm automatically associates labels with
+profiles generated by each target. These labels localise the source of profiles within the Juju
+topology by specifying its "model name", "model UUID", "application name" and "unit name". However
+unit name is associated only with wildcard targets but not with fully qualified targets.
+
+Multiple jobs with labels are allowed, but each job must be given a unique name:
+
+```
+[
+    {
+        "job_name": "my-first-job",
+        "static_configs": [
+            {
+                "targets": ["*:7000"],
+                "labels": {
+                    "some-key": "some-value"
+                }
+            }
+        ]
+    },
+    {
+        "job_name": "my-second-job",
+        "static_configs": [
+            {
+                "targets": ["*:8000"],
+                "labels": {
+                    "some-other-key": "some-other-value"
+                }
+            }
+        ]
+    }
+]
+```
+
+**Important:** `job_name` should be a fixed string (e.g. hardcoded literal). For instance, if you
+include variable elements, like your `unit.name`, it may break the continuity of the profile time
+series gathered by Parca when the leader unit changes (e.g. on upgrade or rescale).
+
+## Consumer Library Usage
+
+The `ProfilingEndpointConsumer` object may be used by Parca charms to manage relations with their
+scrape targets. For this purposes a Parca charm needs to do two things
+
+1. Instantiate the `ProfilingEndpointConsumer` object by providing it a
+reference to the parent (Parca) charm and optionally the name of the relation that the Parca charm
+uses to interact with scrape targets. This relation must confirm to the `parca_scrape` interface
+and it is strongly recommended that this relation be named `profiling-endpoint` which is its
+default value.
+
+For example a Parca charm may instantiate the `ProfilingEndpointConsumer` in its constructor as
+follows
+
+    from charms.parca.v0.parca_scrape import ProfilingEndpointConsumer
+
+    def __init__(self, *args):
+        super().__init__(*args)
+        # ...
+        self.profiling_consumer = ProfilingEndpointConsumer(self)
+        # ...
+
+2. A Parca charm also needs to respond to the `TargetsChangedEvent` event of the
+`ProfilingEndpointConsumer` by adding itself as an observer for these events, as in
+
+    self.framework.observe(
+        self.profiling_consumer.on.targets_changed,
+        self._on_scrape_targets_changed,
+    )
+
+In responding to the `TargetsChangedEvent` event the Parca charm must update the Parca
+configuration so that any new scrape targets are added and/or old ones removed from the list of
+scraped endpoints. For this purpose the `ProfilingEndpointConsumer` object exposes a `jobs()`
+method that returns a list of scrape jobs. Each element of this list is the Parca scrape
+configuration for that job. In order to update the Parca configuration, the Parca charm needs to
+replace the current list of jobs with the list provided by `jobs()` as follows
+
+    def _on_scrape_targets_changed(self, event):
+        ...
+        scrape_jobs = self.profiling_consumer.jobs()
+        for job in scrape_jobs:
+            parca_scrape_config.append(job)
+        ...
+
+## Relation Data
+
+Units of profiles provider charms advertise their names and addresses over unit relation data using
+the `parca_scrape_unit_name` and `parca_scrape_unit_address` keys. While the `scrape_metadata`,
+`scrape_jobs` and `alert_rules` keys in application relation data of profiles provider charms hold
+eponymous information.
+
+"""  # noqa: W505
+
+import ipaddress
+import json
+import logging
+import socket
+from typing import List, Optional, Union
+
+import ops
+from charms.observability_libs.v0.juju_topology import JujuTopology
+
+# The unique Charmhub library identifier, never change it
+LIBID = "7b30b495435746acb645ca414898621f"
+
+# Increment this major API version when introducing breaking changes
+LIBAPI = 0
+
+# Increment this PATCH version before using `charmcraft publish-lib` or reset
+# to 0 if you are raising the major API version
+LIBPATCH = 5
+
+logger = logging.getLogger(__name__)
+
+
+ALLOWED_KEYS = {"job_name", "static_configs", "scrape_interval", "scrape_timeout"}
+DEFAULT_JOB = {"static_configs": [{"targets": ["*:80"]}]}
+DEFAULT_RELATION_NAME = "profiling-endpoint"
+RELATION_INTERFACE_NAME = "parca_scrape"
+
+
+class RelationNotFoundError(Exception):
+    """Raise if there is no relation with the given name is found."""
+
+    def __init__(self, relation_name: str):
+        self.relation_name = relation_name
+        self.message = "No relation named '{}' found".format(relation_name)
+        super().__init__(self.message)
+
+
+class RelationInterfaceMismatchError(Exception):
+    """Raise if the relation with the given name has a different interface."""
+
+    def __init__(
+        self,
+        relation_name: str,
+        expected_relation_interface: str,
+        actual_relation_interface: str,
+    ):
+        self.relation_name = relation_name
+        self.expected_relation_interface = expected_relation_interface
+        self.actual_relation_interface = actual_relation_interface
+        self.message = (
+            "The '{}' relation has '{}' as interface rather than the expected '{}'".format(
+                relation_name, actual_relation_interface, expected_relation_interface
+            )
+        )
+
+        super().__init__(self.message)
+
+
+class RelationRoleMismatchError(Exception):
+    """Raise if the relation with the given name has a different role."""
+
+    def __init__(
+        self,
+        relation_name: str,
+        expected_relation_role: ops.RelationRole,
+        actual_relation_role: ops.RelationRole,
+    ):
+        self.relation_name = relation_name
+        self.expected_relation_interface = expected_relation_role
+        self.actual_relation_role = actual_relation_role
+        self.message = "The '{}' relation has role '{}' rather than the expected '{}'".format(
+            relation_name, repr(actual_relation_role), repr(expected_relation_role)
+        )
+
+        super().__init__(self.message)
+
+
+def _validate_relation_by_interface_and_direction(
+    charm: ops.CharmBase,
+    relation_name: str,
+    expected_relation_interface: str,
+    expected_relation_role: ops.RelationRole,
+):
+    """Verify that a relation has the necessary characteristics.
+
+    Verifies that the `relation_name` provided: (1) exists in metadata.yaml,
+    (2) declares as interface the interface name passed as `relation_interface`
+    and (3) has the right "direction", i.e., it is a relation that `charm`
+    provides or requires.
+
+    Args:
+        charm: a `ops.CharmBase` object to scan for the matching relation.
+        relation_name: the name of the relation to be verified.
+        expected_relation_interface: the interface name to be matched by the
+            relation named `relation_name`.
+        expected_relation_role: whether the `relation_name` must be either
+            provided or required by `charm`.
+
+    Raises:
+        RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+            with the same name as provided via `relation_name` argument.
+        RelationInterfaceMismatchError: The relation with the same name as provided
+            via `relation_name` argument does not have the same relation interface
+            as specified via the `expected_relation_interface` argument.
+        ops.RelationRoleMismatchError: If the relation with the same name as provided
+            via `relation_name` argument does not have the same role as specified
+            via the `expected_relation_role` argument.
+    """
+    if relation_name not in charm.meta.relations:
+        raise RelationNotFoundError(relation_name)
+
+    relation = charm.meta.relations[relation_name]
+
+    actual_relation_interface = relation.interface_name
+    if actual_relation_interface != expected_relation_interface:
+        raise RelationInterfaceMismatchError(
+            relation_name, expected_relation_interface, actual_relation_interface
+        )
+
+    if expected_relation_role == ops.RelationRole.provides:
+        if relation_name not in charm.meta.provides:
+            raise ops.RelationRoleMismatchError(
+                relation_name, ops.RelationRole.provides, ops.RelationRole.requires
+            )
+    elif expected_relation_role == ops.RelationRole.requires:
+        if relation_name not in charm.meta.requires:
+            raise ops.RelationRoleMismatchError(
+                relation_name, ops.RelationRole.requires, ops.RelationRole.provides
+            )
+    else:
+        raise Exception("Unexpected RelationDirection: {}".format(expected_relation_role))
+
+
+def _sanitize_scrape_configuration(job) -> dict:
+    """Restrict permissible scrape configuration options.
+
+    If job is empty then a default job is returned. The default job is:
+
+    ```python
+    {"static_configs": [{"targets": ["*:80"]}]}
+    ```
+
+    Args:
+        job: a dict containing a single Parca job specification.
+
+    Returns:
+        a dictionary containing a sanitized job specification.
+    """
+    sanitized_job = DEFAULT_JOB.copy()
+    sanitized_job.update({key: value for key, value in job.items() if key in ALLOWED_KEYS})
+    return sanitized_job
+
+
+class ProviderTopology(JujuTopology):
+    """Class for initializing topology information for ProfilingEndpointProvider."""
+
+    @property
+    def scrape_identifier(self):
+        """Format the topology information into a scrape identifier."""
+        # This is used only by Profiling[Consumer|Provider] and does not need a
+        # unit name, so only check for the charm name
+        return "juju_{}_parca_scrape".format(self.identifier)
+
+
+class TargetsChangedEvent(ops.EventBase):
+    """Event emitted when Parca scrape targets change."""
+
+    def __init__(self, handle, relation_id):
+        super().__init__(handle)
+        self.relation_id = relation_id
+
+    def snapshot(self):
+        """Save scrape target relation information."""
+        return {"relation_id": self.relation_id}
+
+    def restore(self, snapshot):
+        """Restore scrape target relation information."""
+        self.relation_id = snapshot["relation_id"]
+
+
+class MonitoringEvents(ops.ObjectEvents):
+    """Event descriptor for events raised by `ProfilingEndpointConsumer`."""
+
+    targets_changed = ops.EventSource(TargetsChangedEvent)
+
+
+class ProfilingEndpointConsumer(ops.Object):
+    """Parca based monitoring service."""
+
+    on = MonitoringEvents()
+
+    def __init__(self, charm: ops.CharmBase, relation_name: str = DEFAULT_RELATION_NAME):
+        """Construct a Parca based monitoring service.
+
+        Args:
+            charm: a `ops.CharmBase` instance that manages this instance of the Parca service.
+            relation_name: an optional string name of the relation between `charm`
+                and the Parca charmed service. The default is "profiling-endpoint".
+
+        Raises:
+            RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+                with the same name as provided via `relation_name` argument.
+            RelationInterfaceMismatchError: The relation with the same name as provided
+                via `relation_name` argument does not have the `parca_scrape` relation
+                interface.
+            ops.RelationRoleMismatchError: If the relation with the same name as provided
+                via `relation_name` argument does not have the `ops.RelationRole.requires`
+                role.
+        """
+        _validate_relation_by_interface_and_direction(
+            charm, relation_name, RELATION_INTERFACE_NAME, ops.RelationRole.requires
+        )
+
+        super().__init__(charm, relation_name)
+        self._charm = charm
+        self._relation_name = relation_name
+        events = self._charm.on[relation_name]
+        self.framework.observe(
+            events.relation_changed, self.on_profiling_provider_relation_changed
+        )
+        self.framework.observe(
+            events.relation_departed, self._on_profiling_provider_relation_departed
+        )
+
+    def on_profiling_provider_relation_changed(self, event):
+        """Handle changes with related profiling providers.
+
+        Anytime there are changes in relations between Parca and profiling provider charms the
+        Parca charm is informed, through a `TargetsChangedEvent` event. The Parca charm can then
+        choose to update its scrape configuration.
+
+        Args:
+            event: a `CharmEvent` resulting in the Parca charm updating its scrape configuration
+        """
+        rel_id = event.relation.id
+
+        self.on.targets_changed.emit(relation_id=rel_id)
+
+    def _on_profiling_provider_relation_departed(self, event):
+        """Update job config when a profiling provider departs.
+
+        When a profiling provider departs the Parca charm is informed through a
+        `TargetsChangedEvent` event so that it can update its scrape configuration to ensure that
+        the departed profiling provider is removed from the list of scrape jobs.
+
+        Args:
+            event: a `CharmEvent` that indicates a profiling provider unit has departed.
+        """
+        rel_id = event.relation.id
+        self.on.targets_changed.emit(relation_id=rel_id)
+
+    def jobs(self) -> list:
+        """Fetch the list of scrape jobs.
+
+        Returns:
+            A list consisting of all the static scrape configurations for each related
+            `ProfilingEndpointProvider` that has specified its scrape targets.
+        """
+        scrape_jobs = []
+
+        for relation in self._charm.model.relations[self._relation_name]:
+            static_scrape_jobs = self._static_scrape_config(relation)
+            if static_scrape_jobs:
+                scrape_jobs.extend(static_scrape_jobs)
+
+        return scrape_jobs
+
+    def _static_scrape_config(self, relation) -> list:
+        """Generate the static scrape configuration for a single relation.
+
+        If the relation data includes `scrape_metadata` then the value of this key is used to
+        annotate the scrape jobs with Juju Topology labels before returning them.
+
+        Args:
+            relation: an `ops.model.Relation` object whose static scrape configuration is required.
+
+        Returns:
+            A list (possibly empty) of scrape jobs. Each job is a valid Parca scrape configuration
+            for that job, represented as a Python dictionary.
+        """
+        if not relation.units:
+            return []
+
+        scrape_jobs = json.loads(relation.data[relation.app].get("scrape_jobs", "[]"))
+
+        if not scrape_jobs:
+            return []
+
+        scrape_metadata = json.loads(relation.data[relation.app].get("scrape_metadata", "{}"))
+
+        if not scrape_metadata:
+            return scrape_jobs
+
+        job_name_prefix = JujuTopology.from_dict(scrape_metadata).identifier
+
+        hosts = self._relation_hosts(relation)
+
+        labeled_job_configs = []
+        for job in scrape_jobs:
+            config = self._labeled_static_job_config(
+                _sanitize_scrape_configuration(job),
+                job_name_prefix,
+                hosts,
+                scrape_metadata,
+            )
+            labeled_job_configs.append(config)
+
+        return labeled_job_configs
+
+    def _relation_hosts(self, relation) -> dict:
+        """Fetch unit names and address of all profiling provider units for a single relation.
+
+        Args:
+            relation: An `ops.model.Relation` object for which the unit name to
+                address mapping is required.
+
+        Returns:
+            A dictionary that maps unit names to unit addresses for the specified relation.
+        """
+        hosts = {}
+        for unit in relation.units:
+            # TODO deprecate and remove unit.name
+            unit_name = relation.data[unit].get("parca_scrape_unit_name") or unit.name
+            # TODO deprecate and remove "parca_scrape_host"
+            unit_address = relation.data[unit].get("parca_scrape_unit_address") or relation.data[
+                unit
+            ].get("parca_scrape_host")
+            if unit_name and unit_address:
+                hosts.update({unit_name: unit_address})
+
+        return hosts
+
+    def _labeled_static_job_config(self, job, job_name_prefix, hosts, scrape_metadata) -> dict:
+        """Construct labeled job configuration for a single job.
+
+        Args:
+            job: a dictionary representing the job configuration as obtained from
+                `ProfilingEndpointProvider` over relation data.
+            job_name_prefix: a string that may either be used as the
+                job name if the job has no associated name or used as a prefix for
+                the job if it does have a job name.
+            hosts: a dictionary mapping host names to host address for
+                all units of the relation for which this job configuration must be constructed.
+            scrape_metadata: scrape configuration metadata obtained
+                from `ProfilingEndpointProvider` from the same relation for
+                which this job configuration is being constructed.
+
+        Returns:
+            A dictionary representing a Parca job configuration for a single job.
+        """
+        name = job.get("job_name")
+        job_name = "{}_{}".format(job_name_prefix, name) if name else job_name_prefix
+
+        labeled_job = job.copy()
+        labeled_job["job_name"] = job_name
+
+        static_configs = job.get("static_configs")
+        labeled_job["static_configs"] = []
+
+        # relabel instance labels so that instance identifiers are globally unique
+        # stable over unit recreation
+        instance_relabel_config = {
+            "source_labels": ["juju_model", "juju_model_uuid", "juju_application"],
+            "separator": "_",
+            "target_label": "instance",
+            "regex": "(.*)",
+        }
+
+        # label all static configs in the Parca job labeling inserts Juju topology information and
+        # sets a relable config for instance labels
+        for static_config in static_configs:
+            labels = static_config.get("labels", {}) if static_configs else {}
+            all_targets = static_config.get("targets", [])
+
+            # split all targets into those which will have unit labels and those which will not
+            ports = []
+            unitless_targets = []
+            for target in all_targets:
+                host, port = target.split(":")
+                if host.strip() == "*":
+                    ports.append(port.strip())
+                else:
+                    unitless_targets.append(target)
+
+            # label scrape targets that do not have unit labels
+            if unitless_targets:
+                unitless_config = self._labeled_unitless_config(
+                    unitless_targets, labels, scrape_metadata
+                )
+                labeled_job["static_configs"].append(unitless_config)
+
+            # label scrape targets that do have unit labels
+            for host_name, host_address in hosts.items():
+                static_config = self._labeled_unit_config(
+                    host_name, host_address, ports, labels, scrape_metadata
+                )
+                labeled_job["static_configs"].append(static_config)
+                if "juju_unit" not in instance_relabel_config["source_labels"]:
+                    instance_relabel_config["source_labels"].append("juju_unit")  # type: ignore
+
+        # ensure topology relabeling of instance label is last in order of relabelings
+        relabel_configs = job.get("relabel_configs", [])
+        relabel_configs.append(instance_relabel_config)
+        labeled_job["relabel_configs"] = relabel_configs
+        return labeled_job
+
+    def _set_juju_labels(self, labels, scrape_metadata) -> dict:
+        """Create a copy of metric labels with Juju topology information.
+
+        Args:
+            labels: a dictionary containing Parca metric labels.
+            scrape_metadata: scrape related metadata provided by `ProfilingEndpointProvider`.
+
+        Returns:
+            a copy of the `labels` dictionary augmented with Juju topology information with the
+            exception of unit name.
+        """
+        juju_labels = labels.copy()  # deep copy not needed
+        juju_labels.update(ProviderTopology.from_dict(scrape_metadata).label_matcher_dict)
+
+        return juju_labels
+
+    def _labeled_unitless_config(self, targets, labels, scrape_metadata) -> dict:
+        """Return static scrape configuration for fully qualified host addresses.
+
+        Fully qualified hosts are those scrape targets for which the address are specified by the
+        `ProfilingEndpointProvider` as part of the scrape job specification set in application
+        relation data. The address specified need not belong to any unit of the
+        `ProfilingEndpointProvider` charm. As a result there is no reliable way to determine the
+        name (Juju topology unit name) for such a target.
+
+        Args:
+            targets: a list of addresses of fully qualified hosts.
+            labels: labels specified by `ProfilingEndpointProvider` clients which are associated
+                with `targets`.
+            scrape_metadata: scrape related metadata provided by `ProfilingEndpointProvider`.
+
+        Returns:
+            A dict containing the static scrape configuration for a list of fully qualified hosts.
+        """
+        juju_labels = self._set_juju_labels(labels, scrape_metadata)
+        unitless_config = {"targets": targets, "labels": juju_labels}
+        return unitless_config
+
+    def _labeled_unit_config(
+        self, unit_name, host_address, ports, labels, scrape_metadata
+    ) -> dict:
+        """Return static scrape configuration for a wildcard host.
+
+        Wildcard hosts are those scrape targets whose name (Juju unit name) and address (unit IP
+        address) is set into unit relation data by the `ProfilingEndpointProvider` charm, which
+        sets this data for ALL its units.
+
+        Args:
+            unit_name: a string representing the unit name of the wildcard host.
+            host_address: a string representing the address of the wildcard host.
+            ports: list of ports on which this wildcard host exposes its profiles.
+            labels: a dictionary of labels provided by `ProfilingEndpointProvider` intended to be
+                associated with this wildcard host.
+            scrape_metadata: scrape related metadata provided by `ProfilingEndpointProvider`.
+
+        Returns:
+            A dictionary containing the static scrape configuration
+            for a single wildcard host.
+        """
+        juju_labels = self._set_juju_labels(labels, scrape_metadata)
+
+        juju_labels["juju_unit"] = unit_name
+
+        static_config = {"labels": juju_labels}
+
+        if ports:
+            targets = []
+            for port in ports:
+                targets.append("{}:{}".format(host_address, port))
+            static_config["targets"] = targets  # type: ignore
+        else:
+            static_config["targets"] = [host_address]  # type: ignore
+
+        return static_config
+
+
+class ProfilingEndpointProvider(ops.Object):
+    """Profiling endpoint for Parca."""
+
+    def __init__(
+        self,
+        charm,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        jobs=None,
+        refresh_event: Optional[Union[ops.BoundEvent, List[ops.BoundEvent]]] = None,
+    ):
+        """Construct a profiling provider for a Parca charm.
+
+        If your charm exposes a Parca profiling endpoint, the `ProfilingEndpointProvider` object
+        enables your charm to easily communicate how to reach that endpoint.
+
+        By default, a charm instantiating this object has the profiling endpoints of each of its
+        units scraped by the related Parca charms.
+
+        The scraped profiles are automatically tagged by the Parca charms with Juju topology data
+        via the `juju_model_name`, `juju_model_uuid`, `juju_application_name` and `juju_unit`
+        labels. To support such tagging `ProfilingEndpointProvider` automatically forwards scrape
+        metadata to a `ProfilingEndpointConsumer` (Parca charm).
+
+        Scrape targets provided by `ProfilingEndpointProvider` can be customized when instantiating
+        this object. For example in the case of a charm exposing the profiling endpoint for each of
+        its units on port 8080, the `ProfilingEndpointProvider` can be
+        instantiated as follows:
+
+            self.profiling_endpoint_provider = ProfilingEndpointProvider(
+                self, jobs=[{"static_configs": [{"targets": ["*:8080"]}]}]
+            )
+
+        The notation `*:<port>` means "scrape each unit of this charm on port `<port>`.
+
+        Args:
+            charm: a `ops.CharmBase` object that manages this
+                `ProfilingEndpointProvider` object. Typically this is `self` in the instantiating
+                class.
+            relation_name: an optional string name of the relation between `charm`
+                and the Parca charmed service. The default is "profiling-endpoint". It is strongly
+                advised not to change the default, so that people deploying your charm will have a
+                consistent experience with all other charms that provide profiling endpoints.
+            jobs: an optional list of dictionaries where each dictionary represents the Parca
+                scrape configuration for a single job. When not provided, a default scrape
+                configuration is provided polling all units of the charm on port `80` using the
+                `ProfilingEndpointProvider` object.
+            refresh_event: an optional bound event or list of bound events which
+                will be observed to re-set scrape job data (IP address and others)
+
+        Raises:
+            RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+                with the same name as provided via `relation_name` argument.
+            RelationInterfaceMismatchError: The relation with the same name as provided
+                via `relation_name` argument does not have the `parca_scrape` relation
+                interface.
+            ops.RelationRoleMismatchError: If the relation with the same name as provided
+                via `relation_name` argument does not have the `ops.RelationRole.provides`
+                role.
+        """
+        _validate_relation_by_interface_and_direction(
+            charm, relation_name, RELATION_INTERFACE_NAME, ops.RelationRole.provides
+        )
+
+        super().__init__(charm, relation_name)
+        self.topology = ProviderTopology.from_charm(charm)
+
+        self._charm = charm
+        self._relation_name = relation_name
+        # sanitize job configurations to the supported subset of parameters
+        jobs = [] if jobs is None else jobs
+        self._jobs = [_sanitize_scrape_configuration(job) for job in jobs]
+
+        events = self._charm.on[self._relation_name]
+        self.framework.observe(events.relation_joined, self._set_scrape_job_spec)
+        self.framework.observe(events.relation_changed, self._set_scrape_job_spec)
+
+        if not refresh_event:
+            if len(self._charm.meta.containers) == 1:
+                if "kubernetes" in self._charm.meta.series:
+                    # This is a podspec charm
+                    refresh_event = [self._charm.on.update_status]
+                else:
+                    # This is a sidecar/pebble charm
+                    container = list(self._charm.meta.containers.values())[0]
+                    refresh_event = [self._charm.on[container.name.replace("-", "_")].pebble_ready]
+            else:
+                refresh_event = [self._charm.on.update_status]
+
+        else:
+            if not isinstance(refresh_event, list):
+                refresh_event = [refresh_event]
+
+        for ev in refresh_event:
+            self.framework.observe(ev, self._set_unit_ip)
+
+        self.framework.observe(self._charm.on.upgrade_charm, self._set_scrape_job_spec)
+        # If there is no leader during relation_joined we will still need to set alert rules.
+        self.framework.observe(self._charm.on.leader_elected, self._set_scrape_job_spec)
+
+    def _set_scrape_job_spec(self, event):
+        """Ensure scrape target information is made available to Parca.
+
+        When a profiling provider charm is related to a Parca charm, the profiling provider sets
+        specification and metadata related to its own scrape configuration. This information is set
+        using Juju application data. Each of the consumer units also sets its own host address in
+        Juju unit relation data.
+        """
+        self._set_unit_ip(event)
+
+        if not self._charm.unit.is_leader():
+            return
+
+        for relation in self._charm.model.relations[self._relation_name]:
+            relation.data[self._charm.app]["scrape_metadata"] = json.dumps(self._scrape_metadata)
+            relation.data[self._charm.app]["scrape_jobs"] = json.dumps(self._scrape_jobs)
+
+    def _set_unit_ip(self, _):
+        """Set unit host address.
+
+        Each time a profiling provider charm container is restarted it updates its own host address
+        in the unit relation data for the Parca charm. The only argument specified is an event and
+        it is ignored.
+        """
+        for relation in self._charm.model.relations[self._relation_name]:
+            relation.data[self._charm.unit]["parca_scrape_unit_address"] = socket.getfqdn()
+            relation.data[self._charm.unit]["parca_scrape_unit_name"] = str(
+                self._charm.model.unit.name
+            )
+
+    def _is_valid_unit_address(self, address: str) -> bool:
+        """Validate a unit address.
+
+        Args:
+            address: a string representing a unit address
+        """
+        try:
+            _ = ipaddress.ip_address(address)
+            return True
+        except ValueError:
+            return False
+
+    @property
+    def _scrape_jobs(self) -> list:
+        """Fetch list of scrape jobs.
+
+        Returns:
+           A list of dictionaries, where each dictionary specifies a single scrape job for Parca.
+        """
+        return self._jobs if self._jobs else [DEFAULT_JOB]
+
+    @property
+    def _scrape_metadata(self) -> dict:
+        """Generate scrape metadata.
+
+        Returns:
+            Scrape configuration metadata for this profiling provider charm.
+        """
+        return self.topology.as_dict()
diff --git a/lib/charms/prometheus_k8s/v0/prometheus_scrape.py b/lib/charms/prometheus_k8s/v0/prometheus_scrape.py
index 72c3fe7..e3d35c6 100644
--- a/lib/charms/prometheus_k8s/v0/prometheus_scrape.py
+++ b/lib/charms/prometheus_k8s/v0/prometheus_scrape.py
@@ -178,7 +178,7 @@ def __init__(self, *args):
 - `scrape_timeout`
 - `proxy_url`
 - `relabel_configs`
-- `metrics_relabel_configs`
+- `metric_relabel_configs`
 - `sample_limit`
 - `label_limit`
 - `label_name_length_limit`
@@ -362,7 +362,7 @@ def _on_scrape_targets_changed(self, event):
 
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
-LIBPATCH = 45
+LIBPATCH = 47
 
 PYDEPS = ["cosl"]
 
@@ -377,7 +377,7 @@ def _on_scrape_targets_changed(self, event):
     "scrape_timeout",
     "proxy_url",
     "relabel_configs",
-    "metrics_relabel_configs",
+    "metric_relabel_configs",
     "sample_limit",
     "label_limit",
     "label_name_length_limit",
@@ -521,8 +521,8 @@ def expand_wildcard_targets_into_individual_jobs(
                         # for such a target. Therefore labeling with Juju topology, excluding the
                         # unit name.
                         non_wildcard_static_config["labels"] = {
-                            **non_wildcard_static_config.get("labels", {}),
                             **topology.label_matcher_dict,
+                            **non_wildcard_static_config.get("labels", {}),
                         }
 
                     non_wildcard_static_configs.append(non_wildcard_static_config)
@@ -547,9 +547,9 @@ def expand_wildcard_targets_into_individual_jobs(
                         if topology:
                             # Add topology labels
                             modified_static_config["labels"] = {
-                                **modified_static_config.get("labels", {}),
                                 **topology.label_matcher_dict,
                                 **{"juju_unit": unit_name},
+                                **modified_static_config.get("labels", {}),
                             }
 
                             # Instance relabeling for topology should be last in order.
diff --git a/lib/charms/tempo_k8s/v1/charm_tracing.py b/lib/charms/tempo_k8s/v1/charm_tracing.py
index 39ebcd4..d7b37eb 100644
--- a/lib/charms/tempo_k8s/v1/charm_tracing.py
+++ b/lib/charms/tempo_k8s/v1/charm_tracing.py
@@ -146,7 +146,7 @@ def my_tracing_endpoint(self) -> Optional[str]:
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
 
-LIBPATCH = 5
+LIBPATCH = 8
 
 PYDEPS = ["opentelemetry-exporter-otlp-proto-http==1.21.0"]
 
@@ -504,22 +504,24 @@ def trace_type(cls: _T) -> _T:
     for name, method in inspect.getmembers(cls, predicate=inspect.isfunction):
         logger.info(f"discovered {method}")
 
-        if method.__name__.startswith("__"):
-            logger.info(f"skipping {method} (dunder)")
+        if not method.__name__.isidentifier():
+            logger.info(f"skipping {method}")
             continue
 
-        isstatic = isinstance(inspect.getattr_static(cls, method.__name__), staticmethod)
-        setattr(cls, name, trace_method(method, static=isstatic))
+        new_method = trace_method(method)
+        if isinstance(inspect.getattr_static(cls, method.__name__), staticmethod):
+            new_method = staticmethod(new_method)
+        setattr(cls, name, new_method)
 
     return cls
 
 
-def trace_method(method: _F, static: bool = False) -> _F:
+def trace_method(method: _F) -> _F:
     """Trace this method.
 
     A span will be opened when this method is called and closed when it returns.
     """
-    return _trace_callable(method, "method", static=static)
+    return _trace_callable(method, "method")
 
 
 def trace_function(function: _F) -> _F:
@@ -530,20 +532,14 @@ def trace_function(function: _F) -> _F:
     return _trace_callable(function, "function")
 
 
-def _trace_callable(callable: _F, qualifier: str, static: bool = False) -> _F:
+def _trace_callable(callable: _F, qualifier: str) -> _F:
     logger.info(f"instrumenting {callable}")
 
     # sig = inspect.signature(callable)
     @functools.wraps(callable)
     def wrapped_function(*args, **kwargs):  # type: ignore
         name = getattr(callable, "__qualname__", getattr(callable, "__name__", str(callable)))
-        with _span(f"{'(static) ' if static else ''}{qualifier} call: {name}"):  # type: ignore
-            if static:
-                # fixme: do we or don't we need [1:]?
-                #  The _trace_callable decorator doesn't always play nice with @staticmethods.
-                #  Sometimes it will receive 'self', sometimes it won't.
-                # return callable(*args, **kwargs)  # type: ignore
-                return callable(*args[1:], **kwargs)  # type: ignore
+        with _span(f"{qualifier} call: {name}"):  # type: ignore
             return callable(*args, **kwargs)  # type: ignore
 
     # wrapped_function.__signature__ = sig
diff --git a/lib/charms/tempo_k8s/v1/tracing.py b/lib/charms/tempo_k8s/v1/tracing.py
index 30566ad..8a03852 100644
--- a/lib/charms/tempo_k8s/v1/tracing.py
+++ b/lib/charms/tempo_k8s/v1/tracing.py
@@ -93,7 +93,7 @@ def __init__(self, *args):
 
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
-LIBPATCH = 4
+LIBPATCH = 6
 
 PYDEPS = ["pydantic>=2"]
 
@@ -106,7 +106,7 @@ def __init__(self, *args):
     "otlp_grpc", "otlp_http", "zipkin", "tempo", "jaeger_http_thrift", "jaeger_grpc"
 ]
 
-RawIngester = Tuple[IngesterProtocol, int, str]
+RawIngester = Tuple[IngesterProtocol, int]
 BUILTIN_JUJU_KEYS = {"ingress-address", "private-address", "egress-subnets"}
 
 
@@ -190,7 +190,6 @@ class Ingester(BaseModel):  # noqa: D101
 
     protocol: IngesterProtocol
     port: int
-    path: Optional[str] = None
 
 
 class TracingProviderAppData(DatabagModel):  # noqa: D101
@@ -362,6 +361,8 @@ def __init__(
 
         Args:
             charm: a `CharmBase` instance that manages this instance of the Tempo service.
+            host: hostname.
+            ingesters: list of ingester protocols that are enabled on this endpoint.
             relation_name: an optional string name of the relation between `charm`
                 and the Tempo charmed service. The default is "tracing".
 
@@ -397,8 +398,8 @@ def _on_relation_event(self, _):
                     TracingProviderAppData(
                         host=self._host,
                         ingesters=[
-                            Ingester(port=port, protocol=protocol, path=path)
-                            for protocol, port, path in self._ingesters
+                            Ingester(port=port, protocol=protocol)
+                            for protocol, port in self._ingesters
                         ],
                     ).dump(relation.data[self._charm.app])
 
@@ -581,8 +582,7 @@ def _get_ingester(
             # TCP protocols don't want an http/https scheme prefix
             base_url = base_url.split("://")[1]
 
-        suffix = receiver.path or ""
-        return f"{base_url}{suffix}"
+        return f"{base_url}"
 
     def otlp_grpc_endpoint(self, relation: Optional[Relation] = None) -> Optional[str]:
         """Ingester endpoint for the ``otlp_grpc`` protocol."""
diff --git a/lib/charms/tempo_k8s/v2/tracing.py b/lib/charms/tempo_k8s/v2/tracing.py
new file mode 100644
index 0000000..b4e341c
--- /dev/null
+++ b/lib/charms/tempo_k8s/v2/tracing.py
@@ -0,0 +1,923 @@
+# Copyright 2024 Canonical Ltd.
+# See LICENSE file for licensing details.
+"""## Overview.
+
+This document explains how to integrate with the Tempo charm for the purpose of pushing traces to a
+tracing endpoint provided by Tempo. It also explains how alternative implementations of the Tempo charm
+may maintain the same interface and be backward compatible with all currently integrated charms.
+
+## Requirer Library Usage
+
+Charms seeking to push traces to Tempo, must do so using the `TracingEndpointRequirer`
+object from this charm library. For the simplest use cases, using the `TracingEndpointRequirer`
+object only requires instantiating it, typically in the constructor of your charm. The
+`TracingEndpointRequirer` constructor requires the name of the relation over which a tracing endpoint
+ is exposed by the Tempo charm, and a list of protocols it intends to send traces with.
+ This relation must use the `tracing` interface.
+ The `TracingEndpointRequirer` object may be instantiated as follows
+
+    from charms.tempo_k8s.v2.tracing import TracingEndpointRequirer
+
+    def __init__(self, *args):
+        super().__init__(*args)
+        # ...
+        self.tracing = TracingEndpointRequirer(self,
+            protocols=['otlp_grpc', 'otlp_http', 'jaeger_http_thrift']
+        )
+        # ...
+
+Note that the first argument (`self`) to `TracingEndpointRequirer` is always a reference to the
+parent charm.
+
+Alternatively to providing the list of requested protocols at init time, the charm can do it at
+any point in time by calling the
+`TracingEndpointRequirer.request_protocols(*protocol:str, relation:Optional[Relation])` method.
+Using this method also allows you to use per-relation protocols.
+
+Units of provider charms obtain the tempo endpoint to which they will push their traces by calling
+`TracingEndpointRequirer.get_endpoint(protocol: str)`, where `protocol` is, for example:
+- `otlp_grpc`
+- `otlp_http`
+- `zipkin`
+- `tempo`
+
+If the `protocol` is not in the list of protocols that the charm requested at endpoint set-up time,
+the library will raise an error.
+
+## Requirer Library Usage
+
+The `TracingEndpointProvider` object may be used by charms to manage relations with their
+trace sources. For this purposes a Tempo-like charm needs to do two things
+
+1. Instantiate the `TracingEndpointProvider` object by providing it a
+reference to the parent (Tempo) charm and optionally the name of the relation that the Tempo charm
+uses to interact with its trace sources. This relation must conform to the `tracing` interface
+and it is strongly recommended that this relation be named `tracing` which is its
+default value.
+
+For example a Tempo charm may instantiate the `TracingEndpointProvider` in its constructor as
+follows
+
+    from charms.tempo_k8s.v2.tracing import TracingEndpointProvider
+
+    def __init__(self, *args):
+        super().__init__(*args)
+        # ...
+        self.tracing = TracingEndpointProvider(self)
+        # ...
+
+
+
+"""  # noqa: W505
+import enum
+import json
+import logging
+from typing import (
+    TYPE_CHECKING,
+    Any,
+    Dict,
+    List,
+    Literal,
+    MutableMapping,
+    Optional,
+    Sequence,
+    Tuple,
+    cast,
+)
+
+import pydantic
+from ops.charm import (
+    CharmBase,
+    CharmEvents,
+    RelationBrokenEvent,
+    RelationEvent,
+    RelationRole,
+)
+from ops.framework import EventSource, Object
+from ops.model import ModelError, Relation
+from pydantic import BaseModel, ConfigDict, Field
+
+# The unique Charmhub library identifier, never change it
+LIBID = "12977e9aa0b34367903d8afeb8c3d85d"
+
+# Increment this major API version when introducing breaking changes
+LIBAPI = 2
+
+# Increment this PATCH version before using `charmcraft publish-lib` or reset
+# to 0 if you are raising the major API version
+LIBPATCH = 6
+
+PYDEPS = ["pydantic"]
+
+logger = logging.getLogger(__name__)
+
+DEFAULT_RELATION_NAME = "tracing"
+RELATION_INTERFACE_NAME = "tracing"
+
+ReceiverProtocol = Literal[
+    "zipkin",
+    "kafka",
+    "opencensus",
+    "tempo_http",
+    "tempo_grpc",
+    "otlp_grpc",
+    "otlp_http",
+]
+
+RawReceiver = Tuple[ReceiverProtocol, str]
+"""Helper type. A raw receiver is defined as a tuple consisting of the protocol name, and the (external, if available),
+(secured, if available) resolvable server url.
+"""
+
+BUILTIN_JUJU_KEYS = {"ingress-address", "private-address", "egress-subnets"}
+
+
+class TransportProtocolType(str, enum.Enum):
+    """Receiver Type."""
+
+    http = "http"
+    grpc = "grpc"
+
+
+receiver_protocol_to_transport_protocol = {
+    "zipkin": TransportProtocolType.http,
+    "kafka": TransportProtocolType.http,
+    "opencensus": TransportProtocolType.http,
+    "tempo_http": TransportProtocolType.http,
+    "tempo_grpc": TransportProtocolType.grpc,
+    "otlp_grpc": TransportProtocolType.grpc,
+    "otlp_http": TransportProtocolType.http,
+}
+"""A mapping between telemetry protocols and their corresponding transport protocol.
+"""
+
+
+class TracingError(Exception):
+    """Base class for custom errors raised by this library."""
+
+
+class NotReadyError(TracingError):
+    """Raised by the provider wrapper if a requirer hasn't published the required data (yet)."""
+
+
+class ProtocolNotRequestedError(TracingError):
+    """Raised if the user attempts to obtain an endpoint for a protocol it did not request."""
+
+
+class DataValidationError(TracingError):
+    """Raised when data validation fails on IPU relation data."""
+
+
+class AmbiguousRelationUsageError(TracingError):
+    """Raised when one wrongly assumes that there can only be one relation on an endpoint."""
+
+
+if int(pydantic.version.VERSION.split(".")[0]) < 2:
+
+    class DatabagModel(BaseModel):  # type: ignore
+        """Base databag model."""
+
+        class Config:
+            """Pydantic config."""
+
+            # ignore any extra fields in the databag
+            extra = "ignore"
+            """Ignore any extra fields in the databag."""
+            allow_population_by_field_name = True
+            """Allow instantiating this class by field name (instead of forcing alias)."""
+
+        _NEST_UNDER = None
+
+        @classmethod
+        def load(cls, databag: MutableMapping):
+            """Load this model from a Juju databag."""
+            if cls._NEST_UNDER:
+                return cls.parse_obj(json.loads(databag[cls._NEST_UNDER]))
+
+            try:
+                data = {
+                    k: json.loads(v)
+                    for k, v in databag.items()
+                    # Don't attempt to parse model-external values
+                    if k in {f.alias for f in cls.__fields__.values()}
+                }
+            except json.JSONDecodeError as e:
+                msg = f"invalid databag contents: expecting json. {databag}"
+                logger.error(msg)
+                raise DataValidationError(msg) from e
+
+            try:
+                return cls.parse_raw(json.dumps(data))  # type: ignore
+            except pydantic.ValidationError as e:
+                msg = f"failed to validate databag: {databag}"
+                logger.debug(msg, exc_info=True)
+                raise DataValidationError(msg) from e
+
+        def dump(self, databag: Optional[MutableMapping] = None, clear: bool = True):
+            """Write the contents of this model to Juju databag.
+
+            :param databag: the databag to write the data to.
+            :param clear: ensure the databag is cleared before writing it.
+            """
+            if clear and databag:
+                databag.clear()
+
+            if databag is None:
+                databag = {}
+
+            if self._NEST_UNDER:
+                databag[self._NEST_UNDER] = self.json(by_alias=True)
+                return databag
+
+            dct = self.dict()
+            for key, field in self.__fields__.items():  # type: ignore
+                value = dct[key]
+                databag[field.alias or key] = json.dumps(value)
+
+            return databag
+
+else:
+    from pydantic import ConfigDict
+
+    class DatabagModel(BaseModel):
+        """Base databag model."""
+
+        model_config = ConfigDict(
+            # ignore any extra fields in the databag
+            extra="ignore",
+            # Allow instantiating this class by field name (instead of forcing alias).
+            populate_by_name=True,
+            # Custom config key: whether to nest the whole datastructure (as json)
+            # under a field or spread it out at the toplevel.
+            _NEST_UNDER=None,  # type: ignore
+        )
+        """Pydantic config."""
+
+        @classmethod
+        def load(cls, databag: MutableMapping):
+            """Load this model from a Juju databag."""
+            nest_under = cls.model_config.get("_NEST_UNDER")  # type: ignore
+            if nest_under:
+                return cls.model_validate(json.loads(databag[nest_under]))  # type: ignore
+
+            try:
+                data = {
+                    k: json.loads(v)
+                    for k, v in databag.items()
+                    # Don't attempt to parse model-external values
+                    if k in {(f.alias or n) for n, f in cls.__fields__.items()}
+                }
+            except json.JSONDecodeError as e:
+                msg = f"invalid databag contents: expecting json. {databag}"
+                logger.error(msg)
+                raise DataValidationError(msg) from e
+
+            try:
+                return cls.model_validate_json(json.dumps(data))  # type: ignore
+            except pydantic.ValidationError as e:
+                msg = f"failed to validate databag: {databag}"
+                logger.debug(msg, exc_info=True)
+                raise DataValidationError(msg) from e
+
+        def dump(self, databag: Optional[MutableMapping] = None, clear: bool = True):
+            """Write the contents of this model to Juju databag.
+
+            :param databag: the databag to write the data to.
+            :param clear: ensure the databag is cleared before writing it.
+            """
+            if clear and databag:
+                databag.clear()
+
+            if databag is None:
+                databag = {}
+            nest_under = self.model_config.get("_NEST_UNDER")
+            if nest_under:
+                databag[nest_under] = self.model_dump_json(  # type: ignore
+                    by_alias=True,
+                    # skip keys whose values are default
+                    exclude_defaults=True,
+                )
+                return databag
+
+            dct = self.model_dump()  # type: ignore
+            for key, field in self.model_fields.items():  # type: ignore
+                value = dct[key]
+                if value == field.default:
+                    continue
+                databag[field.alias or key] = json.dumps(value)
+
+            return databag
+
+
+# todo use models from charm-relation-interfaces
+if int(pydantic.version.VERSION.split(".")[0]) < 2:
+
+    class ProtocolType(BaseModel):  # type: ignore
+        """Protocol Type."""
+
+        class Config:
+            """Pydantic config."""
+
+            use_enum_values = True
+            """Allow serializing enum values."""
+
+        name: str = Field(
+            ...,
+            description="Receiver protocol name. What protocols are supported (and what they are called) "
+            "may differ per provider.",
+            examples=["otlp_grpc", "otlp_http", "tempo_http"],
+        )
+
+        type: TransportProtocolType = Field(
+            ...,
+            description="The transport protocol used by this receiver.",
+            examples=["http", "grpc"],
+        )
+
+else:
+
+    class ProtocolType(BaseModel):
+        """Protocol Type."""
+
+        model_config = ConfigDict(
+            # Allow serializing enum values.
+            use_enum_values=True
+        )
+        """Pydantic config."""
+
+        name: str = Field(
+            ...,
+            description="Receiver protocol name. What protocols are supported (and what they are called) "
+            "may differ per provider.",
+            examples=["otlp_grpc", "otlp_http", "tempo_http"],
+        )
+
+        type: TransportProtocolType = Field(
+            ...,
+            description="The transport protocol used by this receiver.",
+            examples=["http", "grpc"],
+        )
+
+
+class Receiver(BaseModel):
+    """Specification of an active receiver."""
+
+    protocol: ProtocolType = Field(..., description="Receiver protocol name and type.")
+    url: str = Field(
+        ...,
+        description="""URL at which the receiver is reachable. If there's an ingress, it would be the external URL.
+        Otherwise, it would be the service's fqdn or internal IP.
+        If the protocol type is grpc, the url will not contain a scheme.""",
+        examples=[
+            "http://traefik_address:2331",
+            "https://traefik_address:2331",
+            "http://tempo_public_ip:2331",
+            "https://tempo_public_ip:2331",
+            "tempo_public_ip:2331",
+        ],
+    )
+
+
+class TracingProviderAppData(DatabagModel):  # noqa: D101
+    """Application databag model for the tracing provider."""
+
+    receivers: List[Receiver] = Field(
+        ...,
+        description="List of all receivers enabled on the tracing provider.",
+    )
+
+
+class TracingRequirerAppData(DatabagModel):  # noqa: D101
+    """Application databag model for the tracing requirer."""
+
+    receivers: List[ReceiverProtocol]
+    """Requested receivers."""
+
+
+class _AutoSnapshotEvent(RelationEvent):
+    __args__: Tuple[str, ...] = ()
+    __optional_kwargs__: Dict[str, Any] = {}
+
+    @classmethod
+    def __attrs__(cls):
+        return cls.__args__ + tuple(cls.__optional_kwargs__.keys())
+
+    def __init__(self, handle, relation, *args, **kwargs):
+        super().__init__(handle, relation)
+
+        if not len(self.__args__) == len(args):
+            raise TypeError("expected {} args, got {}".format(len(self.__args__), len(args)))
+
+        for attr, obj in zip(self.__args__, args):
+            setattr(self, attr, obj)
+        for attr, default in self.__optional_kwargs__.items():
+            obj = kwargs.get(attr, default)
+            setattr(self, attr, obj)
+
+    def snapshot(self) -> dict:
+        dct = super().snapshot()
+        for attr in self.__attrs__():
+            obj = getattr(self, attr)
+            try:
+                dct[attr] = obj
+            except ValueError as e:
+                raise ValueError(
+                    "cannot automagically serialize {}: "
+                    "override this method and do it "
+                    "manually.".format(obj)
+                ) from e
+
+        return dct
+
+    def restore(self, snapshot: dict) -> None:
+        super().restore(snapshot)
+        for attr, obj in snapshot.items():
+            setattr(self, attr, obj)
+
+
+class RelationNotFoundError(Exception):
+    """Raised if no relation with the given name is found."""
+
+    def __init__(self, relation_name: str):
+        self.relation_name = relation_name
+        self.message = "No relation named '{}' found".format(relation_name)
+        super().__init__(self.message)
+
+
+class RelationInterfaceMismatchError(Exception):
+    """Raised if the relation with the given name has an unexpected interface."""
+
+    def __init__(
+        self,
+        relation_name: str,
+        expected_relation_interface: str,
+        actual_relation_interface: str,
+    ):
+        self.relation_name = relation_name
+        self.expected_relation_interface = expected_relation_interface
+        self.actual_relation_interface = actual_relation_interface
+        self.message = (
+            "The '{}' relation has '{}' as interface rather than the expected '{}'".format(
+                relation_name, actual_relation_interface, expected_relation_interface
+            )
+        )
+
+        super().__init__(self.message)
+
+
+class RelationRoleMismatchError(Exception):
+    """Raised if the relation with the given name has a different role than expected."""
+
+    def __init__(
+        self,
+        relation_name: str,
+        expected_relation_role: RelationRole,
+        actual_relation_role: RelationRole,
+    ):
+        self.relation_name = relation_name
+        self.expected_relation_interface = expected_relation_role
+        self.actual_relation_role = actual_relation_role
+        self.message = "The '{}' relation has role '{}' rather than the expected '{}'".format(
+            relation_name, repr(actual_relation_role), repr(expected_relation_role)
+        )
+
+        super().__init__(self.message)
+
+
+def _validate_relation_by_interface_and_direction(
+    charm: CharmBase,
+    relation_name: str,
+    expected_relation_interface: str,
+    expected_relation_role: RelationRole,
+):
+    """Validate a relation.
+
+    Verifies that the `relation_name` provided: (1) exists in metadata.yaml,
+    (2) declares as interface the interface name passed as `relation_interface`
+    and (3) has the right "direction", i.e., it is a relation that `charm`
+    provides or requires.
+
+    Args:
+        charm: a `CharmBase` object to scan for the matching relation.
+        relation_name: the name of the relation to be verified.
+        expected_relation_interface: the interface name to be matched by the
+            relation named `relation_name`.
+        expected_relation_role: whether the `relation_name` must be either
+            provided or required by `charm`.
+
+    Raises:
+        RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+            with the same name as provided via `relation_name` argument.
+        RelationInterfaceMismatchError: The relation with the same name as provided
+            via `relation_name` argument does not have the same relation interface
+            as specified via the `expected_relation_interface` argument.
+        RelationRoleMismatchError: If the relation with the same name as provided
+            via `relation_name` argument does not have the same role as specified
+            via the `expected_relation_role` argument.
+    """
+    if relation_name not in charm.meta.relations:
+        raise RelationNotFoundError(relation_name)
+
+    relation = charm.meta.relations[relation_name]
+
+    # fixme: why do we need to cast here?
+    actual_relation_interface = cast(str, relation.interface_name)
+
+    if actual_relation_interface != expected_relation_interface:
+        raise RelationInterfaceMismatchError(
+            relation_name, expected_relation_interface, actual_relation_interface
+        )
+
+    if expected_relation_role is RelationRole.provides:
+        if relation_name not in charm.meta.provides:
+            raise RelationRoleMismatchError(
+                relation_name, RelationRole.provides, RelationRole.requires
+            )
+    elif expected_relation_role is RelationRole.requires:
+        if relation_name not in charm.meta.requires:
+            raise RelationRoleMismatchError(
+                relation_name, RelationRole.requires, RelationRole.provides
+            )
+    else:
+        raise TypeError("Unexpected RelationDirection: {}".format(expected_relation_role))
+
+
+class RequestEvent(RelationEvent):
+    """Event emitted when a remote requests a tracing endpoint."""
+
+    @property
+    def requested_receivers(self) -> List[ReceiverProtocol]:
+        """List of receiver protocols that have been requested."""
+        relation = self.relation
+        app = relation.app
+        if not app:
+            raise NotReadyError("relation.app is None")
+
+        return TracingRequirerAppData.load(relation.data[app]).receivers
+
+
+class BrokenEvent(RelationBrokenEvent):
+    """Event emitted when a relation on tracing is broken."""
+
+
+class TracingEndpointProviderEvents(CharmEvents):
+    """TracingEndpointProvider events."""
+
+    request = EventSource(RequestEvent)
+    broken = EventSource(BrokenEvent)
+
+
+class TracingEndpointProvider(Object):
+    """Class representing a trace receiver service."""
+
+    on = TracingEndpointProviderEvents()  # type: ignore
+
+    def __init__(
+        self,
+        charm: CharmBase,
+        external_url: Optional[str] = None,
+        relation_name: str = DEFAULT_RELATION_NAME,
+    ):
+        """Initialize.
+
+        Args:
+            charm: a `CharmBase` instance that manages this instance of the Tempo service.
+            external_url: external address of the node hosting the tempo server,
+                if an ingress is present.
+            relation_name: an optional string name of the relation between `charm`
+                and the Tempo charmed service. The default is "tracing".
+
+        Raises:
+            RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+                with the same name as provided via `relation_name` argument.
+            RelationInterfaceMismatchError: The relation with the same name as provided
+                via `relation_name` argument does not have the `tracing` relation
+                interface.
+            RelationRoleMismatchError: If the relation with the same name as provided
+                via `relation_name` argument does not have the `RelationRole.requires`
+                role.
+        """
+        _validate_relation_by_interface_and_direction(
+            charm, relation_name, RELATION_INTERFACE_NAME, RelationRole.provides
+        )
+
+        super().__init__(charm, relation_name + "tracing-provider")
+        self._charm = charm
+        self._external_url = external_url
+        self._relation_name = relation_name
+        self.framework.observe(
+            self._charm.on[relation_name].relation_joined, self._on_relation_event
+        )
+        self.framework.observe(
+            self._charm.on[relation_name].relation_created, self._on_relation_event
+        )
+        self.framework.observe(
+            self._charm.on[relation_name].relation_changed, self._on_relation_event
+        )
+        self.framework.observe(
+            self._charm.on[relation_name].relation_broken, self._on_relation_broken_event
+        )
+
+    def _on_relation_broken_event(self, e: RelationBrokenEvent):
+        """Handle relation broken events."""
+        self.on.broken.emit(e.relation)
+
+    def _on_relation_event(self, e: RelationEvent):
+        """Handle relation created/joined/changed events."""
+        if self.is_requirer_ready(e.relation):
+            self.on.request.emit(e.relation)
+
+    def is_requirer_ready(self, relation: Relation):
+        """Attempt to determine if requirer has already populated app data."""
+        try:
+            self._get_requested_protocols(relation)
+        except NotReadyError:
+            return False
+        return True
+
+    @staticmethod
+    def _get_requested_protocols(relation: Relation):
+        app = relation.app
+        if not app:
+            raise NotReadyError("relation.app is None")
+
+        try:
+            databag = TracingRequirerAppData.load(relation.data[app])
+        except (json.JSONDecodeError, pydantic.ValidationError, DataValidationError):
+            logger.info(f"relation {relation} is not ready to talk tracing")
+            raise NotReadyError()
+        return databag.receivers
+
+    def requested_protocols(self):
+        """All receiver protocols that have been requested by our related apps."""
+        requested_protocols = set()
+        for relation in self.relations:
+            try:
+                protocols = self._get_requested_protocols(relation)
+            except NotReadyError:
+                continue
+            requested_protocols.update(protocols)
+        return requested_protocols
+
+    @property
+    def relations(self) -> List[Relation]:
+        """All relations active on this endpoint."""
+        return self._charm.model.relations[self._relation_name]
+
+    def publish_receivers(self, receivers: Sequence[RawReceiver]):
+        """Let all requirers know that these receivers are active and listening."""
+        if not self._charm.unit.is_leader():
+            raise RuntimeError("only leader can do this")
+
+        for relation in self.relations:
+            try:
+                TracingProviderAppData(
+                    receivers=[
+                        Receiver(
+                            url=url,
+                            protocol=ProtocolType(
+                                name=protocol,
+                                type=receiver_protocol_to_transport_protocol[protocol],
+                            ),
+                        )
+                        for protocol, url in receivers
+                    ],
+                ).dump(relation.data[self._charm.app])
+
+            except ModelError as e:
+                # args are bytes
+                msg = e.args[0]
+                if isinstance(msg, bytes):
+                    if msg.startswith(
+                        b"ERROR cannot read relation application settings: permission denied"
+                    ):
+                        logger.error(
+                            f"encountered error {e} while attempting to update_relation_data."
+                            f"The relation must be gone."
+                        )
+                        continue
+                raise
+
+
+class EndpointRemovedEvent(RelationBrokenEvent):
+    """Event representing a change in one of the receiver endpoints."""
+
+
+class EndpointChangedEvent(_AutoSnapshotEvent):
+    """Event representing a change in one of the receiver endpoints."""
+
+    __args__ = ("_receivers",)
+
+    if TYPE_CHECKING:
+        _receivers = []  # type: List[dict]
+
+    @property
+    def receivers(self) -> List[Receiver]:
+        """Cast receivers back from dict."""
+        return [Receiver(**i) for i in self._receivers]
+
+
+class TracingEndpointRequirerEvents(CharmEvents):
+    """TracingEndpointRequirer events."""
+
+    endpoint_changed = EventSource(EndpointChangedEvent)
+    endpoint_removed = EventSource(EndpointRemovedEvent)
+
+
+class TracingEndpointRequirer(Object):
+    """A tracing endpoint for Tempo."""
+
+    on = TracingEndpointRequirerEvents()  # type: ignore
+
+    def __init__(
+        self,
+        charm: CharmBase,
+        relation_name: str = DEFAULT_RELATION_NAME,
+        protocols: Optional[List[ReceiverProtocol]] = None,
+    ):
+        """Construct a tracing requirer for a Tempo charm.
+
+        If your application supports pushing traces to a distributed tracing backend, the
+        `TracingEndpointRequirer` object enables your charm to easily access endpoint information
+        exchanged over a `tracing` relation interface.
+
+        Args:
+            charm: a `CharmBase` object that manages this
+                `TracingEndpointRequirer` object. Typically, this is `self` in the instantiating
+                class.
+            relation_name: an optional string name of the relation between `charm`
+                and the Tempo charmed service. The default is "tracing". It is strongly
+                advised not to change the default, so that people deploying your charm will have a
+                consistent experience with all other charms that provide tracing endpoints.
+            protocols: optional list of protocols that the charm intends to send traces with.
+                The provider will enable receivers for these and only these protocols,
+                so be sure to enable all protocols the charm or its workload are going to need.
+
+        Raises:
+            RelationNotFoundError: If there is no relation in the charm's metadata.yaml
+                with the same name as provided via `relation_name` argument.
+            RelationInterfaceMismatchError: The relation with the same name as provided
+                via `relation_name` argument does not have the `tracing` relation
+                interface.
+            RelationRoleMismatchError: If the relation with the same name as provided
+                via `relation_name` argument does not have the `RelationRole.provides`
+                role.
+        """
+        _validate_relation_by_interface_and_direction(
+            charm, relation_name, RELATION_INTERFACE_NAME, RelationRole.requires
+        )
+
+        super().__init__(charm, relation_name)
+
+        self._is_single_endpoint = charm.meta.relations[relation_name].limit == 1
+
+        self._charm = charm
+        self._relation_name = relation_name
+
+        events = self._charm.on[self._relation_name]
+        self.framework.observe(events.relation_changed, self._on_tracing_relation_changed)
+        self.framework.observe(events.relation_broken, self._on_tracing_relation_broken)
+
+        if protocols:
+            self.request_protocols(protocols)
+
+    def request_protocols(
+        self, protocols: Sequence[ReceiverProtocol], relation: Optional[Relation] = None
+    ):
+        """Publish the list of protocols which the provider should activate."""
+        # todo: should we check if _is_single_endpoint and len(self.relations) > 1 and raise, here?
+        relations = [relation] if relation else self.relations
+
+        if not protocols:
+            # empty sequence
+            raise ValueError(
+                "You need to pass a nonempty sequence of protocols to `request_protocols`."
+            )
+
+        try:
+            if self._charm.unit.is_leader():
+                for relation in relations:
+                    TracingRequirerAppData(
+                        receivers=list(protocols),
+                    ).dump(relation.data[self._charm.app])
+
+        except ModelError as e:
+            # args are bytes
+            msg = e.args[0]
+            if isinstance(msg, bytes):
+                if msg.startswith(
+                    b"ERROR cannot read relation application settings: permission denied"
+                ):
+                    logger.error(
+                        f"encountered error {e} while attempting to request_protocols."
+                        f"The relation must be gone."
+                    )
+                    return
+            raise
+
+    @property
+    def relations(self) -> List[Relation]:
+        """The tracing relations associated with this endpoint."""
+        return self._charm.model.relations[self._relation_name]
+
+    @property
+    def _relation(self) -> Optional[Relation]:
+        """If this wraps a single endpoint, the relation bound to it, if any."""
+        if not self._is_single_endpoint:
+            objname = type(self).__name__
+            raise AmbiguousRelationUsageError(
+                f"This {objname} wraps a {self._relation_name} endpoint that has "
+                "limit != 1. We can't determine what relation, of the possibly many, you are "
+                f"talking about. Please pass a relation instance while calling {objname}, "
+                "or set limit=1 in the charm metadata."
+            )
+        relations = self.relations
+        return relations[0] if relations else None
+
+    def is_ready(self, relation: Optional[Relation] = None):
+        """Is this endpoint ready?"""
+        relation = relation or self._relation
+        if not relation:
+            logger.debug(f"no relation on {self._relation_name !r}: tracing not ready")
+            return False
+        if relation.data is None:
+            logger.error(f"relation data is None for {relation}")
+            return False
+        if not relation.app:
+            logger.error(f"{relation} event received but there is no relation.app")
+            return False
+        try:
+            databag = dict(relation.data[relation.app])
+            TracingProviderAppData.load(databag)
+
+        except (json.JSONDecodeError, pydantic.ValidationError, DataValidationError):
+            logger.info(f"failed validating relation data for {relation}")
+            return False
+        return True
+
+    def _on_tracing_relation_changed(self, event):
+        """Notify the providers that there is new endpoint information available."""
+        relation = event.relation
+        if not self.is_ready(relation):
+            self.on.endpoint_removed.emit(relation)  # type: ignore
+            return
+
+        data = TracingProviderAppData.load(relation.data[relation.app])
+        self.on.endpoint_changed.emit(relation, [i.dict() for i in data.receivers])  # type: ignore
+
+    def _on_tracing_relation_broken(self, event: RelationBrokenEvent):
+        """Notify the providers that the endpoint is broken."""
+        relation = event.relation
+        self.on.endpoint_removed.emit(relation)  # type: ignore
+
+    def get_all_endpoints(
+        self, relation: Optional[Relation] = None
+    ) -> Optional[TracingProviderAppData]:
+        """Unmarshalled relation data."""
+        relation = relation or self._relation
+        if not self.is_ready(relation):
+            return
+        return TracingProviderAppData.load(relation.data[relation.app])  # type: ignore
+
+    def _get_endpoint(
+        self, relation: Optional[Relation], protocol: ReceiverProtocol
+    ) -> Optional[str]:
+        app_data = self.get_all_endpoints(relation)
+        if not app_data:
+            return None
+        receivers: List[Receiver] = list(
+            filter(lambda i: i.protocol.name == protocol, app_data.receivers)
+        )
+        if not receivers:
+            logger.error(f"no receiver found with protocol={protocol!r}")
+            return
+        if len(receivers) > 1:
+            logger.error(
+                f"too many receivers with protocol={protocol!r}; using first one. Found: {receivers}"
+            )
+            return
+
+        receiver = receivers[0]
+        return receiver.url
+
+    def get_endpoint(
+        self, protocol: ReceiverProtocol, relation: Optional[Relation] = None
+    ) -> Optional[str]:
+        """Receiver endpoint for the given protocol."""
+        endpoint = self._get_endpoint(relation or self._relation, protocol=protocol)
+        if not endpoint:
+            requested_protocols = set()
+            relations = [relation] if relation else self.relations
+            for relation in relations:
+                try:
+                    databag = TracingRequirerAppData.load(relation.data[self._charm.app])
+                except DataValidationError:
+                    continue
+
+                requested_protocols.update(databag.receivers)
+
+            if protocol not in requested_protocols:
+                raise ProtocolNotRequestedError(protocol, relation)
+
+            return None
+        return endpoint
diff --git a/lib/charms/tls_certificates_interface/v3/tls_certificates.py b/lib/charms/tls_certificates_interface/v3/tls_certificates.py
index cbdd80d..33f34b6 100644
--- a/lib/charms/tls_certificates_interface/v3/tls_certificates.py
+++ b/lib/charms/tls_certificates_interface/v3/tls_certificates.py
@@ -111,6 +111,7 @@ def _on_certificate_request(self, event: CertificateCreationRequestEvent) -> Non
             ca=ca_certificate,
             chain=[ca_certificate, certificate],
             relation_id=event.relation_id,
+            recommended_expiry_notification_time=720,
         )
 
     def _on_certificate_revocation_request(self, event: CertificateRevocationRequestEvent) -> None:
@@ -316,7 +317,7 @@ def _on_all_certificates_invalidated(self, event: AllCertificatesInvalidatedEven
 
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
-LIBPATCH = 10
+LIBPATCH = 15
 
 PYDEPS = ["cryptography", "jsonschema"]
 
@@ -453,11 +454,35 @@ class ProviderCertificate:
     ca: str
     chain: List[str]
     revoked: bool
+    expiry_time: datetime
+    expiry_notification_time: Optional[datetime] = None
 
     def chain_as_pem(self) -> str:
         """Return full certificate chain as a PEM string."""
         return "\n\n".join(reversed(self.chain))
 
+    def to_json(self) -> str:
+        """Return the object as a JSON string.
+
+        Returns:
+            str: JSON representation of the object
+        """
+        return json.dumps(
+            {
+                "relation_id": self.relation_id,
+                "application_name": self.application_name,
+                "csr": self.csr,
+                "certificate": self.certificate,
+                "ca": self.ca,
+                "chain": self.chain,
+                "revoked": self.revoked,
+                "expiry_time": self.expiry_time.isoformat(),
+                "expiry_notification_time": self.expiry_notification_time.isoformat()
+                if self.expiry_notification_time
+                else None,
+            }
+        )
+
 
 class CertificateAvailableEvent(EventBase):
     """Charm Event triggered when a TLS certificate is available."""
@@ -682,21 +707,49 @@ def _get_closest_future_time(
     )
 
 
-def _get_certificate_expiry_time(certificate: str) -> Optional[datetime]:
-    """Extract expiry time from a certificate string.
+def calculate_expiry_notification_time(
+    validity_start_time: datetime,
+    expiry_time: datetime,
+    provider_recommended_notification_time: Optional[int],
+    requirer_recommended_notification_time: Optional[int],
+) -> datetime:
+    """Calculate a reasonable time to notify the user about the certificate expiry.
+
+    It takes into account the time recommended by the provider and by the requirer.
+    Time recommended by the provider is preferred,
+    then time recommended by the requirer,
+    then dynamically calculated time.
 
     Args:
-        certificate (str): x509 certificate as a string
+        validity_start_time: Certificate validity time
+        expiry_time: Certificate expiry time
+        provider_recommended_notification_time:
+            Time in hours prior to expiry to notify the user.
+            Recommended by the provider.
+        requirer_recommended_notification_time:
+            Time in hours prior to expiry to notify the user.
+            Recommended by the requirer.
 
     Returns:
-        Optional[datetime]: Expiry datetime or None
+        datetime: Time to notify the user about the certificate expiry.
     """
-    try:
-        certificate_object = x509.load_pem_x509_certificate(data=certificate.encode())
-        return certificate_object.not_valid_after_utc
-    except ValueError:
-        logger.warning("Could not load certificate.")
-        return None
+    if provider_recommended_notification_time is not None:
+        provider_recommended_notification_time = abs(provider_recommended_notification_time)
+        provider_recommendation_time_delta = (
+            expiry_time - timedelta(hours=provider_recommended_notification_time)
+        )
+        if validity_start_time < provider_recommendation_time_delta:
+            return provider_recommendation_time_delta
+
+    if requirer_recommended_notification_time is not None:
+        requirer_recommended_notification_time = abs(requirer_recommended_notification_time)
+        requirer_recommendation_time_delta = (
+            expiry_time - timedelta(hours=requirer_recommended_notification_time)
+        )
+        if validity_start_time < requirer_recommendation_time_delta:
+            return requirer_recommendation_time_delta
+    calculated_hours = (expiry_time - validity_start_time).total_seconds() / (3600 * 3)
+    return expiry_time - timedelta(hours=calculated_hours)
 
 
 def generate_ca(
@@ -965,6 +1018,8 @@ def generate_csr(  # noqa: C901
     organization: Optional[str] = None,
     email_address: Optional[str] = None,
     country_name: Optional[str] = None,
+    state_or_province_name: Optional[str] = None,
+    locality_name: Optional[str] = None,
     private_key_password: Optional[bytes] = None,
     sans: Optional[List[str]] = None,
     sans_oid: Optional[List[str]] = None,
@@ -983,6 +1038,8 @@ def generate_csr(  # noqa: C901
         organization (str): Name of organization.
         email_address (str): Email address.
         country_name (str): Country Name.
+        state_or_province_name (str): State or Province Name.
+        locality_name (str): Locality Name.
         private_key_password (bytes): Private key password
         sans (list): Use sans_dns - this will be deprecated in a future release
             List of DNS subject alternative names (keeping it for now for backward compatibility)
@@ -1008,6 +1065,12 @@ def generate_csr(  # noqa: C901
         subject_name.append(x509.NameAttribute(x509.NameOID.EMAIL_ADDRESS, email_address))
     if country_name:
         subject_name.append(x509.NameAttribute(x509.NameOID.COUNTRY_NAME, country_name))
+    if state_or_province_name:
+        subject_name.append(
+            x509.NameAttribute(x509.NameOID.STATE_OR_PROVINCE_NAME, state_or_province_name)
+        )
+    if locality_name:
+        subject_name.append(x509.NameAttribute(x509.NameOID.LOCALITY_NAME, locality_name))
     csr = x509.CertificateSigningRequestBuilder(subject_name=x509.Name(subject_name))
 
     _sans: List[x509.GeneralName] = []
@@ -1030,6 +1093,13 @@ def generate_csr(  # noqa: C901
     return signed_certificate.public_bytes(serialization.Encoding.PEM)
 
 
+def get_sha256_hex(data: str) -> str:
+    """Calculate the hash of the provided data and return the hexadecimal representation."""
+    digest = hashes.Hash(hashes.SHA256())
+    digest.update(data.encode())
+    return digest.finalize().hex()
+
+
 def csr_matches_certificate(csr: str, cert: str) -> bool:
     """Check if a CSR matches a certificate.
 
@@ -1135,6 +1205,7 @@ def _add_certificate(
         certificate_signing_request: str,
         ca: str,
         chain: List[str],
+        recommended_expiry_notification_time: Optional[int] = None,
     ) -> None:
         """Add certificate to relation data.
 
@@ -1144,6 +1215,8 @@ def _add_certificate(
             certificate_signing_request (str): Certificate Signing Request
             ca (str): CA Certificate
             chain (list): CA Chain
+            recommended_expiry_notification_time (int):
+                Time in hours before the certificate expires to notify the user.
 
         Returns:
             None
@@ -1161,6 +1234,7 @@ def _add_certificate(
             "certificate_signing_request": certificate_signing_request,
             "ca": ca,
             "chain": chain,
+            "recommended_expiry_notification_time": recommended_expiry_notification_time,
         }
         provider_relation_data = self._load_app_relation_data(relation)
         provider_certificates = provider_relation_data.get("certificates", [])
@@ -1227,6 +1301,7 @@ def set_relation_certificate(
         ca: str,
         chain: List[str],
         relation_id: int,
+        recommended_expiry_notification_time: Optional[int] = None,
     ) -> None:
         """Add certificates to relation data.
 
@@ -1236,6 +1311,8 @@ def set_relation_certificate(
             ca (str): CA Certificate
             chain (list): CA Chain
             relation_id (int): Juju relation ID
+            recommended_expiry_notification_time (int):
+                Recommended time in hours before the certificate expires to notify the user.
 
         Returns:
             None
@@ -1257,6 +1334,7 @@ def set_relation_certificate(
             certificate_signing_request=certificate_signing_request.strip(),
             ca=ca.strip(),
             chain=[cert.strip() for cert in chain],
+            recommended_expiry_notification_time=recommended_expiry_notification_time,
         )
 
     def remove_certificate(self, certificate: str) -> None:
@@ -1310,6 +1388,13 @@ def get_provider_certificates(
             provider_relation_data = self._load_app_relation_data(relation)
             provider_certificates = provider_relation_data.get("certificates", [])
             for certificate in provider_certificates:
+                try:
+                    certificate_object = x509.load_pem_x509_certificate(
+                        data=certificate["certificate"].encode()
+                    )
+                except ValueError as e:
+                    logger.error("Could not load certificate - Skipping: %s", e)
+                    continue
                 provider_certificate = ProviderCertificate(
                     relation_id=relation.id,
                     application_name=relation.app.name,
@@ -1318,6 +1403,10 @@ def get_provider_certificates(
                     ca=certificate["ca"],
                     chain=certificate["chain"],
                     revoked=certificate.get("revoked", False),
+                    expiry_time=certificate_object.not_valid_after_utc,
+                    expiry_notification_time=certificate.get(
+                        "recommended_expiry_notification_time"
+                    ),
                 )
                 certificates.append(provider_certificate)
         return certificates
@@ -1475,15 +1564,17 @@ def __init__(
         self,
         charm: CharmBase,
         relationship_name: str,
-        expiry_notification_time: int = 168,
+        expiry_notification_time: Optional[int] = None,
     ):
         """Generate/use private key and observes relation changed event.
 
         Args:
             charm: Charm object
             relationship_name: Juju relation name
-            expiry_notification_time (int): Time difference between now and expiry (in hours).
-                Used to trigger the CertificateExpiring event. Default: 7 days.
+            expiry_notification_time (int): Number of hours prior to certificate expiry.
+                Used to trigger the CertificateExpiring event.
+                This value is used as a recommendation only,
+                The actual value is calculated taking into account the provider's recommendation.
         """
         super().__init__(charm, relationship_name)
         if not JujuVersion.from_environ().has_secrets:
@@ -1544,9 +1635,25 @@ def get_provider_certificates(self) -> List[ProviderCertificate]:
             if not certificate:
                 logger.warning("No certificate found in relation data - Skipping")
                 continue
+            try:
+                certificate_object = x509.load_pem_x509_certificate(data=certificate.encode())
+            except ValueError as e:
+                logger.error("Could not load certificate - Skipping: %s", e)
+                continue
             ca = provider_certificate_dict.get("ca")
             chain = provider_certificate_dict.get("chain", [])
             csr = provider_certificate_dict.get("certificate_signing_request")
+            recommended_expiry_notification_time = provider_certificate_dict.get(
+                "recommended_expiry_notification_time"
+            )
+            expiry_time = certificate_object.not_valid_after_utc
+            validity_start_time = certificate_object.not_valid_before_utc
+            expiry_notification_time = calculate_expiry_notification_time(
+                validity_start_time=validity_start_time,
+                expiry_time=expiry_time,
+                provider_recommended_notification_time=recommended_expiry_notification_time,
+                requirer_recommended_notification_time=self.expiry_notification_time,
+            )
             if not csr:
                 logger.warning("No CSR found in relation data - Skipping")
                 continue
@@ -1559,6 +1666,8 @@ def get_provider_certificates(self) -> List[ProviderCertificate]:
                 ca=ca,
                 chain=chain,
                 revoked=revoked,
+                expiry_time=expiry_time,
+                expiry_notification_time=expiry_notification_time,
             )
             provider_certificates.append(provider_certificate)
         return provider_certificates
@@ -1708,13 +1817,9 @@ def get_expiring_certificates(self) -> List[ProviderCertificate]:
         expiring_certificates: List[ProviderCertificate] = []
         for requirer_csr in self.get_certificate_signing_requests(fulfilled_only=True):
             if cert := self._find_certificate_in_relation_data(requirer_csr.csr):
-                expiry_time = _get_certificate_expiry_time(cert.certificate)
-                if not expiry_time:
+                if not cert.expiry_time or not cert.expiry_notification_time:
                     continue
-                expiry_notification_time = expiry_time - timedelta(
-                    hours=self.expiry_notification_time
-                )
-                if datetime.now(timezone.utc) > expiry_notification_time:
+                if datetime.now(timezone.utc) > cert.expiry_notification_time:
                     expiring_certificates.append(cert)
         return expiring_certificates
 
@@ -1774,9 +1879,15 @@ def _on_relation_changed(self, event: RelationChangedEvent) -> None:
         ]
         for certificate in provider_certificates:
             if certificate.csr in requirer_csrs:
+                csr_in_sha256_hex = get_sha256_hex(certificate.csr)
                 if certificate.revoked:
                     with suppress(SecretNotFoundError):
-                        secret = self.model.get_secret(label=f"{LIBID}-{certificate.csr}")
+                        logger.debug(
+                            "Removing secret with label %s",
+                            f"{LIBID}-{csr_in_sha256_hex}",
+                        )
+                        secret = self.model.get_secret(
+                            label=f"{LIBID}-{csr_in_sha256_hex}")
                         secret.remove_all_revisions()
                     self.on.certificate_invalidated.emit(
                         reason="revoked",
@@ -1787,16 +1898,24 @@ def _on_relation_changed(self, event: RelationChangedEvent) -> None:
                     )
                 else:
                     try:
-                        secret = self.model.get_secret(label=f"{LIBID}-{certificate.csr}")
-                        secret.set_content({"certificate": certificate.certificate})
+                        logger.debug(
+                            "Setting secret with label %s", f"{LIBID}-{csr_in_sha256_hex}"
+                        )
+                        secret = self.model.get_secret(label=f"{LIBID}-{csr_in_sha256_hex}")
+                        secret.set_content(
+                            {"certificate": certificate.certificate, "csr": certificate.csr}
+                        )
                         secret.set_info(
-                            expire=self._get_next_secret_expiry_time(certificate.certificate),
+                            expire=self._get_next_secret_expiry_time(certificate),
                         )
                     except SecretNotFoundError:
+                        logger.debug(
+                            "Creating new secret with label %s", f"{LIBID}-{csr_in_sha256_hex}"
+                        )
                         secret = self.charm.unit.add_secret(
-                            {"certificate": certificate.certificate},
-                            label=f"{LIBID}-{certificate.csr}",
-                            expire=self._get_next_secret_expiry_time(certificate.certificate),
+                            {"certificate": certificate.certificate, "csr": certificate.csr},
+                            label=f"{LIBID}-{csr_in_sha256_hex}",
+                            expire=self._get_next_secret_expiry_time(certificate),
                         )
                     self.on.certificate_available.emit(
                         certificate_signing_request=certificate.csr,
@@ -1805,7 +1924,7 @@ def _on_relation_changed(self, event: RelationChangedEvent) -> None:
                         chain=certificate.chain,
                     )
 
-    def _get_next_secret_expiry_time(self, certificate: str) -> Optional[datetime]:
+    def _get_next_secret_expiry_time(self, certificate: ProviderCertificate) -> Optional[datetime]:
         """Return the expiry time or expiry notification time.
 
         Extracts the expiry time from the provided certificate, calculates the
@@ -1813,17 +1932,18 @@ def _get_next_secret_expiry_time(self, certificate: str) -> Optional[datetime]:
         the future.
 
         Args:
-            certificate: x509 certificate
+            certificate: ProviderCertificate object
 
         Returns:
             Optional[datetime]: None if the certificate expiry time cannot be read,
                                 next expiry time otherwise.
         """
-        expiry_time = _get_certificate_expiry_time(certificate)
-        if not expiry_time:
+        if not certificate.expiry_time or not certificate.expiry_notification_time:
             return None
-        expiry_notification_time = expiry_time - timedelta(hours=self.expiry_notification_time)
-        return _get_closest_future_time(expiry_notification_time, expiry_time)
+        return _get_closest_future_time(
+            certificate.expiry_notification_time,
+            certificate.expiry_time,
+        )
 
     def _on_relation_broken(self, event: RelationBrokenEvent) -> None:
         """Handle Relation Broken Event.
@@ -1857,27 +1977,26 @@ def _on_secret_expired(self, event: SecretExpiredEvent) -> None:
         """
         if not event.secret.label or not event.secret.label.startswith(f"{LIBID}-"):
             return
-        csr = event.secret.label[len(f"{LIBID}-") :]
+        csr = event.secret.get_content()["csr"]
         provider_certificate = self._find_certificate_in_relation_data(csr)
         if not provider_certificate:
             # A secret expired but we did not find matching certificate. Cleaning up
             event.secret.remove_all_revisions()
             return
 
-        expiry_time = _get_certificate_expiry_time(provider_certificate.certificate)
-        if not expiry_time:
+        if not provider_certificate.expiry_time:
             # A secret expired but matching certificate is invalid. Cleaning up
             event.secret.remove_all_revisions()
             return
 
-        if datetime.now(timezone.utc) < expiry_time:
+        if datetime.now(timezone.utc) < provider_certificate.expiry_time:
             logger.warning("Certificate almost expired")
             self.on.certificate_expiring.emit(
                 certificate=provider_certificate.certificate,
-                expiry=expiry_time.isoformat(),
+                expiry=provider_certificate.expiry_time.isoformat(),
             )
             event.secret.set_info(
-                expire=_get_certificate_expiry_time(provider_certificate.certificate),
+                expire=provider_certificate.expiry_time,
             )
         else:
             logger.warning("Certificate is expired")
diff --git a/lib/charms/traefik_k8s/v2/ingress.py b/lib/charms/traefik_k8s/v2/ingress.py
index ed464ea..407cfb5 100644
--- a/lib/charms/traefik_k8s/v2/ingress.py
+++ b/lib/charms/traefik_k8s/v2/ingress.py
@@ -72,7 +72,7 @@ def _on_ingress_revoked(self, event: IngressPerAppRevokedEvent):
 
 # Increment this PATCH version before using `charmcraft publish-lib` or reset
 # to 0 if you are raising the major API version
-LIBPATCH = 12
+LIBPATCH = 13
 
 PYDEPS = ["pydantic"]
 
@@ -590,7 +590,7 @@ def proxied_endpoints(self) -> Dict[str, Dict[str, str]]:
             if PYDANTIC_IS_V1:
                 results[ingress_relation.app.name] = ingress_data.ingress.dict()
             else:
-                results[ingress_relation.app.name] = ingress_data.ingress.model_dump(mode=json)  # type: ignore
+                results[ingress_relation.app.name] = ingress_data.ingress.model_dump(mode="json")
         return results
 
 
diff --git a/lib/charms/traefik_route_k8s/v0/traefik_route.py b/lib/charms/traefik_route_k8s/v0/traefik_route.py
new file mode 100644
index 0000000..de2da55
--- /dev/null
+++ b/lib/charms/traefik_route_k8s/v0/traefik_route.py
@@ -0,0 +1,378 @@
+#!/usr/bin/env python3
+# Copyright 2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+r"""# Interface Library for traefik_route.
+
+This library wraps relation endpoints for traefik_route. The requirer of this
+relation is the traefik-route-k8s charm, or any charm capable of providing
+Traefik configuration files. The provider is the traefik-k8s charm, or another
+charm willing to consume Traefik configuration files.
+
+## Getting Started
+
+To get started using the library, you just need to fetch the library using `charmcraft`.
+
+```shell
+cd some-charm
+charmcraft fetch-lib charms.traefik_route_k8s.v0.traefik_route
+```
+
+To use the library from the provider side (Traefik):
+
+```yaml
+requires:
+    traefik_route:
+        interface: traefik_route
+        limit: 1
+```
+
+```python
+from charms.traefik_route_k8s.v0.traefik_route import TraefikRouteProvider
+
+class TraefikCharm(CharmBase):
+  def __init__(self, *args):
+    # ...
+    self.traefik_route = TraefikRouteProvider(self)
+
+    self.framework.observe(
+        self.traefik_route.on.ready, self._handle_traefik_route_ready
+    )
+
+    def _handle_traefik_route_ready(self, event):
+        config: str = self.traefik_route.get_config(event.relation)  # yaml
+        # use config to configure Traefik
+```
+
+To use the library from the requirer side (TraefikRoute):
+
+```yaml
+requires:
+    traefik-route:
+        interface: traefik_route
+        limit: 1
+        optional: false
+```
+
+```python
+# ...
+from charms.traefik_route_k8s.v0.traefik_route import TraefikRouteRequirer
+
+class TraefikRouteCharm(CharmBase):
+  def __init__(self, *args):
+    # ...
+    traefik_route = TraefikRouteRequirer(
+        self, self.model.relations.get("traefik-route"),
+        "traefik-route"
+    )
+    if traefik_route.is_ready():
+        traefik_route.submit_to_traefik(
+            config={'my': {'traefik': 'configuration'}}
+        )
+
+```
+"""
+import logging
+from typing import Optional
+
+import yaml
+from ops.charm import CharmBase, CharmEvents, RelationEvent
+from ops.framework import EventSource, Object, StoredState
+from ops.model import Relation
+
+# The unique Charmhub library identifier, never change it
+LIBID = "fe2ac43a373949f2bf61383b9f35c83c"
+
+# Increment this major API version when introducing breaking changes
+LIBAPI = 0
+
+# Increment this PATCH version before using `charmcraft publish-lib` or reset
+# to 0 if you are raising the major API version
+LIBPATCH = 10
+
+log = logging.getLogger(__name__)
+
+
+class TraefikRouteException(RuntimeError):
+    """Base class for exceptions raised by TraefikRoute."""
+
+
+class UnauthorizedError(TraefikRouteException):
+    """Raised when the unit needs leadership to perform some action."""
+
+
+class TraefikRouteProviderReadyEvent(RelationEvent):
+    """Event emitted when Traefik is ready to provide ingress for a routed unit."""
+
+
+class TraefikRouteProviderDataRemovedEvent(RelationEvent):
+    """Event emitted when a routed ingress relation is removed."""
+
+
+class TraefikRouteRequirerReadyEvent(RelationEvent):
+    """Event emitted when a unit requesting ingress has provided all data Traefik needs."""
+
+
+class TraefikRouteRequirerEvents(CharmEvents):
+    """Container for TraefikRouteRequirer events."""
+
+    ready = EventSource(TraefikRouteRequirerReadyEvent)
+
+
+class TraefikRouteProviderEvents(CharmEvents):
+    """Container for TraefikRouteProvider events."""
+
+    ready = EventSource(TraefikRouteProviderReadyEvent)  # TODO rename to data_provided in v1
+    data_removed = EventSource(TraefikRouteProviderDataRemovedEvent)
+
+
+class TraefikRouteProvider(Object):
+    """Implementation of the provider of traefik_route.
+
+    This will presumably be owned by a Traefik charm.
+    The main idea is that Traefik will observe the `ready` event and, upon
+    receiving it, will fetch the config from the TraefikRoute's application databag,
+    apply it, and update its own app databag to let Route know that the ingress
+    is there.
+    The TraefikRouteProvider provides api to do this easily.
+    """
+
+    on = TraefikRouteProviderEvents()  # pyright: ignore
+    _stored = StoredState()
+
+    def __init__(
+        self,
+        charm: CharmBase,
+        relation_name: str = "traefik-route",
+        external_host: str = "",
+        *,
+        scheme: str = "http",
+    ):
+        """Constructor for TraefikRouteProvider.
+
+        Args:
+            charm: The charm that is instantiating the instance.
+            relation_name: The name of the relation relation_name to bind to
+                (defaults to "traefik-route").
+            external_host: The external host.
+            scheme: The scheme.
+        """
+        super().__init__(charm, relation_name)
+        self._stored.set_default(external_host=None, scheme=None)
+
+        self._charm = charm
+        self._relation_name = relation_name
+
+        if (
+            self._stored.external_host != external_host  # pyright: ignore
+            or self._stored.scheme != scheme  # pyright: ignore
+        ):
+            # If traefik endpoint details changed, update
+            self.update_traefik_address(external_host=external_host, scheme=scheme)
+
+        self.framework.observe(
+            self._charm.on[relation_name].relation_changed, self._on_relation_changed
+        )
+        self.framework.observe(
+            self._charm.on[relation_name].relation_broken, self._on_relation_broken
+        )
+
+    @property
+    def external_host(self) -> str:
+        """Return the external host set by Traefik, if any."""
+        self._update_stored()
+        return self._stored.external_host or ""  # type: ignore
+
+    @property
+    def scheme(self) -> str:
+        """Return the scheme set by Traefik, if any."""
+        self._update_stored()
+        return self._stored.scheme or ""  # type: ignore
+
+    @property
+    def relations(self):
+        """The list of Relation instances associated with this endpoint."""
+        return list(self._charm.model.relations[self._relation_name])
+
+    def _update_stored(self) -> None:
+        """Ensure that the stored data is up-to-date.
+
+        This is split out into a separate method since, in the case of multi-unit deployments,
+        removal of a `TraefikRouteRequirer` will not cause a `RelationEvent`, but the guard on
+        app data ensures that only the previous leader will know what it is. Separating it
+        allows for reuse both when the property is called and if the relation changes, so a
+        leader change where the new leader checks the property will do the right thing.
+        """
+        if not self._charm.unit.is_leader():
+            return
+
+        for relation in self._charm.model.relations[self._relation_name]:
+            if not relation.app:
+                self._stored.external_host = ""
+                self._stored.scheme = ""
+                return
+            external_host = relation.data[relation.app].get("external_host", "")
+            self._stored.external_host = (
+                external_host or self._stored.external_host  # pyright: ignore
+            )
+            scheme = relation.data[relation.app].get("scheme", "")
+            self._stored.scheme = scheme or self._stored.scheme  # pyright: ignore
+
+    def _on_relation_changed(self, event: RelationEvent):
+        if self.is_ready(event.relation):
+            # todo check data is valid here?
+            self.update_traefik_address()
+            self.on.ready.emit(event.relation)
+
+    def _on_relation_broken(self, event: RelationEvent):
+        self.on.data_removed.emit(event.relation)
+
+    def update_traefik_address(
+        self, *, external_host: Optional[str] = None, scheme: Optional[str] = None
+    ):
+        """Ensure that requirers know the external host for Traefik."""
+        if not self._charm.unit.is_leader():
+            return
+
+        for relation in self._charm.model.relations[self._relation_name]:
+            relation.data[self._charm.app]["external_host"] = external_host or self.external_host
+            relation.data[self._charm.app]["scheme"] = scheme or self.scheme
+
+        # We first attempt to write relation data (which may raise) and only then update stored
+        # state.
+        self._stored.external_host = external_host
+        self._stored.scheme = scheme
+
+    def is_ready(self, relation: Relation) -> bool:
+        """Whether TraefikRoute is ready on this relation.
+
+        Returns True when the remote app shared the config; False otherwise.
+        """
+        if not relation.app or not relation.data[relation.app]:
+            return False
+        return "config" in relation.data[relation.app]
+
+    def get_config(self, relation: Relation) -> Optional[str]:
+        """Renamed to ``get_dynamic_config``."""
+        log.warning("``TraefikRouteProvider.get_config`` is deprecated. "
+                    "Use ``TraefikRouteProvider.get_dynamic_config`` instead")
+        return self.get_dynamic_config(relation)
+
+    def get_dynamic_config(self, relation: Relation) -> Optional[str]:
+        """Retrieve the dynamic config published by the remote application."""
+        if not self.is_ready(relation):
+            return None
+        return relation.data[relation.app].get("config")
+
+    def get_static_config(self, relation: Relation) -> Optional[str]:
+        """Retrieve the static config published by the remote application."""
+        if not self.is_ready(relation):
+            return None
+        return relation.data[relation.app].get("static")
+
+
+class TraefikRouteRequirer(Object):
+    """Wrapper for the requirer side of traefik-route.
+
+    The traefik_route requirer will publish to the application databag an object like:
+    {
+        'config': <Traefik_config>
+        'static': <Traefik_config>  # optional
+    }
+
+    NB: TraefikRouteRequirer does no validation; it assumes that the
+    traefik-route-k8s charm will provide valid yaml-encoded config.
+    The TraefikRouteRequirer provides api to store this config in the
+    application databag.
+    """
+
+    on = TraefikRouteRequirerEvents()  # pyright: ignore
+    _stored = StoredState()
+
+    def __init__(self, charm: CharmBase, relation: Relation, relation_name: str = "traefik-route"):
+        super(TraefikRouteRequirer, self).__init__(charm, relation_name)
+        self._stored.set_default(external_host=None, scheme=None)
+
+        self._charm = charm
+        self._relation = relation
+
+        self.framework.observe(
+            self._charm.on[relation_name].relation_changed, self._on_relation_changed
+        )
+        self.framework.observe(
+            self._charm.on[relation_name].relation_broken, self._on_relation_broken
+        )
+
+    @property
+    def external_host(self) -> str:
+        """Return the external host set by Traefik, if any."""
+        self._update_stored()
+        return self._stored.external_host or ""  # type: ignore
+
+    @property
+    def scheme(self) -> str:
+        """Return the scheme set by Traefik, if any."""
+        self._update_stored()
+        return self._stored.scheme or ""  # type: ignore
+
+    def _update_stored(self) -> None:
+        """Ensure that the stored host is up-to-date.
+
+        This is split out into a separate method since, in the case of multi-unit deployments,
+        removal of a `TraefikRouteRequirer` will not cause a `RelationEvent`, but the guard on
+        app data ensures that only the previous leader will know what it is. Separating it
+        allows for reuse both when the property is called and if the relation changes, so a
+        leader change where the new leader checks the property will do the right thing.
+        """
+        if not self._charm.unit.is_leader():
+            return
+
+        if self._relation:
+            for relation in self._charm.model.relations[self._relation.name]:
+                if not relation.app:
+                    self._stored.external_host = ""
+                    self._stored.scheme = ""
+                    return
+                external_host = relation.data[relation.app].get("external_host", "")
+                self._stored.external_host = (
+                    external_host or self._stored.external_host  # pyright: ignore
+                )
+                scheme = relation.data[relation.app].get("scheme", "")
+                self._stored.scheme = scheme or self._stored.scheme  # pyright: ignore
+
+    def _on_relation_changed(self, event: RelationEvent) -> None:
+        """Update StoredState with external_host and other information from Traefik."""
+        self._update_stored()
+        if self._charm.unit.is_leader():
+            self.on.ready.emit(event.relation)
+
+    def _on_relation_broken(self, event: RelationEvent) -> None:
+        """On RelationBroken, clear the stored data if set and emit an event."""
+        self._stored.external_host = ""
+        if self._charm.unit.is_leader():
+            self.on.ready.emit(event.relation)
+
+    def is_ready(self) -> bool:
+        """Is the TraefikRouteRequirer ready to submit data to Traefik?"""
+        return self._relation is not None
+
+    def submit_to_traefik(self, config: dict, static: dict=None):
+        """Relay an ingress configuration data structure to traefik.
+
+        This will publish to the traefik-route relation databag
+        a chunk of Traefik dynamic config that the traefik charm on the other end can pick
+        up and apply.
+
+        Use ``static`` if you need to update traefik's **static** configuration.
+        Note that this will force traefik to restart to comply.
+        """
+        if not self._charm.unit.is_leader():
+            raise UnauthorizedError()
+
+        app_databag = self._relation.data[self._charm.app]
+
+        # Traefik thrives on yaml, feels pointless to talk json to Route
+        app_databag["config"] = yaml.safe_dump(config)
+
+        if static:
+            app_databag["static"] = yaml.safe_dump(static)
diff --git a/pyproject.toml b/pyproject.toml
index 4614c96..acaf006 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,3 +1,24 @@
+# Copyright 2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+[project]
+name = "tempo-k8s"
+version = "0.1"  # this is in fact irrelevant
+
+[project.optional-dependencies]
+lib_pydeps = [
+]
+
+[tool.pyright]
+extraPaths = ["lib"]
+pythonVersion = "3.8"
+pythonPlatform = "All"
+
+[build]
+build-base = "/tmp/build"
+
+[egg_info]
+egg-base = "/tmp"
+
 # Testing tools configuration
 [tool.coverage.run]
 branch = true
@@ -8,14 +29,18 @@ show_missing = true
 [tool.pytest.ini_options]
 minversion = "6.0"
 log_cli_level = "INFO"
+markers = ["setup", "teardown"]
 
 # Formatting tools configuration
+[tool.isort]
+profile = "black"
+
 [tool.black]
 line-length = 99
 target-version = ["py38"]
 
 # Linting tools configuration
-[tool.ruff]
+[lint]
 line-length = 99
 select = ["E", "W", "F", "C", "N", "D", "I001"]
 extend-ignore = [
@@ -32,19 +57,11 @@ extend-ignore = [
     "D413",
 ]
 ignore = ["E501", "D107"]
-extend-exclude = ["__pycache__", "*.egg_info"]
+extend-exclude = ["__pycache__", "*.egg_info", "*integration/tester*"]
 per-file-ignores = {"tests/*" = ["D100","D101","D102","D103","D104"]}
 
-[tool.ruff.mccabe]
-max-complexity = 15
-
-[tool.ruff.pep8-naming]
-# Allow Pydantic's `@validator` decorator to trigger class method treatment.
-classmethod-decorators = ["classmethod", "pydantic.validator"]
+[lint.mccabe]
+max-complexity = 10
 
-[tool.pyright]
-include = ["src"]
-extraPaths = ["lib"]
-pythonVersion = "3.8"
-pythonPlatform = "Linux"
-reportUnknownParameterType = true
+[tool.codespell]
+skip = "build,lib,venv,icon.svg,.tox,.git,.mypy_cache,.ruff_cache,.vscode,.coverage"
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index cd912f2..af2ddbf 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,8 +1,19 @@
+# pin importlib-metadata version else charmcraft pack will fail to resolve the dependencies for the pydeps-installed otlp library
+importlib-metadata~=6.0.0
 ops
-cosl
-pydantic
-# crossplane is a package from nginxinc to interact with the Nginx config
-crossplane
+jsonschema==4.17.0
+lightkube==0.11.0
+lightkube-models==1.24.1.4
+tenacity==8.2.3
 
+# PYDEPS
 # lib/charms/tempo_k8s/v1/charm_tracing.py
-opentelemetry-exporter-otlp-proto-http
+opentelemetry-exporter-otlp-proto-http==1.21.0
+
+# lib/charms/tls_certificates_interface/v2/tls_certificates.py
+jsonschema
+cryptography
+# lib/charms/tempo_k8s/v1/tracing.py
+pydantic>=2
+# lib/charms/prometheus_k8s/v0/prometheus_scrape.py
+cosl
diff --git a/scripts/generate-traffic.js b/scripts/generate-traffic.js
new file mode 100644
index 0000000..7fc1956
--- /dev/null
+++ b/scripts/generate-traffic.js
@@ -0,0 +1,32 @@
+// This script is intended to be used with [k6](https://k6.io/)
+// If you don't have zinc mapped to your localhost, you can pass
+// the application IP, or ingress IP, using the environment 
+// variable `ZINC_IP`.
+
+
+import http from 'k6/http';
+import { check } from 'k6';
+
+export const options = {
+    scenarios: {
+        rampUp: {
+            executor: 'ramping-vus',
+            startVUs: 0,
+            stages: [
+                { duration: '30s', target: 20 },
+                { duration: '1m', target: 20 },
+                { duration: '30s', target: 100 },
+                { duration: '1m', target: 100 },
+                { duration: '30s', target: 0 },
+            ],
+            gracefulRampDown: '0s'
+        },
+    }
+}
+
+export default function() {
+    const res = http.get(`http://${__ENV.ZINC_IP || 'localhost'}:4080/ui/`);
+    check(res, {
+        'is status 200': (r) => r.status === 200,
+    })
+}
\ No newline at end of file
diff --git a/scripts/tracegen.py b/scripts/tracegen.py
new file mode 100644
index 0000000..3505be4
--- /dev/null
+++ b/scripts/tracegen.py
@@ -0,0 +1,75 @@
+import os
+import time
+from pathlib import Path
+from typing import Any, Literal
+
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter as GRPCExporter
+from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter as HTTPExporter
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import (
+    BatchSpanProcessor,
+    ConsoleSpanExporter,
+)
+
+
+def emit_trace(
+        endpoint: str,
+        log_trace_to_console: bool = False,
+        cert: Path = None,
+        protocol: Literal["grpc", "http", "ALL"] = "grpc",
+        nonce: Any = None
+):
+    os.environ['OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE'] = str(Path(cert).absolute()) if cert else ""
+
+    if protocol == "grpc":
+        span_exporter = GRPCExporter(
+            endpoint=endpoint,
+            insecure=not cert,
+        )
+    elif protocol == "http":
+        span_exporter = HTTPExporter(
+            endpoint=endpoint,
+        )
+    else:  # ALL
+        return (emit_trace(endpoint, log_trace_to_console, cert, "grpc", nonce=nonce) and
+                emit_trace(endpoint, log_trace_to_console, cert, "http", nonce=nonce))
+
+    return _export_trace(span_exporter, log_trace_to_console=log_trace_to_console, nonce=nonce)
+
+
+def _export_trace(span_exporter, log_trace_to_console: bool = False, nonce: Any = None):
+    resource = Resource.create(attributes={
+        "service.name": "tracegen",
+        "nonce": str(nonce)
+    }
+    )
+    provider = TracerProvider(resource=resource)
+
+    if log_trace_to_console:
+        processor = BatchSpanProcessor(ConsoleSpanExporter())
+        provider.add_span_processor(processor)
+
+    span_processor = BatchSpanProcessor(span_exporter)
+    provider.add_span_processor(span_processor)
+    trace.set_tracer_provider(provider)
+
+    tracer = trace.get_tracer(__name__)
+
+    with tracer.start_as_current_span("foo"):
+        with tracer.start_as_current_span("bar"):
+            with tracer.start_as_current_span("baz"):
+                time.sleep(.1)
+
+    return span_exporter.force_flush()
+
+
+if __name__ == '__main__':
+    emit_trace(
+        endpoint=os.getenv("TRACEGEN_ENDPOINT", "http://127.0.0.1:8080"),
+        cert=os.getenv("TRACEGEN_CERT", None),
+        log_trace_to_console=os.getenv("TRACEGEN_VERBOSE", False),
+        protocol=os.getenv("TRACEGEN_PROTOCOL", "http"),
+        nonce=os.getenv("TRACEGEN_NONCE", "24")
+    )
diff --git a/src/charm.py b/src/charm.py
index 87e1412..1787004 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -1,316 +1,604 @@
 #!/usr/bin/env python3
-# Copyright 2023 Canonical
+# Copyright 2024 Canonical Ltd.
 # See LICENSE file for licensing details.
 
-"""Charm the service.
-
-Refer to the following post for a quick-start guide that will help you
-develop a new k8s charm using the Operator Framework:
-
-https://discourse.charmhub.io/t/4208
-"""
-import glob
+"""Charmed Operator for Tempo; a lightweight object storage based tracing backend."""
 import json
 import logging
-import os
-import shutil
+import re
 import socket
-import subprocess
 from pathlib import Path
-from typing import Any, Dict, List, Optional
+from typing import Dict, List
+from typing import Optional, Set, Tuple
 
 import ops
-import yaml
-from charms.data_platform_libs.v0.s3 import (
-    S3Requirer,
+from ops.charm import (
+    CharmBase,
+    CollectStatusEvent,
+    PebbleNoticeEvent,
+    RelationEvent,
+    WorkloadEvent,
 )
+from ops.main import main
+from ops.model import ActiveStatus, BlockedStatus, MaintenanceStatus, WaitingStatus
+from ops.model import Relation
+
+from charms.data_platform_libs.v0.s3 import S3Requirer
 from charms.grafana_k8s.v0.grafana_dashboard import GrafanaDashboardProvider
 from charms.grafana_k8s.v0.grafana_source import GrafanaSourceProvider
-from charms.loki_k8s.v1.loki_push_api import LokiPushApiConsumer
-from charms.observability_libs.v1.cert_handler import CertHandler
+from charms.loki_k8s.v0.loki_push_api import LogProxyConsumer
+from charms.observability_libs.v0.kubernetes_service_patch import KubernetesServicePatch
+from charms.observability_libs.v1.cert_handler import CertHandler, VAULT_SECRET_LABEL
 from charms.prometheus_k8s.v0.prometheus_scrape import MetricsEndpointProvider
-from charms.prometheus_k8s.v1.prometheus_remote_write import PrometheusRemoteWriteProvider
 from charms.tempo_k8s.v1.charm_tracing import trace_charm
-from charms.tempo_k8s.v1.tracing import TracingEndpointRequirer
-from charms.traefik_k8s.v2.ingress import IngressPerAppReadyEvent, IngressPerAppRequirer
-from cosl import JujuTopology
-from cosl.rules import AlertRules
+from charms.tempo_k8s.v2.tracing import (
+    ReceiverProtocol,
+    RequestEvent,
+    TracingEndpointProvider,
+)
+from charms.traefik_route_k8s.v0.traefik_route import TraefikRouteRequirer
+from coordinator import TempoCoordinator
+from tempo import Tempo
 from tempo_cluster import TempoClusterProvider
-from tempo_config import BUCKET_NAME, S3_RELATION_NAME, _S3ConfigData
-from tempo_coordinator import TempoCoordinator
-from nginx import CA_CERT_PATH, CERT_PATH, KEY_PATH, Nginx
-from nginx_prometheus_exporter import NGINX_PROMETHEUS_EXPORTER_PORT, NginxPrometheusExporter
-from ops.charm import CollectStatusEvent
-from ops.model import ModelError, Relation
-from pydantic import ValidationError
-
-# Log messages can be retrieved using juju debug-log
-logger = logging.getLogger(__name__)
 
-NGINX_ORIGINAL_ALERT_RULES_PATH = "./src/prometheus_alert_rules/nginx"
-WORKER_ORIGINAL_ALERT_RULES_PATH = "./src/prometheus_alert_rules/tempo_workers"
-CONSOLIDATED_ALERT_RULES_PATH = "./src/prometheus_alert_rules/consolidated_rules"
+logger = logging.getLogger(__name__)
 
 
 @trace_charm(
-    tracing_endpoint="tempo_endpoint",
-    server_cert="server_cert_path",
-    extra_types=[
-        S3Requirer,
-        TempoClusterProvider,
-        TempoCoordinator,
-        Nginx,
-    ],
+    tracing_endpoint="tempo_otlp_http_endpoint",
+    server_cert="server_cert",
+    extra_types=(Tempo, TracingEndpointProvider),
 )
-class TempoCoordinatorK8SOperatorCharm(ops.CharmBase):
-    """Charm the service."""
+class TempoCharm(CharmBase):
+    """Charmed Operator for Tempo; a distributed tracing backend."""
 
-    def __init__(self, *args: Any):
+    def __init__(self, *args):
         super().__init__(*args)
-
-        # TODO: On any worker relation-joined/departed, need to updade grafana agent's scrape
-        #  targets with the new memberlist.
-        #  (Remote write would still be the same nginx-proxied endpoint.)
-        self._nginx_container = self.unit.get_container("nginx")
-        self._nginx_prometheus_exporter_container = self.unit.get_container(
-            "nginx-prometheus-exporter"
+        self.ingress = TraefikRouteRequirer(self, self.model.get_relation("ingress"), "ingress")  # type: ignore
+        self.tempo_cluster = TempoClusterProvider(self)
+        self.coordinator = TempoCoordinator(self.tempo_cluster,
+                                            is_worker=self.is_worker_node)
+
+        self.tempo = tempo = Tempo(
+            self.unit.get_container("tempo"),
+            external_host=self.hostname,
+            # we need otlp_http receiver for charm_tracing
+            enable_receivers=["otlp_http"],
+            run_worker_node=self.is_worker_node
         )
-        self.server_cert = CertHandler(
-            charm=self,
+
+        self.cert_handler = CertHandler(
+            self,
             key="tempo-server-cert",
             sans=[self.hostname],
         )
-        self.s3_requirer = S3Requirer(self, S3_RELATION_NAME, BUCKET_NAME)
-        self.cluster_provider = TempoClusterProvider(self)
-        self.coordinator = TempoCoordinator(
-            cluster_provider=self.cluster_provider,
-            tls_requirer=self.server_cert,
-        )
-        self.nginx = Nginx(
+
+        self.s3_requirer = S3Requirer(self, Tempo.s3_relation_name, Tempo.s3_bucket_name)
+
+        # configure this tempo as a datasource in grafana
+        self.grafana_source_provider = GrafanaSourceProvider(
             self,
-            cluster_provider=self.cluster_provider,
-            server_name=self.hostname,
+            source_type="tempo",
+            source_url=self._external_http_server_url,
+            refresh_event=[
+                # refresh the source url when TLS config might be changing
+                self.on[self.cert_handler.certificates_relation_name].relation_changed,
+                # or when ingress changes
+                self.ingress.on.ready,
+            ],
         )
-        self.nginx_prometheus_exporter = NginxPrometheusExporter(self)
-        self.remote_write_provider = PrometheusRemoteWriteProvider(
-            charm=self,
-            server_url_func=lambda: TempoCoordinatorK8SOperatorCharm.external_url.fget(self),  # type: ignore
-            endpoint_path="/api/v1/push",
+        # # Patch the juju-created Kubernetes service to contain the right ports
+        external_ports = tempo.get_external_ports(self.app.name)
+        self._service_patcher = KubernetesServicePatch(self, external_ports)
+        # Provide ability for Tempo to be scraped by Prometheus using prometheus_scrape
+        self._scraping = MetricsEndpointProvider(
+            self,
+            relation_name="metrics-endpoint",
+            jobs=[{"static_configs": [{"targets": [f"*:{tempo.tempo_http_server_port}"]}]}],
         )
-        self.tracing = TracingEndpointRequirer(self)
-        self.grafana_dashboard_provider = GrafanaDashboardProvider(
-            self, relation_name="grafana-dashboards-provider"
+        # Enable log forwarding for Loki and other charms that implement loki_push_api
+        self._logging = LogProxyConsumer(
+            self, relation_name="logging", log_files=[self.tempo.log_path], container_name="tempo"
         )
-        self.grafana_source = GrafanaSourceProvider(
-            self,
-            source_type="prometheus",
-            source_port="8080",
-            source_url=f"{self.cluster_provider.get_datasource_address()}:8080/prometheus",
-            extra_fields={"httpHeaderName1": "X-Scope-OrgID"},
-            secure_extra_fields={"httpHeaderValue1": "anonymous"},
+        self._grafana_dashboards = GrafanaDashboardProvider(
+            self, relation_name="grafana-dashboard"
         )
-        self.loki_consumer = LokiPushApiConsumer(self, relation_name="logging-consumer")
 
-        self._consolidate_nginx_rules()
-        self.metrics_endpoints = MetricsEndpointProvider(
-            self,
-            relation_name="self-metrics-endpoint",
-            alert_rules_path=CONSOLIDATED_ALERT_RULES_PATH,
-            jobs=self._scrape_jobs,
-            refresh_event=[
-                self.on.tempo_cluster_relation_joined,
-                self.on.tempo_cluster_relation_changed,
-                self.on.tempo_cluster_relation_departed,
-                self.on.tempo_cluster_relation_broken,
-            ],
+        self.tracing = TracingEndpointProvider(self, external_url=self._external_url)
+        self._inconsistencies = self.coordinator.get_deployment_inconsistencies(
+            clustered=self.is_clustered,
+            scaled=self.is_scaled,
+            has_workers=self.tempo_cluster.has_workers,
+            is_worker_node=self.is_worker_node,
+            has_s3=self.is_s3_ready
         )
-        self.ingress = IngressPerAppRequirer(charm=self, strip_prefix=True)
+        self._is_consistent = not self._inconsistencies
+
+        if not self._is_consistent:
+            logger.error(
+                f"Inconsistent deployment. {self.unit.name} will be shutting down. "
+                "This likely means you need to add an s3 integration. "
+                "This charm will be unresponsive and refuse to handle any event until "
+                "the situation is resolved by the cloud admin, to avoid data loss."
+            )
+            self.framework.observe(self.on.collect_unit_status, self._on_collect_unit_status)
+
+            if self.tempo.is_tempo_service_defined:
+                self.tempo.shutdown()
 
-        ######################################
-        # === EVENT HANDLER REGISTRATION === #
-        ######################################
+            return  # refuse to handle any other event as we can't possibly know what to do.
+
+        # lifecycle
+        self.framework.observe(self.on.leader_elected, self._on_leader_elected)
+        self.framework.observe(self.on.leader_settings_changed, self._on_leader_settings_changed)
+        self.framework.observe(self.on.update_status, self._on_update_status)
+        self.framework.observe(self.on.collect_unit_status, self._on_collect_unit_status)
         self.framework.observe(self.on.config_changed, self._on_config_changed)
-        self.framework.observe(self.on.collect_unit_status, self._on_collect_status)
-        self.framework.observe(self.on.nginx_pebble_ready, self._on_nginx_pebble_ready)
-        self.framework.observe(
-            self.on.nginx_prometheus_exporter_pebble_ready,
-            self._on_nginx_prometheus_exporter_pebble_ready,
-        )
-        self.framework.observe(self.server_cert.on.cert_changed, self._on_server_cert_changed)
-        # Tempo Cluster
-        self.framework.observe(
-            self.on.tempo_cluster_relation_joined, self._on_tempo_cluster_joined
-        )
+        self.framework.observe(self.on.list_receivers_action, self._on_list_receivers_action)
+
+        # ingress
+        ingress = self.on["ingress"]
+        self.framework.observe(ingress.relation_created, self._on_ingress_relation_created)
+        self.framework.observe(ingress.relation_joined, self._on_ingress_relation_joined)
+        self.framework.observe(self.ingress.on.ready, self._on_ingress_ready)
+
+        # workload
+        self.framework.observe(self.on.tempo_pebble_ready, self._on_tempo_pebble_ready)
         self.framework.observe(
-            self.on.tempo_cluster_relation_changed, self._on_tempo_cluster_changed
+            self.on.tempo_pebble_custom_notice, self._on_tempo_pebble_custom_notice
         )
+
+        # s3
         self.framework.observe(
-            self.on.tempo_cluster_relation_departed, self._on_tempo_cluster_changed
+            self.s3_requirer.on.credentials_changed, self._on_s3_credentials_changed
         )
+        self.framework.observe(self.s3_requirer.on.credentials_gone, self._on_s3_credentials_gone)
+
+        # tracing
+        self.framework.observe(self.tracing.on.request, self._on_tracing_request)
+        self.framework.observe(self.tracing.on.broken, self._on_tracing_broken)
         self.framework.observe(
-            self.on.tempo_cluster_relation_broken, self._on_tempo_cluster_changed
+            self.on.tempo_peers_relation_created, self._on_tempo_peers_relation_created
         )
-        # S3 Requirer
-        self.framework.observe(self.s3_requirer.on.credentials_changed, self._on_s3_changed)
-        self.framework.observe(self.s3_requirer.on.credentials_gone, self._on_s3_changed)
-        # Self-monitoring
         self.framework.observe(
-            self.loki_consumer.on.loki_push_api_endpoint_joined, self._on_loki_relation_changed
+            self.on.tempo_peers_relation_changed, self._on_tempo_peers_relation_changed
         )
-        self.framework.observe(
-            self.loki_consumer.on.loki_push_api_endpoint_departed, self._on_loki_relation_changed
+
+        # tls
+        self.framework.observe(self.cert_handler.on.cert_changed, self._on_cert_handler_changed)
+
+        # cluster
+        self.framework.observe(self.tempo_cluster.on.changed, self._on_tempo_cluster_changed)
+
+    ######################
+    # UTILITY PROPERTIES #
+    ######################
+
+    @property
+    def is_worker_node(self) -> bool:
+        """Check whether this Tempo charm is configured to run a worker node."""
+        if self.is_clustered:
+            return self.config.get("coordinator_runs_workload_when_clustered", True)
+        return True
+
+    @property
+    def is_scaled(self) -> bool:
+        """Check whether Tempo is deployed with scale > 1."""
+        relation = self.model.get_relation("tempo-peers")
+        if not relation:
+            return False
+        # does not include self
+        return len(relation.units) > 0
+
+    @property
+    def is_clustered(self) -> bool:
+        """Check whether this Tempo is a coordinator and has worker nodes connected to it."""
+        return self.tempo_cluster.has_workers
+
+    @property
+    def is_s3_ready(self) -> bool:
+        # we have an s3 config
+
+        # we cannot check for self.tempo.can_scale() here, because if we are handling a s3-changed
+        # event, it may be that we have a s3 config which is not yet on disk (it will be put there by
+        # our on_s3_changed event handler momentarily).
+        return bool(self._s3_config)
+
+    @property
+    def hostname(self) -> str:
+        """Unit's hostname."""
+        return socket.getfqdn()
+
+    @property
+    def _external_http_server_url(self) -> str:
+        """External url of the http(s) server."""
+        return f"{self._external_url}:{self.tempo.tempo_http_server_port}"
+
+    @property
+    def _external_url(self) -> str:
+        """Return the external url."""
+        if self.ingress.is_ready():
+            ingress_url = f"{self.ingress.scheme}://{self.ingress.external_host}"
+            logger.debug("This unit's ingress URL: %s", ingress_url)
+            return ingress_url
+
+        # If we do not have an ingress, then use the pod hostname.
+        # The reason to prefer this over the pod name (which is the actual
+        # hostname visible from the pod) or a K8s service, is that those
+        # are routable virtually exclusively inside the cluster (as they rely)
+        # on the cluster's DNS service, while the ip address is _sometimes_
+        # routable from the outside, e.g., when deploying on MicroK8s on Linux.
+        return self._internal_url
+
+    @property
+    def _internal_url(self) -> str:
+        scheme = "https" if self.tls_available else "http"
+        return f"{scheme}://{self.hostname}"
+
+    @property
+    def tls_available(self) -> bool:
+        """Return True if tls is enabled and the necessary certs are found."""
+        return (
+                self.cert_handler.enabled
+                and (self.cert_handler.server_cert is not None)
+                and (self.cert_handler.private_key is not None)
+                and (self.cert_handler.ca_cert is not None)
         )
-        self.framework.observe(self.ingress.on.ready, self._on_ingress_ready)
-        self.framework.observe(self.ingress.on.revoked, self._on_ingress_revoked)
 
-    ##########################
-    # === EVENT HANDLERS === #
-    ##########################
+    @property
+    def _s3_config(self) -> Optional[dict]:
+        if not self.s3_requirer.relations:
+            return None
+        s3_config = self.s3_requirer.get_s3_connection_info()
+        if (
+                s3_config
+                and "bucket" in s3_config
+                and "endpoint" in s3_config
+                and "access-key" in s3_config
+                and "secret-key" in s3_config
+        ):
+            return s3_config
+        return None
+
+    ##################
+    # EVENT HANDLERS #
+    ##################
+    def _on_tracing_broken(self, _):
+        """Update tracing relations' databags once one relation is removed."""
+        self._update_tracing_relations()
+
+    def _on_cert_handler_changed(self, _):
+        was_ready = self.tempo.tls_ready
+
+        if self.tls_available:
+            logger.debug("enabling TLS")
+            self.tempo.configure_tls(
+                cert=self.cert_handler.server_cert,  # type: ignore
+                key=self.cert_handler.private_key,  # type: ignore
+                ca=self.cert_handler.ca_cert,  # type: ignore
+            )
+        else:
+            logger.debug("disabling TLS")
+            self.tempo.clear_tls_config()
 
-    def _on_config_changed(self, _: ops.ConfigChangedEvent):
-        """Handle changed configuration."""
-        self.nginx.configure_pebble_layer(tls=self._is_tls_ready)
-        self._render_workers_alert_rules()
-        self._update_tempo_cluster()
+        if was_ready != self.tempo.tls_ready:
+            # tls readiness change means config change.
+            self._update_tempo_config()
+            # sync scheme change with traefik and related consumers
+            self._configure_ingress()
 
-    def _on_server_cert_changed(self, _):
-        self._update_cert()
-        self.nginx.configure_pebble_layer(tls=self._is_tls_ready)
-        self._update_tempo_cluster()
+            if self.tempo.is_tempo_service_defined:
+                self.tempo.restart()
 
-    def _on_tempo_cluster_joined(self, _):
-        self.nginx.configure_pebble_layer(tls=self._is_tls_ready)
-        self._render_workers_alert_rules()
-        self._update_tempo_cluster()
+        # sync the server cert with the charm container.
+        # technically, because of charm tracing, this will be called first thing on each event
+        self._update_server_cert()
 
-    def _on_tempo_cluster_changed(self, _):
-        self.nginx.configure_pebble_layer(tls=self._is_tls_ready)
-        self._render_workers_alert_rules()
-        self._update_tempo_cluster()
+        # update relations to reflect the new certificate
+        self._update_tracing_relations()
 
-    def _on_tempo_cluster_departed(self, _):
-        self.nginx.configure_pebble_layer(tls=self._is_tls_ready)
-        self._render_workers_alert_rules()
+        # notify the cluster
         self._update_tempo_cluster()
 
-    def _on_s3_changed(self, _):
+    def _on_tracing_request(self, e: RequestEvent):
+        """Handle a remote requesting a tracing endpoint."""
+        logger.debug(f"received tracing request from {e.relation.app}: {e.requested_receivers}")
+        self._update_tracing_relations()
+
+    def _on_tempo_cluster_changed(self, _: RelationEvent):
         self._update_tempo_cluster()
 
-    def _on_collect_status(self, event: CollectStatusEvent):
-        """Handle start event."""
-        if not self.coordinator.is_coherent():
-            missing_roles = [role.value for role in self.coordinator.missing_roles()]
-            event.add_status(
-                ops.BlockedStatus(
-                    f"Incoherent deployment: you are lacking some required Tempo roles "
-                    f"({missing_roles})"
-                )
-            )
-        s3_config_data = self._get_s3_storage_config()
-        if not s3_config_data and self.has_multiple_workers():
-            event.add_status(
-                ops.BlockedStatus(
-                    "When multiple units of Tempo are deployed, you must add a valid S3 relation. S3 relation missing/invalid."
-                )
+    def _on_ingress_relation_created(self, _: RelationEvent):
+        self._configure_ingress()
+
+    def _on_ingress_relation_joined(self, _: RelationEvent):
+        self._configure_ingress()
+
+    def _on_leader_settings_changed(self, _: ops.LeaderSettingsChangedEvent):
+        if not self.is_s3_ready:
+            logger.error(
+                "Losing leadership without s3. " "This unit will soon be in an inconsistent state."
             )
 
-        if self.coordinator.is_recommended():
-            logger.warning("This deployment is below the recommended deployment requirement.")
-            event.add_status(ops.ActiveStatus("degraded"))
+    def _on_leader_elected(self, _: ops.LeaderElectedEvent):
+        # as traefik_route goes through app data, we need to take lead of traefik_route if our leader dies.
+        self._configure_ingress()
+
+    def _on_s3_credentials_changed(self, _):
+        self._on_s3_changed()
+
+    def _on_s3_credentials_gone(self, _):
+        self._on_s3_changed()
+
+    def _on_s3_changed(self):
+        could_scale_before = self.tempo.can_scale()
+
+        self._update_tempo_config()
+
+        can_scale_now = self.tempo.can_scale()
+        # if we had s3, and we don't anymore, we need to replan from 'scaling-monolithic' to 'all'
+        # if we didn't have s3, and now we do, we can replan from 'all' to 'scaling-monolithic'
+        if could_scale_before != can_scale_now:
+            if not self.tempo.is_tempo_service_defined:
+                # TODO: should we be deferring this to the next pebble-ready instead?
+                logger.debug("tempo was not running! Starting it now")
+            self.tempo.plan()
+
+        self._update_tempo_cluster()
+
+    def _on_tempo_peers_relation_created(self, event: ops.RelationCreatedEvent):
+        if self._local_ip:
+            event.relation.data[self.unit]["local-ip"] = self._local_ip
+
+    def _on_tempo_peers_relation_changed(self, _):
+        if self._update_tempo_config():
+            self.tempo.restart()
+
+    def _update_tempo_config(self) -> bool:
+        peers = self.peers()
+        relation = self.model.get_relation("tempo-peers")
+        # get unit addresses for all the other units from a databag
+        if peers and relation:
+            addresses = [relation.data[unit].get("local-ip") for unit in peers]
+            addresses = list(filter(None, addresses))
         else:
-            event.add_status(ops.ActiveStatus())
+            addresses = []
+
+        # add own address
+        if self._local_ip:
+            addresses.append(self._local_ip)
+
+        return self.tempo.update_config(self._requested_receivers(), self._s3_config, addresses)
+
+    @property
+    def _local_ip(self) -> Optional[str]:
+        try:
+            return str(self.model.get_binding('tempo-peers').network.bind_address)
+        except (ops.ModelError, KeyError) as e:
+            logger.debug("failed to obtain local ip from tempo-peers binding", exc_info=True)
+            logger.error(f"unable to get local IP at this time: failed with {type(e)}; "
+                         f"see debug log for more info")
+            return None
+
+    def _on_config_changed(self, _):
+        # check if certificate files haven't disappeared and recreate them if needed
+        if self.tls_available and not self.tempo.tls_ready:
+            logger.debug("enabling TLS")
+            self.tempo.configure_tls(
+                cert=self.cert_handler.server_cert,  # type: ignore
+                key=self.cert_handler.private_key,  # type: ignore
+                ca=self.cert_handler.ca_cert,  # type: ignore
+            )
 
-    def _on_loki_relation_changed(self, _):
         self._update_tempo_cluster()
 
-    def _on_nginx_pebble_ready(self, _) -> None:
-        self.nginx.configure_pebble_layer(tls=self._is_tls_ready)
+    def _on_tempo_pebble_custom_notice(self, event: PebbleNoticeEvent):
+        if event.notice.key == self.tempo.tempo_ready_notice_key:
+            logger.debug("pebble api reported ready")
+            # collect-unit-status should do the rest and report that pebble is ready.
+            self.tempo.receive_tempo_ready_notice()
 
-    def _on_nginx_prometheus_exporter_pebble_ready(self, _) -> None:
-        self.nginx_prometheus_exporter.configure_pebble_layer()
+    def _on_tempo_pebble_ready(self, event: WorkloadEvent):
+        if not self.tempo.container.can_connect():
+            logger.warning("container not ready, cannot configure; will retry soon")
+            return event.defer()
 
-    def _on_ingress_ready(self, event: IngressPerAppReadyEvent):
-        """Log the obtained ingress address.
+        self.tempo.update_config(self._requested_receivers(), self._s3_config)
+        self.tempo.plan()
 
-        This event refreshes the PrometheusRemoteWriteProvider address.
-        """
-        logger.info("Ingress for app ready on '%s'", event.url)
+        self.unit.set_workload_version(self.version)
+        self.unit.status = ActiveStatus()
 
-    def _on_ingress_revoked(self, _) -> None:
-        """Log the ingress address being revoked.
+    def _on_update_status(self, _):
+        """Update the status of the application."""
+        self.unit.set_workload_version(self.version)
 
-        This event refreshes the PrometheusRemoteWriteProvider address.
-        """
-        logger.info("Ingress for app revoked")
+    def _on_ingress_ready(self, _event):
+        # whenever there's a change in ingress, we need to update all tracing relations
+        self._update_tracing_relations()
 
-    ######################
-    # === PROPERTIES === #
-    ######################
+    def _on_ingress_revoked(self, _event):
+        # whenever there's a change in ingress, we need to update all tracing relations
+        self._update_tracing_relations()
 
-    @property
-    def hostname(self) -> str:
-        """Unit's hostname."""
-        return socket.getfqdn()
+    def _on_list_receivers_action(self, event: ops.ActionEvent):
+        res = {}
+        for receiver in self._requested_receivers():
+            res[receiver.replace("_", "-")] = (
+                f"{self.ingress.external_host or self.tempo.url}:{self.tempo.receiver_ports[receiver]}"
+            )
+        event.set_results(res)
+
+    # keep this event handler at the bottom
+    def _on_collect_unit_status(self, e: CollectStatusEvent):
+        # todo add [nginx.workload] statuses
+
+        if not self.tempo.container.can_connect():
+            e.add_status(WaitingStatus("[workload.tempo] Tempo container not ready"))
+        if not self.tempo.is_ready():
+            e.add_status(WaitingStatus("[workload.tempo] Tempo API not ready just yet..."))
+
+        # todo: how to surface this inconsistent state?
+        # if not self.tempo.can_scale() and self.is_s3_ready):
+        #     e.add_status(BlockedStatus("[s3] s3 ready but tempo not configured."))
+
+        # TODO: should we set these statuses on the leader only, or on all units?
+        if issues := self._inconsistencies:
+            for issue in issues:
+                e.add_status(
+                    BlockedStatus("[consistency.issues]" + issue)
+                )
+            e.add_status(
+                BlockedStatus(
+                    "[consistency] Unit *disabled*."
+                )
+            )
+        else:
+            if self.is_clustered:
+                # no issues: tempo is consistent
+                if not self.coordinator.is_recommended:
+                    e.add_status(ActiveStatus("[coordinator] degraded"))
+                else:
+                    e.add_status(ActiveStatus())
+            else:
+                e.add_status(ActiveStatus())
+
+    ###################
+    # UTILITY METHODS #
+    ###################
+    def _configure_ingress(self) -> None:
+        """Make sure the traefik route and tracing relation data are up-to-date."""
+        if not self.unit.is_leader():
+            return
 
-    @property
-    def _is_cert_available(self) -> bool:
-        return (
-            self.server_cert.enabled
-            and (self.server_cert.server_cert is not None)
-            and (self.server_cert.private_key is not None)
-            and (self.server_cert.ca_cert is not None)
-        )
+        if self.ingress.is_ready():
+            self.ingress.submit_to_traefik(
+                self._ingress_config, static=self._static_ingress_config
+            )
+            if self.ingress.external_host:
+                self._update_tracing_relations()
+
+    def _update_tracing_relations(self):
+        tracing_relations = self.model.relations["tracing"]
+        if not tracing_relations:
+            # todo: set waiting status and configure tempo to run without receivers if possible,
+            #  else perhaps postpone starting the workload at all.
+            logger.warning("no tracing relations: Tempo has no receivers configured.")
+            return
 
-    @property
-    def _is_tls_ready(self) -> bool:
-        return (
-            self._nginx_container.can_connect()
-            and self._nginx_container.exists(CERT_PATH)
-            and self._nginx_container.exists(KEY_PATH)
-            and self._nginx_container.exists(CA_CERT_PATH)
-        )
+        requested_receivers = self._requested_receivers()
+        # publish requested protocols to all relations
+        if self.unit.is_leader():
+            self.tracing.publish_receivers(
+                [(p, self.tempo.get_receiver_url(p, self.ingress)) for p in requested_receivers]
+            )
 
-    @property
-    def tempo_worker_relations(self) -> List[ops.Relation]:
-        """Returns the list of worker relations."""
-        return self.model.relations.get("tempo_worker", [])
+        self._restart_if_receivers_changed()
 
-    @property
-    def _workers_scrape_jobs(self) -> List[Dict[str, Any]]:
-        scrape_jobs = []
-        worker_topologies = self.cluster_provider.gather_topology()
-
-        for worker in worker_topologies:
-            job = {
-                "static_configs": [
-                    {
-                        "targets": [f"{worker['address']}:8080"],
-                    }
-                ],
-                # setting these as "labels" in the static config gets some of them
-                # replaced by the coordinator topology
-                # https://github.com/canonical/prometheus-k8s-operator/issues/571
-                "relabel_configs": [
-                    {"target_label": "juju_charm", "replacement": "tempo-worker-k8s"},
-                    {"target_label": "juju_unit", "replacement": worker["unit"]},
-                    {"target_label": "juju_application", "replacement": worker["app"]},
-                    {"target_label": "juju_model", "replacement": self.model.name},
-                    {"target_label": "juju_model_uuid", "replacement": self.model.uuid},
-                ],
-            }
-            scrape_jobs.append(job)
-        return scrape_jobs
+        self._update_tempo_cluster()
 
-    @property
-    def _nginx_scrape_jobs(self) -> List[Dict[str, Any]]:
-        job: Dict[str, Any] = {
-            "static_configs": [{"targets": [f"{self.hostname}:{NGINX_PROMETHEUS_EXPORTER_PORT}"]}]
-        }
-        return [job]
+    def _restart_if_receivers_changed(self):
+        # if the receivers have changed, we need to reconfigure tempo
+        self.unit.status = MaintenanceStatus("reconfiguring Tempo...")
+        updated = self._update_tempo_config()
+        if not updated:
+            logger.debug("Config not updated; skipping tempo restart")
+        if updated:
+            restarted = self.tempo.is_tempo_service_defined and self.tempo.restart()
+            if not restarted:
+                # assume that this will be handled at the next pebble-ready
+                logger.debug("Cannot reconfigure/restart tempo at this time.")
+
+    def _requested_receivers(self) -> Tuple[ReceiverProtocol, ...]:
+        """List what receivers we should activate, based on the active tracing relations."""
+        # we start with the sum of the requested endpoints from the requirers
+        requested_protocols = set(self.tracing.requested_protocols())
+
+        # and publish only those we support
+        requested_receivers = requested_protocols.intersection(set(self.tempo.receiver_ports))
+        requested_receivers.update(self.tempo.enabled_receivers)
+        return tuple(requested_receivers)
 
     @property
-    def _scrape_jobs(self) -> List[Dict[str, Any]]:
-        return self._workers_scrape_jobs + self._nginx_scrape_jobs
+    def version(self) -> str:
+        """Reports the current Tempo version."""
+        container = self.unit.get_container("tempo")
+        if container.can_connect() and container.get_services("tempo"):
+            try:
+                return self._get_version() or ""
+            # Catching Exception is not ideal, but we don't care much for the error here, and just
+            # default to setting a blank version since there isn't much the admin can do!
+            except Exception as e:
+                logger.warning("unable to get version from API: %s", str(e))
+                logger.debug(e, exc_info=True)
+                return ""
+        return ""
+
+    def _get_version(self) -> Optional[str]:
+        """Fetch the version from the running workload using the Tempo CLI.
+
+        Helper function.
+        """
+        container = self.unit.get_container("tempo")
+        proc = container.exec(["/tempo", "-version"])
+        out, err = proc.wait_output()
+
+        # example output:
+        # / # /tempo --version
+        # tempo, version  (branch: HEAD, revision: fd5743d5d)
+        #   build user:
+        #   build date:
+        #   go version:       go1.18.5
+        #   platform:         linux/amd64
+
+        if version_head := re.search(r"tempo, version (.*) \(branch: (.*), revision: (.*)\)", out):
+            v_head, b_head, r_head = version_head.groups()
+            version = f"{v_head}:{b_head}/{r_head}"
+        elif version_headless := re.search(r"tempo, version (\S+)", out):
+            version = version_headless.groups()[0]
+        else:
+            logger.warning(
+                f"unable to determine tempo workload version: output {out} "
+                f"does not match any known pattern"
+            )
+            return
+        return version
+
+    def server_cert(self):
+        """For charm tracing."""
+        self._update_server_cert()
+        return self.tempo.server_cert_path
+
+    def _update_server_cert(self):
+        """Server certificate for charm tracing tls, if tls is enabled."""
+        server_cert = Path(self.tempo.server_cert_path)
+        if self.tls_available:
+            if not server_cert.exists():
+                server_cert.parent.mkdir(parents=True, exist_ok=True)
+                if self.cert_handler.server_cert:
+                    server_cert.write_text(self.cert_handler.server_cert)
+        else:  # tls unavailable: delete local cert
+            server_cert.unlink(missing_ok=True)
+
+    def tempo_otlp_http_endpoint(self) -> Optional[str]:
+        """Endpoint at which the charm tracing information will be forwarded."""
+        # the charm container and the tempo workload container have apparently the same
+        # IP, so we can talk to tempo at localhost.
+        if self.tempo.is_ready():
+            return f"{self._internal_url}:{self.tempo.receiver_ports['otlp_http']}"
+
+        return None
+
+    def peers(self) -> Optional[Set[ops.model.Unit]]:
+        relation = self.model.get_relation("tempo-peers")
+        if not relation:
+            return None
+
+        # self is not included in relation.units
+        return relation.units
+
+    def _is_s3_ready(self) -> bool:
+        return bool(self._s3_config)
 
     @property
     def loki_endpoints_by_unit(self) -> Dict[str, str]:
@@ -337,175 +625,69 @@ def loki_endpoints_by_unit(self) -> Dict[str, str]:
 
         return endpoints
 
-    @property
-    def tempo_endpoint(self) -> Optional[str]:
-        """Tempo endpoint for charm tracing."""
-        if self.tracing.is_ready():
-            return self.tracing.otlp_http_endpoint()
-        else:
-            return None
-
-    @property
-    def server_cert_path(self) -> Optional[str]:
-        """Server certificate path for tls tracing."""
-        return CERT_PATH
-
-    @property
-    def internal_url(self) -> str:
-        """Returns workload's FQDN. Used for ingress."""
-        scheme = "https" if self._is_tls_ready else "http"
-        return f"{scheme}://{self.hostname}:8080"
-
-    @property
-    def external_url(self) -> str:
-        """Return the external hostname to be passed to ingress via the relation."""
-        try:
-            if ingress_url := self.ingress.url:
-                return ingress_url
-        except ModelError as e:
-            logger.error("Failed obtaining external url: %s.", e)
-        return self.internal_url
-
-    ###########################
-    # === UTILITY METHODS === #
-    ###########################
-
-    def _render_workers_alert_rules(self):
-        self._remove_rendered_rules()
-
-        apps = set()
-        for worker in self.cluster_provider.gather_topology():
-            if worker["app"] in apps:
-                continue
-
-            apps.add(worker["app"])
-            topology_dict = {
-                "model": self.model.name,
-                "model_uuid": self.model.uuid,
-                "application": worker["app"],
-                "unit": worker["unit"],
-                "charm_name": "tempo-worker-k8s",
-            }
-            topology = JujuTopology.from_dict(topology_dict)
-            alert_rules = AlertRules(query_type="promql", topology=topology)
-            alert_rules.add_path(WORKER_ORIGINAL_ALERT_RULES_PATH, recursive=True)
-            alert_rules_contents = yaml.dump(alert_rules.as_dict())
-
-            file_name = f"{CONSOLIDATED_ALERT_RULES_PATH}/rendered_{worker['app']}.rules"
-            with open(file_name, "w") as writer:
-                writer.write(alert_rules_contents)
-
-    def _remove_rendered_rules(self):
-        files = glob.glob(f"{CONSOLIDATED_ALERT_RULES_PATH}/rendered_*")
-        for f in files:
-            os.remove(f)
-
-    def _consolidate_nginx_rules(self):
-        os.makedirs(CONSOLIDATED_ALERT_RULES_PATH, exist_ok=True)
-        for filename in glob.glob(os.path.join(NGINX_ORIGINAL_ALERT_RULES_PATH, "*.*")):
-            shutil.copy(filename, f"{CONSOLIDATED_ALERT_RULES_PATH}/")
-
-    def _update_tempo_cluster(self):  # common exit hook
+    def _update_tempo_cluster(self):
         """Build the config and publish everything to the application databag."""
-        if not self.coordinator.is_coherent():
+        if not self.coordinator.is_coherent:
             return
-        tls = self._is_tls_ready
 
-        s3_config_data = self._get_s3_storage_config()
+        kwargs = {}
+
+        if self.tls_available:
+            # we share the certs in plaintext as they're not sensitive information
+            kwargs['ca_cert'] = self.cert_handler.ca_cert
+            kwargs['server_cert'] = self.cert_handler.server_cert
+            kwargs['privkey_secret_id'] = self.tempo_cluster.publish_privkey(VAULT_SECRET_LABEL)
 
         # On every function call, we always publish everything to the databag; however, if there
-        # are no changes, Juju will safely ignore the updates
-        self.cluster_provider.publish_data(
-            tempo_config=self.coordinator.build_config(
-                s3_config_data=s3_config_data, tls_enabled=tls
-            ),
+        # are no changes, Juju will notice there's no delta and do nothing
+        self.tempo_cluster.publish_data(
+            tempo_config=self.tempo.generate_config(self._requested_receivers(), self._s3_config),
             loki_endpoints=self.loki_endpoints_by_unit,
+            # TODO tempo receiver for charm tracing
+            **kwargs
         )
 
-        if tls:
-            self.publish_grant_secrets()
+    @property
+    def _static_ingress_config(self) -> dict:
+        entry_points = {}
+        for protocol, port in self.tempo.all_ports.items():
+            sanitized_protocol = protocol.replace("_", "-")
+            entry_points[sanitized_protocol] = {"address": f":{port}"}
 
-    def has_multiple_workers(self) -> bool:
-        """Return True if there are multiple workers forming the Tempo cluster."""
-        tempo_cluster_relations = self.model.relations.get("tempo-cluster", [])
-        remote_units_count = sum(
-            len(relation.units)
-            for relation in tempo_cluster_relations
-            if relation.app != self.model.app
-        )
-        return remote_units_count > 1
+        return {"entryPoints": entry_points}
 
-    def publish_grant_secrets(self) -> None:
-        """Publish and Grant secrets to the tempo-cluster relation."""
-        secrets = {
-            "private_key_secret_id": self.server_cert.private_key_secret_id,
-            "ca_server_cert_secret_id": self.server_cert.ca_server_cert_secret_id,
+    @property
+    def _ingress_config(self) -> dict:
+        """Build a raw ingress configuration for Traefik."""
+        http_routers = {}
+        http_services = {}
+        for protocol, port in self.tempo.all_ports.items():
+            sanitized_protocol = protocol.replace("_", "-")
+            http_routers[f"juju-{self.model.name}-{self.model.app.name}-{sanitized_protocol}"] = {
+                "entryPoints": [sanitized_protocol],
+                "service": f"juju-{self.model.name}-{self.model.app.name}-service-{sanitized_protocol}",
+                # TODO better matcher
+                "rule": "ClientIP(`0.0.0.0/0`)",
+            }
+            if sanitized_protocol.endswith("grpc") and not self.tls_available:
+                # to send traces to unsecured GRPC endpoints, we need h2c
+                # see https://doc.traefik.io/traefik/v2.0/user-guides/grpc/#with-http-h2c
+                http_services[
+                    f"juju-{self.model.name}-{self.model.app.name}-service-{sanitized_protocol}"
+                ] = {"loadBalancer": {"servers": [{"url": f"h2c://{self.hostname}:{port}"}]}}
+            else:
+                # anything else, including secured GRPC, can use _internal_url
+                # ref https://doc.traefik.io/traefik/v2.0/user-guides/grpc/#with-https
+                http_services[
+                    f"juju-{self.model.name}-{self.model.app.name}-service-{sanitized_protocol}"
+                ] = {"loadBalancer": {"servers": [{"url": f"{self._internal_url}:{port}"}]}}
+        return {
+            "http": {
+                "routers": http_routers,
+                "services": http_services,
+            },
         }
 
-        relations = self.model.relations["tempo-cluster"]
-        for relation in relations:
-            relation.data[self.model.app]["secrets"] = json.dumps(secrets)
-            logger.debug("Secrets published")
-
-            for secret_id in secrets.values():
-                secret = self.model.get_secret(id=secret_id)
-                secret.grant(relation)
-
-    def _get_s3_storage_config(self):
-        """Retrieve S3 storage configuration."""
-        if not self.s3_requirer.relations:
-            return None
-        raw = self.s3_requirer.get_s3_connection_info()
-        try:
-            return _S3ConfigData(**raw)
-        except ValidationError:
-            msg = f"failed to validate s3 config data: {raw}"
-            logger.error(msg, exc_info=True)
-            return None
-
-    def _update_cert(self):
-        if not self._nginx_container.can_connect():
-            return
-
-        ca_cert_path = Path("/usr/local/share/ca-certificates/ca.crt")
-
-        if self._is_cert_available:
-            # Save the workload certificates
-            self._nginx_container.push(
-                CERT_PATH,
-                self.server_cert.server_cert,  # pyright: ignore
-                make_dirs=True,
-            )
-            self._nginx_container.push(
-                KEY_PATH,
-                self.server_cert.private_key,  # pyright: ignore
-                make_dirs=True,
-            )
-            # Save the CA among the trusted CAs and trust it
-            self._nginx_container.push(
-                ca_cert_path,
-                self.server_cert.ca_cert,  # pyright: ignore
-                make_dirs=True,
-            )
-            self._nginx_container.push(
-                CA_CERT_PATH,
-                self.server_cert.ca_cert,  # pyright: ignore
-                make_dirs=True,
-            )
-        else:
-            self._nginx_container.remove_path(CERT_PATH, recursive=True)
-            self._nginx_container.remove_path(KEY_PATH, recursive=True)
-            self._nginx_container.remove_path(ca_cert_path, recursive=True)
-            self._nginx_container.remove_path(
-                CA_CERT_PATH, recursive=True
-            )  # TODO: remove (see FIXME ^)
-            # Repeat for the charm container.
-            ca_cert_path.unlink(missing_ok=True)
-
-        # TODO: We need to install update-ca-certificates in Nginx Rock
-        # self._nginx_container.exec(["update-ca-certificates", "--fresh"]).wait()
-        subprocess.run(["update-ca-certificates", "--fresh"])
-
 
 if __name__ == "__main__":  # pragma: nocover
-    ops.main.main(TempoCoordinatorK8SOperatorCharm)
+    main(TempoCharm)
diff --git a/src/coordinator.py b/src/coordinator.py
new file mode 100644
index 0000000..cc87926
--- /dev/null
+++ b/src/coordinator.py
@@ -0,0 +1,112 @@
+import collections
+import logging
+from collections import Counter
+from typing import Dict, List, Set
+
+from tempo_cluster import TempoClusterProvider, TempoRole
+
+logger = logging.getLogger(__name__)
+
+MINIMAL_DEPLOYMENT = {
+    TempoRole.querier: 1,
+    TempoRole.query_frontend: 1,
+    TempoRole.ingester: 1,
+    TempoRole.distributor: 1,
+    TempoRole.compactor: 1,
+    TempoRole.metrics_generator: 1,
+}
+"""The minimal set of roles that need to be allocated for the
+deployment to be considered consistent (otherwise we set blocked)."""
+
+# TODO: find out what the actual recommended deployment is
+RECOMMENDED_DEPLOYMENT = Counter(
+    {
+        TempoRole.querier: 1,
+        TempoRole.query_frontend: 1,
+        TempoRole.ingester: 3,
+        TempoRole.distributor: 1,
+        TempoRole.compactor: 1,
+        TempoRole.metrics_generator: 1,
+    }
+)
+"""The set of roles that need to be allocated for the
+deployment to be considered robust according to the official 
+recommendations/guidelines."""
+
+
+class TempoCoordinator:
+    """Tempo coordinator."""
+
+    def __init__(self, cluster_provider: TempoClusterProvider, is_worker: bool = False):
+        self._is_worker = is_worker
+
+        if is_worker:
+            # roles fulfilled by the local node, if the coordinator is also running a worker.
+            local_roles = collections.Counter(r for r in TempoRole if r is not TempoRole.all)
+        else:
+            local_roles = None
+
+        self._cluster_provider = cluster_provider
+        self._roles: Dict[TempoRole, int] = self._cluster_provider.gather_roles(base=local_roles)
+
+        # Whether the roles list makes up a coherent mimir deployment.
+        self.is_coherent = set(self._roles.keys()).issuperset(MINIMAL_DEPLOYMENT)
+        self.missing_roles: Set[TempoRole] = set(MINIMAL_DEPLOYMENT).difference(self._roles.keys())
+        # If the coordinator is incoherent, return the roles that are missing for it to become so.
+
+        def _is_recommended():
+            for role, min_n in RECOMMENDED_DEPLOYMENT.items():
+                if self._roles.get(role, 0) < min_n:
+                    return False
+            return True
+
+        self.is_recommended: bool = _is_recommended()
+        # Whether the present roles are a superset of the minimal deployment.
+        # I.E. If all required roles are assigned, and each role has the recommended amount of units.
+        # python>=3.11 would support roles >= RECOMMENDED_DEPLOYMENT
+
+    def get_deployment_inconsistencies(
+        self, clustered: bool, scaled: bool, has_workers: bool, is_worker_node: bool, has_s3: bool
+    ) -> List[str]:
+        """Determine whether the deployment as a whole is consistent.
+
+        Return a list of failed consistency checks.
+        """
+        return self._get_deployment_inconsistencies(
+            clustered=clustered,
+            scaled=scaled,
+            has_workers=has_workers,
+            is_worker_node=is_worker_node,
+            has_s3=has_s3,
+            coherent=self.is_coherent,
+            missing_roles=self.missing_roles,
+        )
+
+    @staticmethod
+    def _get_deployment_inconsistencies(
+        clustered: bool,
+        scaled: bool,
+        has_workers: bool,
+        is_worker_node: bool,
+        has_s3: bool,
+        coherent: bool,
+        missing_roles: Set[TempoRole] = None,
+    ) -> List[str]:
+        """Determine whether the deployment as a whole is consistent.
+
+        Return a list of failed consistency checks.
+        """
+
+        failures = []
+        # is_monolith = not (scaled or clustered or has_workers) and is_worker_node
+
+        if not is_worker_node and not has_workers:
+            failures.append("Tempo must either be a worker node or have some workers.")
+        if not has_s3:
+            if scaled:
+                failures.append("Tempo is scaled but has no s3 integration.")
+            if clustered:
+                failures.append("Tempo is clustered but has no s3 integration.")
+        elif not coherent:
+            failures.append(f"Incoherent coordinator: missing roles: {missing_roles}.")
+        return failures
diff --git a/src/grafana_dashboards/tempo.json b/src/grafana_dashboards/tempo.json
new file mode 100644
index 0000000..d7cf799
--- /dev/null
+++ b/src/grafana_dashboards/tempo.json
@@ -0,0 +1,3706 @@
+{
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "datasource",
+          "uid": "grafana"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "target": {
+          "limit": 100,
+          "matchAny": false,
+          "tags": [],
+          "type": "dashboard"
+        },
+        "type": "dashboard"
+      },
+      {
+        "datasource": {
+          "uid": "${prometheusds}"
+        },
+        "enable": true,
+        "expr": "{diff_namsespace=\"$juju_model\", container=\"kube-diff-logger\"}",
+        "hide": true,
+        "iconColor": "rgba(255, 96, 96, 1)",
+        "name": "diffs",
+        "showIn": 0,
+        "target": {}
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "collapsed": false,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 312,
+      "panels": [
+
+      ],
+      "title": "At a glance",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "normal"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 3,
+        "x": 0,
+        "y": 0
+      },
+      "id": 320,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": [
+            "last"
+          ],
+          "fields": "/^version$/"
+        },
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "tempo_build_info{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+          "format": "table",
+          "interval": "",
+          "legendFormat": "{{pod}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Tempo version",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "normal"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 7,
+        "x": 3,
+        "y": 0
+      },
+      "id": 321,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "tempo_ingester_traces_created_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+          "interval": "",
+          "legendFormat": "{{pod}}",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Traces count",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "normal"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 7,
+        "x": 10,
+        "y": 0
+      },
+      "id": 322,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "builder",
+          "expr": "tempo_receiver_accepted_spans{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+          "interval": "",
+          "legendFormat": "",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Spans count",
+      "type": "stat"
+    },
+    {
+      "datasource": {
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 7,
+        "x": 17,
+        "y": 0
+      },
+      "id": 323,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "expr": "tempo_ingester_flush_size_bytes_sum{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+          "interval": "",
+          "legendFormat": "{{pod}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Ingested bytes",
+      "type": "stat"
+    },
+    {
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 6
+      },
+      "id": 134,
+      "title": "General",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 16,
+        "x": 0,
+        "y": 7
+      },
+      "id": 132,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "sum by(juju_unit) (rate(process_cpu_seconds_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]))",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "CPU",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "points",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 6,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "always",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 7
+      },
+      "id": 130,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "last"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.99, sum by(le) (rate(tempo_ingester_flush_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".99",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.9, sum by(le) (rate(tempo_ingester_flush_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".9",
+          "range": true,
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.5, sum by(le) (rate(tempo_ingester_flush_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".5",
+          "range": true,
+          "refId": "C"
+        }
+      ],
+      "title": "Ingesters Flush Duration",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 0,
+        "y": 15
+      },
+      "id": 136,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "builder",
+          "expr": "rate(tempo_receiver_accepted_spans{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])",
+          "legendFormat": "accepted spans",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "rate(tempo_receiver_refused_spans{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])",
+          "hide": false,
+          "legendFormat": "refused spans",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Input Spans (rate)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 8,
+        "y": 15
+      },
+      "id": 117,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "sum by (juju_unit) (rate(tempo_ingester_flush_duration_seconds_sum{juju_application=~\"$juju_application\", juju_model=~\"$juju_model\", juju_model_uuid=~\"$juju_model_uuid\", juju_unit=~\"$juju_unit\"}[5m]))\n",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Flush Duration (rate)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "points",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 6,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "always",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 8,
+        "x": 16,
+        "y": 15
+      },
+      "id": 139,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "last"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.99, sum by(le) (rate(tempo_distributor_push_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".99",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.9, sum by(le) (rate(tempo_distributor_push_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "legendFormat": ".9",
+          "range": true,
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.5, sum by(le) (rate(tempo_distributor_push_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".5",
+          "range": true,
+          "refId": "C"
+        }
+      ],
+      "title": "Distributor Push Duration",
+      "type": "timeseries"
+    },
+    {
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 23
+      },
+      "id": 129,
+      "title": "Network",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 18,
+        "x": 0,
+        "y": 24
+      },
+      "id": 78,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.99, sum by(le) (rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".99",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.9, sum by(le) (rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".9",
+          "range": true,
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.5, sum by(le) (rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".5",
+          "range": true,
+          "refId": "C"
+        }
+      ],
+      "title": "HTTP Requests Latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 18,
+        "y": 24
+      },
+      "id": 119,
+      "options": {
+        "legend": {
+          "calcs": [
+            "last"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "sum by(protocol) (tempo_tcp_connections{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Open TCP Connections (by protocol)",
+      "type": "timeseries"
+    },
+    {
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 32
+      },
+      "id": 123,
+      "title": "Backend",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 18,
+        "x": 0,
+        "y": 33
+      },
+      "id": 121,
+      "options": {
+        "legend": {
+          "calcs": [
+            "last"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "sum by(status) (tempodb_backend_objects_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Number of Traces",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "points",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 6,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "always",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 18,
+        "y": 33
+      },
+      "id": 51,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "last"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.99, sum by(le) (rate(tempodb_retention_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".99",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.9, sum by(le) (rate(tempodb_retention_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".9",
+          "range": true,
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.5, sum by(le) (rate(tempodb_retention_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".5",
+          "range": true,
+          "refId": "C"
+        }
+      ],
+      "title": "Retention Duration",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "decbytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 0,
+        "y": 41
+      },
+      "id": 125,
+      "options": {
+        "legend": {
+          "calcs": [
+            "last"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "sum by(status) (tempodb_backend_bytes_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Bytes Used",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "decbytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 6,
+        "y": 41
+      },
+      "id": 127,
+      "options": {
+        "legend": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "editorMode": "code",
+          "expr": "sum by(level) (tempodb_compaction_bytes_written_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"})",
+          "legendFormat": "__auto",
+          "range": true,
+          "refId": "A"
+        }
+      ],
+      "title": "Bytes Compacted (by level)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 12,
+        "y": 41
+      },
+      "id": 53,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "sum(increase(tempodb_retention_deleted_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]))",
+          "interval": "",
+          "legendFormat": "deleted",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "sum(increase(tempodb_retention_marked_for_deletion_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]))",
+          "interval": "",
+          "legendFormat": "marked_for_deletion",
+          "range": true,
+          "refId": "B"
+        }
+      ],
+      "title": "Retention",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "points",
+            "fillOpacity": 10,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 6,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "always",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 6,
+        "x": 18,
+        "y": 41
+      },
+      "id": 35,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "last"
+          ],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.99, sum by(le) (rate(tempodb_blocklist_poll_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".99",
+          "range": true,
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.9, sum by(le) (rate(tempodb_blocklist_poll_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "legendFormat": ".9",
+          "range": true,
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P0899610D3ECDF9EC"
+          },
+          "editorMode": "code",
+          "expr": "histogram_quantile(0.5, sum by(le) (rate(tempodb_blocklist_poll_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])))",
+          "interval": "",
+          "legendFormat": ".5",
+          "range": true,
+          "refId": "C"
+        }
+      ],
+      "title": "Blocklist Poll Duration",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 49
+      },
+      "id": 38,
+      "panels": [],
+      "title": "CPU / Memory / Goroutines",
+      "type": "row"
+    },
+    {
+      "datasource": {
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 0,
+        "y": 50
+      },
+      "id": 24,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "expr": "rate(go_gc_duration_seconds_count{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])",
+          "interval": "",
+          "legendFormat": "{{pod}}",
+          "refId": "A"
+        }
+      ],
+      "title": "GCs (rate)",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "decbytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 6,
+        "y": 50
+      },
+      "id": 25,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "expr": "go_memstats_heap_inuse_bytes{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+          "interval": "",
+          "legendFormat": "{{pod}}",
+          "refId": "A"
+        }
+      ],
+      "title": "go heap",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 12,
+        "y": 50
+      },
+      "id": 23,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "expr": "go_goroutines{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+          "legendFormat": "{{pod}}",
+          "refId": "A"
+        }
+      ],
+      "title": "goroutines",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "uid": "${prometheusds}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "never",
+            "spanNulls": true,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "decbytes"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 5,
+        "w": 6,
+        "x": 18,
+        "y": 50
+      },
+      "id": 43,
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "pluginVersion": "9.0.0-d373beebpre",
+      "targets": [
+        {
+          "expr": "process_resident_memory_bytes{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+          "interval": "",
+          "legendFormat": "{{pod}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Resident memory",
+      "type": "timeseries"
+    },
+    {
+      "collapsed": true,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 55
+      },
+      "id": 49,
+      "panels": [
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "axisSoftMax": 1,
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 0,
+            "y": 2
+          },
+          "id": 33,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "desc"
+            }
+          },
+          "pluginVersion": "9.0.0-d373beebpre",
+          "targets": [
+            {
+              "expr": "tempodb_work_queue_length{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"} / tempodb_work_queue_max{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}",
+              "legendFormat": "{{instance}}",
+              "refId": "A"
+            }
+          ],
+          "title": "% of work queue filled",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 6,
+            "y": 2
+          },
+          "id": 32,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "desc"
+            }
+          },
+          "pluginVersion": "9.0.0-d373beebpre",
+          "targets": [
+            {
+              "expr": "sum(increase(tempodb_compaction_errors_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (job)",
+              "legendFormat": "compaction_err",
+              "refId": "B"
+            },
+            {
+              "expr": "sum(increase(tempodb_retention_errors_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (job)",
+              "legendFormat": "retention_err",
+              "refId": "C"
+            },
+            {
+              "expr": "sum(increase(tempodb_blocklist_poll_errors_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (job)",
+              "legendFormat": "blocklist_err",
+              "refId": "D"
+            }
+          ],
+          "title": "Maintenance Errors",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "normal"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 12,
+            "y": 2
+          },
+          "id": 47,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d373beebpre",
+          "targets": [
+            {
+              "exemplar": true,
+              "expr": "avg(tempodb_blocklist_length{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}) by (tenant)",
+              "instant": false,
+              "interval": "",
+              "legendFormat": "{{tenant}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Blocklist Length",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 18,
+            "y": 2
+          },
+          "id": 70,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d373beebpre",
+          "targets": [
+            {
+              "expr": "increase(kube_pod_container_status_restarts_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]) > 0",
+              "hide": false,
+              "interval": "",
+              "legendFormat": "{{pod}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Container Restarts",
+          "type": "timeseries"
+        }
+      ],
+      "title": "Maintenance",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 56
+      },
+      "id": 19,
+      "panels": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 0,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "normal"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 0,
+            "y": 4
+          },
+          "id": 10,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d452322apre",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${prometheusds}"
+              },
+              "editorMode": "code",
+              "expr": "sum by(cluster) (rate(tempo_distributor_spans_received_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]))",
+              "interval": "",
+              "legendFormat": "{{pod}}",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Spans Received",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "normal"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 6,
+            "y": 4
+          },
+          "id": 9,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d452322apre",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${prometheusds}"
+              },
+              "editorMode": "builder",
+              "expr": "sum by(cluster) (rate(tempo_ingester_traces_created_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]))",
+              "interval": "",
+              "legendFormat": "",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Traces Created",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 12,
+            "y": 4
+          },
+          "id": 8,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d452322apre",
+          "targets": [
+            {
+              "expr": "sum(increase(tempo_ingester_blocks_flushed_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[1h]))",
+              "interval": "",
+              "legendFormat": "{{pod}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Blocks Flushed (1h)",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 6,
+            "x": 18,
+            "y": 4
+          },
+          "id": 26,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d452322apre",
+          "targets": [
+            {
+              "expr": "sum(increase(tempo_ingester_blocks_cleared_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[1h]))",
+              "interval": "",
+              "legendFormat": "",
+              "refId": "A"
+            }
+          ],
+          "title": "Blocks Cleared (1h)",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "none"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 10,
+            "w": 12,
+            "x": 0,
+            "y": 9
+          },
+          "id": 72,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d452322apre",
+          "targets": [
+            {
+              "expr": "sum(rate(tempo_receiver_accepted_spans{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]))",
+              "interval": "",
+              "legendFormat": "accepted",
+              "refId": "A"
+            },
+            {
+              "expr": "sum(rate(tempo_receiver_refused_spans{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m]))",
+              "interval": "",
+              "legendFormat": "refused",
+              "refId": "B"
+            }
+          ],
+          "title": "OTEL Distributor Spans/second",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 12,
+            "x": 12,
+            "y": 9
+          },
+          "id": 79,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d452322apre",
+          "targets": [
+            {
+              "expr": "histogram_quantile(.99, sum(rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (le))",
+              "interval": "",
+              "legendFormat": ".99",
+              "refId": "A"
+            },
+            {
+              "expr": "histogram_quantile(.9, sum(rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (le))",
+              "interval": "",
+              "legendFormat": ".9",
+              "refId": "B"
+            },
+            {
+              "expr": "histogram_quantile(.5, sum(rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (le))",
+              "interval": "",
+              "legendFormat": ".5",
+              "refId": "C"
+            }
+          ],
+          "title": "Push Latency (Gateway)",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisCenteredZero": false,
+                "axisColorMode": "text",
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "insertNulls": false,
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": false,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green",
+                    "value": null
+                  },
+                  {
+                    "color": "red",
+                    "value": 80
+                  }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 5,
+            "w": 12,
+            "x": 12,
+            "y": 14
+          },
+          "id": 2,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d452322apre",
+          "targets": [
+            {
+              "expr": "histogram_quantile(.99, sum(rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\", route=~\"/tempopb.Pusher/Push.*\"}[5m])) by (le))",
+              "interval": "",
+              "legendFormat": ".99",
+              "refId": "A"
+            },
+            {
+              "expr": "histogram_quantile(.9, sum(rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\", route=~\"/tempopb.Pusher/Push.*\"}[5m])) by (le))",
+              "interval": "",
+              "legendFormat": ".9",
+              "refId": "B"
+            },
+            {
+              "expr": "histogram_quantile(.5, sum(rate(tempo_request_duration_seconds_bucket{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\", route=~\"/tempopb.Pusher/Push.*\"}[5m])) by (le))",
+              "interval": "",
+              "legendFormat": ".5",
+              "refId": "C"
+            }
+          ],
+          "title": "Push Latency (Ingester)",
+          "type": "timeseries"
+        }
+      ],
+      "title": "Write",
+      "type": "row"
+    },
+    {
+      "collapsed": true,
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${prometheusds}"
+      },
+      "gridPos": {
+        "h": 1,
+        "w": 24,
+        "x": 0,
+        "y": 61
+      },
+      "id": 81,
+      "panels": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "none"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green"
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 6,
+            "w": 6,
+            "x": 0,
+            "y": 61
+          },
+          "id": 88,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": true
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d373beebpre",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${prometheusds}"
+              },
+              "editorMode": "builder",
+              "expr": "sum(rate(tempodb_compaction_bytes_written_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (level)",
+              "interval": "",
+              "legendFormat": "__auto",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Bytes Written / s",
+          "type": "timeseries"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${prometheusds}"
+          },
+          "fieldConfig": {
+            "defaults": {
+              "color": {
+                "mode": "palette-classic"
+              },
+              "custom": {
+                "axisLabel": "",
+                "axisPlacement": "auto",
+                "barAlignment": 0,
+                "drawStyle": "line",
+                "fillOpacity": 10,
+                "gradientMode": "none",
+                "hideFrom": {
+                  "legend": false,
+                  "tooltip": false,
+                  "viz": false
+                },
+                "lineInterpolation": "linear",
+                "lineWidth": 1,
+                "pointSize": 5,
+                "scaleDistribution": {
+                  "type": "linear"
+                },
+                "showPoints": "never",
+                "spanNulls": true,
+                "stacking": {
+                  "group": "A",
+                  "mode": "normal"
+                },
+                "thresholdsStyle": {
+                  "mode": "off"
+                }
+              },
+              "mappings": [],
+              "thresholds": {
+                "mode": "absolute",
+                "steps": [
+                  {
+                    "color": "green"
+                  }
+                ]
+              },
+              "unit": "short"
+            },
+            "overrides": []
+          },
+          "gridPos": {
+            "h": 6,
+            "w": 6,
+            "x": 6,
+            "y": 61
+          },
+          "id": 86,
+          "options": {
+            "legend": {
+              "calcs": [],
+              "displayMode": "list",
+              "placement": "bottom",
+              "showLegend": false
+            },
+            "tooltip": {
+              "mode": "multi",
+              "sort": "none"
+            }
+          },
+          "pluginVersion": "9.0.0-d373beebpre",
+          "targets": [
+            {
+              "datasource": {
+                "type": "prometheus",
+                "uid": "${prometheusds}"
+              },
+              "editorMode": "code",
+              "expr": "sum(increase(tempodb_compaction_blocks_total{juju_application=~\"$juju_application\",juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_unit=~\"$juju_unit\"}[5m])) by (level)",
+              "interval": "",
+              "legendFormat": "__auto",
+              "range": true,
+              "refId": "A"
+            }
+          ],
+          "title": "Blocks Compacted",
+          "type": "timeseries"
+        }
+      ],
+      "title": "Compactor",
+      "type": "row"
+    }
+  ],
+  "refresh": false,
+  "schemaVersion": 37,
+  "style": "dark",
+  "tags": [
+    "tempo"
+  ],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": true,
+          "text": [
+            "All"
+          ],
+          "value": [
+            "$__all"
+          ]
+        },
+        "hide": 0,
+        "includeAll": true,
+        "label": "Loki datasource",
+        "multi": true,
+        "name": "lokids",
+        "options": [],
+        "query": "loki",
+        "queryValue": "",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      },
+      {
+        "current": {
+          "selected": true,
+          "text": [
+            "All"
+          ],
+          "value": [
+            "$__all"
+          ]
+        },
+        "hide": 0,
+        "includeAll": true,
+        "label": "Prometheus datasource",
+        "multi": true,
+        "name": "prometheusds",
+        "options": [],
+        "query": "prometheus",
+        "queryValue": "",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      },
+      {
+        "allValue": ".*",
+        "current": {
+          "selected": false,
+          "text": "All",
+          "value": "$__all"
+        },
+        "datasource": {
+          "uid": "${prometheusds}"
+        },
+        "definition": "label_values(up{juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_application=~\"$juju_application\"},juju_unit)",
+        "hide": 0,
+        "includeAll": true,
+        "label": "Juju unit",
+        "multi": true,
+        "name": "juju_unit",
+        "options": [],
+        "query": {
+          "query": "label_values(up{juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\",juju_application=~\"$juju_application\"},juju_unit)",
+          "refId": "StandardVariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 0,
+        "tagValuesQuery": "",
+        "tags": [],
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      },
+      {
+        "allValue": ".*",
+        "current": {
+          "selected": false,
+          "text": "All",
+          "value": "$__all"
+        },
+        "datasource": {
+          "uid": "${prometheusds}"
+        },
+        "definition": "label_values(up{juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\"},juju_application)",
+        "hide": 0,
+        "includeAll": true,
+        "label": "Juju application",
+        "multi": true,
+        "name": "juju_application",
+        "options": [],
+        "query": {
+          "query": "label_values(up{juju_model=~\"$juju_model\",juju_model_uuid=~\"$juju_model_uuid\"},juju_application)",
+          "refId": "StandardVariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 0,
+        "tagValuesQuery": "",
+        "tags": [],
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      },
+      {
+        "allValue": ".*",
+        "current": {
+          "selected": true,
+          "text": [
+            "All"
+          ],
+          "value": [
+            "$__all"
+          ]
+        },
+        "datasource": {
+          "uid": "${prometheusds}"
+        },
+        "definition": "label_values(up{juju_model=~\"$juju_model\"},juju_model_uuid)",
+        "hide": 0,
+        "includeAll": true,
+        "label": "Juju model uuid",
+        "multi": true,
+        "name": "juju_model_uuid",
+        "options": [],
+        "query": {
+          "query": "label_values(up{juju_model=~\"$juju_model\"},juju_model_uuid)",
+          "refId": "StandardVariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 0,
+        "tagValuesQuery": "",
+        "tags": [],
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      },
+      {
+        "allValue": ".*",
+        "current": {
+          "selected": true,
+          "text": [
+            "All"
+          ],
+          "value": [
+            "$__all"
+          ]
+        },
+        "datasource": {
+          "uid": "${prometheusds}"
+        },
+        "definition": "label_values(up,juju_model)",
+        "hide": 0,
+        "includeAll": true,
+        "label": "Juju model",
+        "multi": true,
+        "name": "juju_model",
+        "options": [],
+        "query": {
+          "query": "label_values(up,juju_model)",
+          "refId": "StandardVariableQuery"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 0,
+        "tagValuesQuery": "",
+        "tags": [],
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timepicker": {
+    "refresh_intervals": [
+      "10s",
+      "30s",
+      "1m",
+      "5m",
+      "15m",
+      "30m",
+      "1h",
+      "2h",
+      "1d"
+    ]
+  },
+  "timezone": "",
+  "title": "Tempo Overview",
+  "uid": "a6175b9cc7ec20591890117c39580030",
+  "version": 1,
+  "weekStart": ""
+}
\ No newline at end of file
diff --git a/src/loki_alert_rules/.gitkeep b/src/loki_alert_rules/.gitkeep
new file mode 100644
index 0000000..4837887
--- /dev/null
+++ b/src/loki_alert_rules/.gitkeep
@@ -0,0 +1,2 @@
+alerts and rules from:
+https://github.com/grafana/tempo/tree/main/operations/tempo-mixin-compiled
\ No newline at end of file
diff --git a/src/loki_alert_rules/error_level_logs_found.rule b/src/loki_alert_rules/error_level_logs_found.rule
new file mode 100644
index 0000000..9c7d873
--- /dev/null
+++ b/src/loki_alert_rules/error_level_logs_found.rule
@@ -0,0 +1,7 @@
+alert: HighPercentageError
+expr: 100 * sum by (job) (rate({%%juju_topology%%} |= "level=error"[5m])) / sum by (job) (rate({%%juju_topology%%}[5m])) > 5
+for: 10m
+labels:
+  severity: critical
+annotations:
+  summary: "The {{ $labels.job }} is experiencing {{ printf \"%.2f\" $value }}% error-level logs rate."
\ No newline at end of file
diff --git a/src/prometheus_alert_rules/.gitkeep b/src/prometheus_alert_rules/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/src/prometheus_alert_rules/alerts.yaml b/src/prometheus_alert_rules/alerts.yaml
new file mode 100644
index 0000000..2c21d50
--- /dev/null
+++ b/src/prometheus_alert_rules/alerts.yaml
@@ -0,0 +1,187 @@
+groups:
+- name: tempo_alerts
+  rules:
+  - alert: "TempoBlockListRisingQuickly"
+    expr: |
+      avg by (job, instance)(tempodb_blocklist_length) / avg by (job, instance)(tempodb_blocklist_length offset 7d) > 1.4
+    for: "15m"
+    labels:
+      severity: "critical"
+    annotations:
+      summary: "Tempo block list rising quickly (instance {{ $labels.instance }})"
+      description: "The {{ $labels.job }} is experiencing a 40% rise in tempo blocklist length over the last 7 days.  Consider scaling compactors."
+  - alert: TempoCompactionsFailing
+    expr: sum by (job, instance)(increase(tempodb_compaction_errors_total{}[1h])) > 2 and sum by (job, instance)(increase(tempodb_compaction_errors_total{}[5m])) > 0
+    for: 5m
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo compactions failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing more than 2 compactions failures in the past hour."
+  - alert: TempoCompactorUnhealthy
+    expr: max by (job, instance)(tempo_ring_members{state="Unhealthy", name="compactor"}) > 0
+    for: 15m
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo unhealthy compactor(s) (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is having {{ printf \"%f\" $value }} unhealthy compactor(s)."
+  - alert: TempoCompactorsTooManyOutstandingBlocks
+    expr: sum by (tenant) (tempodb_compaction_outstanding_blocks) / ignoring(tenant) group_left count(tempo_build_info) > 100
+    for: "6h"
+    labels:
+      severity: warning
+    annotations:
+      summary: Tempo too many outstanding compaction blocks (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is having too many outstanding compaction blocks for tenant {{ $labels.tenant }}, increase compactor's CPU or add more compactors."
+  - alert: TempoDiscardedSpans
+    expr:  100 * sum by (instance,job)(rate(tempo_discarded_spans_total[5m])) / sum by (instance,job)(rate(tempo_distributor_spans_received_total[5m]))  > 5
+    for: "5m"
+    labels:
+      severity: warning
+    annotations:
+      summary: Tempo spans insertion failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing {{ printf \"%.2f\" $value }}% discard of spans."
+  - alert: TempoDistributorPushLatency
+    expr:  histogram_quantile(0.99, sum by(le, job, instance) (rate(tempo_distributor_push_duration_seconds_bucket[5m]))) > 3
+    for: "5m"
+    labels:
+      severity: warning
+    annotations:
+      summary: Tempo distributor push latency (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} distributor push is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency."
+  - alert: TempoDistributorUnhealthy
+    expr: max by (job, instance)(tempo_ring_members{state="Unhealthy", name="distributor"}) > 0
+    for: 15m
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo unhealthy distributor(s) (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is having {{ printf \"%f\" $value }} unhealthy distributor(s)."
+  - alert: TempoFailedIngestingRequests
+    expr:  sum by (job,instance)(increase (tempo_ingester_traces_created_total[5m])) / sum by (instance,job)(rate(tempo_request_duration_seconds_count{route='/tempopb.Pusher/PushBytesV2'}[5m])) == 0
+    for: "5m"
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo pushing traces to ingester failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing failure in distributors pushing traces to the ingesters."
+  - alert: TempoFrontendClients
+    expr:  tempo_query_frontend_connected_clients == 0
+    for: "5m"
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo frontend connected clients (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} has no frontend connected clients."
+  - alert: TempoFrontendQueueLatency
+    expr:  histogram_quantile(0.99, sum by(le,instance,job) (rate(tempo_query_frontend_queue_duration_seconds_bucket[15m]))) > 2
+    for: "15m"
+    labels:
+      severity: warning
+    annotations:
+      summary: Tempo frontend queue latency (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} frontend queue is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency."
+  - alert: TempoIngesterFlushLatency
+    expr:  histogram_quantile(0.99, sum by(le,instance,job) (rate(tempo_ingester_flush_duration_seconds_bucket[5m]))) > 5
+    for: "5m"
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo ingester flush latency (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} ingester flush is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency."
+  - alert: TempoIngesterFlushesFailing
+    expr: sum by (instance,job)(increase(tempo_ingester_flush_failed_retries_total[1h])) > 2 and sum by(instance,job)(increase(tempo_ingester_flush_failed_retries_total[5m])) > 0
+    for: 5m
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo ingester flush retries failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing more than 2 flush retries failures in the past hour."
+  - alert: TempoIngesterFlushesUnhealthy
+    expr: sum by (instance,job)(increase(tempo_ingester_failed_flushes_total[1h])) > 2 and sum by (instance,job)(increase(tempo_ingester_failed_flushes_total[5m])) > 0
+    for: 5m
+    labels:
+      severity: warning
+    annotations:
+      summary: Tempo ingester flush failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing more than 2 ingester flush failures in the past hour."
+  - alert: TempoIngestersUnhealthy
+    expr:  max by (instance,job)(tempo_ring_members{state="Unhealthy", name="ingester"}) > 0
+    for: "15m"
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo unhealthy ingester(s) (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is having {{ printf \"%f\" $value }} unhealthy ingester(s)."
+  - alert: TempoKVRequestErrors
+    expr:  100 * sum(rate(tempo_kv_request_duration_seconds_count{status_code=~"5.."}[5m])) by (route,instance,job) / sum(rate(tempo_kv_request_duration_seconds_count[5m])) by (route,instance,job)  > 10
+    for: "15m"
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo kv store request errors (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} {{ $labels.route }} KV store requests is experiencing {{ printf \"%.2f\" $value }}% error rate."
+  - alert: TempoTargetMissing
+    expr: up == 0
+    for: 0m
+    labels:
+      severity: critical
+    annotations:
+      summary: Prometheus target missing (instance {{ $labels.instance }})
+      description: "A Prometheus target has disappeared. An exporter might be crashed."
+  - alert: TempoNoTenantIndexBuilders
+    expr: sum by (tenant,job,instance) (tempodb_blocklist_tenant_index_builder) == 0 and  max by (tenant,job,instance)(tempodb_blocklist_length) > 0
+    for: 5m
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo tenant index builder failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is having no tenant index builders for tenant {{ $labels.tenant }}. Tenant index will quickly become stale."
+  - alert: TempoRequestErrors
+    expr: 100 * sum(rate(tempo_request_duration_seconds_count{status_code=~"5.."}[5m])) by (route,job,instance) / sum(rate(tempo_request_duration_seconds_count[5m])) by (route,job,instance)  > 10
+    for: "15m"
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo request errors (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing {{ printf \"%.2f\" $value }}% request error rate."
+  - alert: TempoRequestLatency
+    expr: histogram_quantile(0.99, sum by(le, route,job,instance)(rate(tempo_request_duration_seconds_bucket[5m]))) > 5
+    for: 5m
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo request latency (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} {{ $labels.route }} is experiencing {{ printf \"%.2f\" $value }}s 99th percentile latency."
+  - alert: TempoRetentionsFailing
+    expr:  sum by (job,instance)(increase(tempodb_retention_errors_total[1h])) > 2 and sum by (job,instance)(increase(tempodb_retention_errors_total[5m])) > 0
+    for: "5m"
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo retentions failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing more than 2 retention failures in the past hour."
+  - alert: TempoTCPConnectionsLimit
+    expr:  100 * tempo_tcp_connections / tempo_tcp_connections_limit >= 80 and tempo_tcp_connections_limit > 0
+    for: "5m"
+    labels:
+      severity: warning
+    annotations:
+      summary: Tempo reaching max number of tcp connections (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is reaching {{ printf \"%.2f\" $value }}% of max tcp {{ $labels.protocol }} connections."
+  - alert: TempoTenantIndexTooOld
+    expr: max by(tenant,instance,job) (tempodb_blocklist_tenant_index_age_seconds) > 600
+    for: 5m
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo tenant old index (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing a tenant {{ $labels.tenant }} with a too old index age of 600 seconds."
+  - alert: TempoUserConfigurableOverridesReloadFailing
+    expr: sum by (instance,job)(increase(tempo_overrides_user_configurable_overrides_reload_failed_total[1h])) > 5 and sum by (instance,job)(increase(tempo_overrides_user_configurable_overrides_reload_failed_total{}[5m])) > 0
+    labels:
+      severity: critical
+    annotations:
+      summary: Tempo user config override reload failing (instance {{ $labels.instance }})
+      description: "The {{ $labels.job }} is experiencing more than 5 user-configurable override reload failures in the past hour."
diff --git a/src/tempo.py b/src/tempo.py
new file mode 100644
index 0000000..f7d5989
--- /dev/null
+++ b/src/tempo.py
@@ -0,0 +1,537 @@
+#!/usr/bin/env python3
+# Copyright 2024 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+"""Tempo workload configuration and client."""
+import logging
+import re
+import socket
+from pathlib import Path
+from subprocess import CalledProcessError, getoutput
+from typing import Dict, List, Optional, Sequence, Tuple
+from urllib.parse import urlparse
+
+import ops
+import tenacity
+import yaml
+from charms.tempo_k8s.v2.tracing import (
+    ReceiverProtocol,
+    receiver_protocol_to_transport_protocol,
+)
+from charms.traefik_route_k8s.v0.traefik_route import TraefikRouteRequirer
+from ops import ModelError
+from ops.pebble import Layer
+
+logger = logging.getLogger(__name__)
+
+
+class Tempo:
+    """Class representing the Tempo client workload configuration."""
+
+    config_path = "/etc/tempo/tempo.yaml"
+
+    # cert path on charm container
+    server_cert_path = "/usr/local/share/ca-certificates/ca.crt"
+
+    # cert paths on tempo container
+    tls_cert_path = "/etc/tempo/tls/server.crt"
+    tls_key_path = "/etc/tempo/tls/server.key"
+    tls_ca_path = "/usr/local/share/ca-certificates/ca.crt"
+
+    _tls_min_version = ""
+    # cfr https://grafana.com/docs/enterprise-traces/latest/configure/reference/#supported-contents-and-default-values
+    # "VersionTLS12"
+
+    wal_path = "/etc/tempo/tempo_wal"
+    log_path = "/var/log/tempo.log"
+    tempo_ready_notice_key = "canonical.com/tempo/workload-ready"
+
+    s3_relation_name = "s3"
+    s3_bucket_name = "tempo"
+
+    memberlist_port = 7946
+
+    server_ports = {
+        "tempo_http": 3200,
+        "tempo_grpc": 9096,  # default grpc listen port is 9095, but that conflicts with promtail.
+    }
+
+    receiver_ports: Dict[str, int] = {
+        "zipkin": 9411,
+        "otlp_grpc": 4317,
+        "otlp_http": 4318,
+        "jaeger_thrift_http": 14268,
+        # todo if necessary add support for:
+        #  "kafka": 42,
+        #  "jaeger_grpc": 14250,
+        #  "opencensus": 43,
+        #  "jaeger_thrift_compact": 44,
+        #  "jaeger_thrift_binary": 45,
+    }
+
+    all_ports = {**server_ports, **receiver_ports}
+
+    def __init__(
+        self,
+        container: ops.Container,
+        external_host: Optional[str] = None,
+        enable_receivers: Optional[Sequence[ReceiverProtocol]] = None,
+        run_worker_node: bool = True,
+    ):
+        # ports source: https://github.com/grafana/tempo/blob/main/example/docker-compose/local/docker-compose.yaml
+
+        # fqdn, if an ingress is not available, else the ingress address.
+        self._external_hostname = external_host or socket.getfqdn()
+        self.container = container
+        self.run_worker_node = run_worker_node
+        self.enabled_receivers = enable_receivers or []
+
+    @property
+    def tempo_http_server_port(self) -> int:
+        """Return the receiver port for the built-in tempo_http protocol."""
+        return self.server_ports["tempo_http"]
+
+    @property
+    def tempo_grpc_server_port(self) -> int:
+        """Return the receiver port for the built-in tempo_http protocol."""
+        return self.server_ports["tempo_grpc"]
+
+    def get_external_ports(self, service_name_prefix: str) -> List[Tuple[str, int, int]]:
+        """List of service names and port mappings for the kubernetes service patch.
+
+        Includes the tempo server as well as the receiver ports.
+        """
+        # todo allow remapping ports?
+        all_ports = {**self.server_ports}
+        return [
+            (
+                (f"{service_name_prefix}-{service_name}").replace("_", "-"),
+                all_ports[service_name],
+                all_ports[service_name],
+            )
+            for service_name in all_ports
+        ]
+
+    @property
+    def url(self) -> str:
+        """Base url at which the tempo server is locally reachable over http."""
+        scheme = "https" if self.tls_ready else "http"
+        return f"{scheme}://{self._external_hostname}"
+
+    def get_receiver_url(self, protocol: ReceiverProtocol, ingress: TraefikRouteRequirer):
+        """Return the receiver endpoint URL based on the protocol.
+
+        if ingress is used, return endpoint provided by the ingress instead.
+        """
+        protocol_type = receiver_protocol_to_transport_protocol.get(protocol)
+        # ingress.is_ready returns True even when traefik hasn't sent any data yet
+        has_ingress = ingress.is_ready() and ingress.external_host and ingress.scheme
+        receiver_port = self.receiver_ports[protocol]
+
+        if has_ingress:
+            url = (
+                ingress.external_host
+                if protocol_type == "grpc"
+                else f"{ingress.scheme}://{ingress.external_host}"
+            )
+        else:
+            url = self._external_hostname if protocol_type == "grpc" else self.url
+
+        return f"{url}:{receiver_port}"
+
+    def plan(self):
+        """Update pebble plan and start the tempo-ready service."""
+        if not self.run_worker_node:
+            logger.info("will not plan tempo container: this node is a coordinator")
+            return
+
+        self.container.add_layer("tempo", self.pebble_layer, combine=True)
+        self.container.add_layer("tempo-ready", self.tempo_ready_layer, combine=True)
+        try:
+            self.container.replan()
+            # is not autostart-enabled, we just run it once on pebble-ready.
+            self.container.start("tempo-ready")
+        except ops.pebble.ChangeError:
+            # replan failed likely because address was still in use. try to (re)start tempo with backoff as a fallback
+            restart_result = self.restart()
+            if not restart_result:
+                logger.exception(
+                    "Starting tempo failed with a ChangeError and restart attempts didn't resolve the issue"
+                )
+
+    def update_config(
+        self,
+        requested_receivers: Sequence[ReceiverProtocol],
+        s3_config: Optional[dict] = None,
+        peers: Optional[List[str]] = None,
+    ) -> bool:
+        """Generate a config and push it to the container it if necessary."""
+        container = self.container
+        if not container.can_connect():
+            logger.debug("Container can't connect: config update skipped.")
+            return False
+
+        new_config = self.generate_config(requested_receivers, s3_config, peers)
+
+        if self.get_current_config() != new_config:
+            logger.debug("Pushing new config to container...")
+            container.push(
+                self.config_path,
+                yaml.safe_dump(new_config),
+                make_dirs=True,
+            )
+            return True
+        return False
+
+    @property
+    def is_tempo_service_defined(self) -> bool:
+        """Check that the tempo service is present in the plan."""
+        try:
+            self.container.get_service("tempo")
+            return True
+        except (ModelError, ops.pebble.ConnectionError):
+            return False
+
+    def restart(self) -> bool:
+        """Try to restart the tempo service."""
+        # restarting tempo can cause errors such as:
+        #  Could not bind to :3200 - Address in use
+        # probably because of some lag with releasing the port. We restart tempo 'too quickly'
+        # and it fails to start. As a workaround, see the @retry logic below.
+        if not self.run_worker_node:
+            return False
+
+        if not self.is_tempo_service_defined:
+            self.plan()
+
+        return self._restart()
+
+    @tenacity.retry(
+        # if restart FAILS (this function returns False)
+        retry=tenacity.retry_if_result(lambda r: r is False),
+        # we wait 3, 9, 27... up to 40 seconds between tries
+        wait=tenacity.wait_exponential(multiplier=3, min=1, max=40),
+        # we give up after 20 attempts
+        stop=tenacity.stop_after_attempt(20),
+        # if there's any exceptions throughout, raise them
+        reraise=True,
+    )
+    def _restart(self) -> bool:
+        if not self.container.can_connect():
+            return False
+
+        try:
+            is_started = self.is_running
+        except ModelError:
+            is_started = False
+
+        # verify if tempo is already inactive, then try to start a new instance
+        if is_started:
+            try:
+                self.container.restart("tempo")
+            except ops.pebble.ChangeError:
+                # if tempo fails to start, we'll try again after some backoff
+                return False
+        else:
+            try:
+                self.container.start("tempo")
+            except ops.pebble.ChangeError:
+                # if tempo fails to start, we'll try again after retry backoff
+                return False
+
+        # set the notice to start checking for tempo server readiness so we don't have to
+        # wait for an update-status
+        self.container.start("tempo-ready")
+        return True
+
+    def shutdown(self):
+        """Gracefully shutdown the tempo process."""
+        for service in ["tempo", "tempo-ready"]:
+            if self.container.get_service(service).is_running():
+                self.container.stop(service)
+                logger.info(f"stopped {service}")
+
+    @property
+    def is_running(self) -> bool:
+        return self.container.get_service("tempo").is_running()
+
+    def get_current_config(self) -> Optional[dict]:
+        """Fetch the current configuration from the container."""
+        if not self.container.can_connect():
+            return None
+        try:
+            return yaml.safe_load(self.container.pull(self.config_path))
+        except ops.pebble.PathError:
+            return None
+
+    def configure_tls(self, *, cert: str, key: str, ca: str):
+        """Push cert, key and CA to the tempo container."""
+        # we save the cacert in the charm container too (for notices)
+        Path(self.server_cert_path).write_text(ca)
+
+        self.container.push(self.tls_cert_path, cert, make_dirs=True)
+        self.container.push(self.tls_key_path, key, make_dirs=True)
+        self.container.push(self.tls_ca_path, ca, make_dirs=True)
+        self.container.exec(["update-ca-certificates"])
+
+    def clear_tls_config(self):
+        """Remove cert, key and CA files from the tempo container."""
+        self.container.remove_path(self.tls_cert_path, recursive=True)
+        self.container.remove_path(self.tls_key_path, recursive=True)
+        self.container.remove_path(self.tls_ca_path, recursive=True)
+
+    @property
+    def tls_ready(self) -> bool:
+        """Whether cert, key, and ca paths are found on disk and Tempo is ready to use tls."""
+        if not self.container.can_connect():
+            return False
+        return all(
+            self.container.exists(tls_path)
+            for tls_path in (self.tls_cert_path, self.tls_key_path, self.tls_ca_path)
+        )
+
+    def _build_server_config(self):
+        server_config = {
+            "http_listen_port": self.tempo_http_server_port,
+            # we need to specify a grpc server port even if we're not using the grpc server,
+            # otherwise it will default to 9595 and make promtail bork
+            "grpc_listen_port": self.tempo_grpc_server_port,
+        }
+        if self.tls_ready:
+            for cfg in ("http_tls_config", "grpc_tls_config"):
+                server_config[cfg] = {  # type: ignore
+                    "cert_file": str(self.tls_cert_path),
+                    "key_file": str(self.tls_key_path),
+                    "client_ca_file": str(self.tls_ca_path),
+                    "client_auth_type": "VerifyClientCertIfGiven",
+                }
+            server_config["tls_min_version"] = self._tls_min_version  # type: ignore
+
+        return server_config
+
+    def generate_config(
+        self,
+        receivers: Sequence[ReceiverProtocol],
+        s3_config: Optional[dict] = None,
+        peers: Optional[List[str]] = None,
+    ) -> dict:
+        """Generate the Tempo configuration.
+
+        Only activate the provided receivers.
+        """
+        config = {
+            "auth_enabled": False,
+            "server": self._build_server_config(),
+            # more configuration information can be found at
+            # https://github.com/open-telemetry/opentelemetry-collector/tree/overlord/receiver
+            "distributor": {"receivers": self._build_receivers_config(receivers)},
+            # the length of time after a trace has not received spans to consider it complete and flush it
+            # cut the head block when it hits this number of traces or ...
+            #   this much time passes
+            "ingester": {
+                "trace_idle_period": "10s",
+                "max_block_bytes": 100,
+                "max_block_duration": "30m",
+            },
+            "memberlist": {
+                "abort_if_cluster_join_fails": False,
+                "bind_port": self.memberlist_port,
+                "join_members": (
+                    [f"{peer}:{self.memberlist_port}" for peer in peers] if peers else []
+                ),
+            },
+            "compactor": {
+                "compaction": {
+                    # blocks in this time window will be compacted together
+                    "compaction_window": "1h",
+                    # maximum size of compacted blocks
+                    "max_compaction_objects": 1000000,
+                    # total trace retention
+                    "block_retention": "720h",
+                    "compacted_block_retention": "1h",
+                    "v2_out_buffer_bytes": 5242880,
+                }
+            },
+            # TODO this won't work for distributed coordinator where query frontend will be on a different unit
+            "querier": {
+                "frontend_worker": {"frontend_address": f"localhost:{self.tempo_grpc_server_port}"}
+            },
+            # see https://grafana.com/docs/tempo/latest/configuration/#storage
+            "storage": self._build_storage_config(s3_config),
+        }
+
+        if self.tls_ready:
+            # cfr:
+            # https://grafana.com/docs/tempo/latest/configuration/network/tls/#client-configuration
+            tls_config = {
+                "tls_enabled": True,
+                "tls_cert_path": self.tls_cert_path,
+                "tls_key_path": self.tls_key_path,
+                "tls_ca_path": self.tls_ca_path,
+                # try with fqdn?
+                "tls_server_name": self._external_hostname,
+            }
+            config["ingester_client"] = {"grpc_client_config": tls_config}
+            config["metrics_generator_client"] = {"grpc_client_config": tls_config}
+
+            config["querier"]["frontend_worker"].update({"grpc_client_config": tls_config})
+
+            # this is not an error.
+            config["memberlist"].update(tls_config)
+
+        return config
+
+    def _build_storage_config(self, s3_config: Optional[dict] = None):
+        storage_config = {
+            "wal": {
+                # where to store the wal locally
+                "path": self.wal_path
+            },
+            "pool": {
+                # number of traces per index record
+                "max_workers": 400,
+                "queue_depth": 20000,
+            },
+        }
+        if s3_config:
+            storage_config.update(
+                {
+                    "backend": "s3",
+                    "s3": {
+                        "bucket": s3_config["bucket"],
+                        "access_key": s3_config["access-key"],
+                        # remove scheme to avoid "Endpoint url cannot have fully qualified paths." on Tempo startup
+                        "endpoint": re.sub(
+                            rf"^{urlparse(s3_config['endpoint']).scheme}://",
+                            "",
+                            s3_config["endpoint"],
+                        ),
+                        "secret_key": s3_config["secret-key"],
+                        "insecure": (
+                            False if s3_config["endpoint"].startswith("https://") else True
+                        ),
+                    },
+                }
+            )
+        else:
+            storage_config.update(
+                {
+                    "backend": "local",
+                    "local": {"path": "/traces"},
+                }
+            )
+        return {"trace": storage_config}
+
+    def can_scale(self) -> bool:
+        """Return whether this tempo instance can scale, i.e., whether s3 is configured."""
+        config = self.get_current_config()
+        if not config:
+            return False
+        return config["storage"]["trace"]["backend"] == "s3"
+
+    @property
+    def pebble_layer(self) -> Layer:
+        """Generate the pebble layer for the Tempo container."""
+        target = "scalable-single-binary" if self.can_scale() else "all"
+        return Layer(
+            {
+                "services": {
+                    "tempo": {
+                        "override": "replace",
+                        "summary": "Main Tempo layer",
+                        "command": f'/bin/sh -c "/tempo -config.file={self.config_path} -target {target} | tee {self.log_path}"',
+                        "startup": "enabled",
+                    }
+                },
+            }
+        )
+
+    @property
+    def tempo_ready_layer(self) -> Layer:
+        """Generate the pebble layer to fire the tempo-ready custom notice."""
+        s = "s" if self.tls_ready else ""
+        return Layer(
+            {
+                "services": {
+                    "tempo-ready": {
+                        "override": "replace",
+                        "summary": "Notify charm when tempo is ready",
+                        "command": f"""watch -n 5 '[ $(wget -q -O- --no-check-certificate http{s}://localhost:{self.tempo_http_server_port}/ready) = "ready" ] &&
+                                   ( /charm/bin/pebble notify {self.tempo_ready_notice_key} ) ||
+                                   ( echo "tempo not ready" )'""",
+                        "startup": "disabled",
+                    }
+                },
+            }
+        )
+
+    def is_ready(self):
+        """Whether the tempo built-in readiness check reports 'ready'."""
+        if self.tls_ready:
+            tls, s = f" --cacert {self.server_cert_path}", "s"
+        else:
+            tls = s = ""
+
+        # cert is for fqdn/ingress, not for IP
+        cmd = f"curl{tls} http{s}://{self._external_hostname}:{self.tempo_http_server_port}/ready"
+
+        try:
+            out = getoutput(cmd).split("\n")[-1]
+        except (CalledProcessError, IndexError):
+            return False
+        return out == "ready"
+
+    def _build_receivers_config(self, receivers: Sequence[ReceiverProtocol]):  # noqa: C901
+        # receivers: the receivers we have to enable because the requirers we're related to
+        # intend to use them
+        # it already includes self.enabled_receivers: receivers we have to enable because *this charm* will use them.
+        receivers_set = set(receivers)
+
+        if not receivers_set:
+            logger.warning("No receivers set. Tempo will be up but not functional.")
+
+        if self.tls_ready:
+            receiver_config = {
+                "tls": {
+                    "ca_file": str(self.tls_ca_path),
+                    "cert_file": str(self.tls_cert_path),
+                    "key_file": str(self.tls_key_path),
+                    "min_version": self._tls_min_version,
+                }
+            }
+        else:
+            receiver_config = None
+
+        config = {}
+
+        if "zipkin" in receivers_set:
+            config["zipkin"] = receiver_config
+        if "opencensus" in receivers_set:
+            config["opencensus"] = receiver_config
+
+        otlp_config = {}
+        if "otlp_http" in receivers_set:
+            otlp_config["http"] = receiver_config
+        if "otlp_grpc" in receivers_set:
+            otlp_config["grpc"] = receiver_config
+        if otlp_config:
+            config["otlp"] = {"protocols": otlp_config}
+
+        jaeger_config = {}
+        if "jaeger_thrift_http" in receivers_set:
+            jaeger_config["thrift_http"] = receiver_config
+        if "jaeger_grpc" in receivers_set:
+            jaeger_config["grpc"] = receiver_config
+        if "jaeger_thrift_binary" in receivers_set:
+            jaeger_config["thrift_binary"] = receiver_config
+        if "jaeger_thrift_compact" in receivers_set:
+            jaeger_config["thrift_compact"] = receiver_config
+        if jaeger_config:
+            config["jaeger"] = {"protocols": jaeger_config}
+
+        return config
+
+    def receive_tempo_ready_notice(self):
+        """Handle the tempo-ready-pebble-notice event by turning off the notice itself."""
+        if self.container.get_services("tempo-ready"):
+            self.container.stop("tempo-ready")
diff --git a/src/tempo_cluster.py b/src/tempo_cluster.py
index 28b19d0..4d6ed99 100644
--- a/src/tempo_cluster.py
+++ b/src/tempo_cluster.py
@@ -2,194 +2,110 @@
 # Copyright 2024 Canonical
 # See LICENSE file for licensing details.
 
-"""This module contains an endpoint wrapper class for the provider side of the ``tempo-cluster`` relation.
+"""This module contains an endpoint wrapper class for the requirer side of the ``tempo-cluster`` relation.
 
-As this relation is cluster-internal and not intended for third-party charms to interact with `tempo-coordinator-k8s`, its only user will be the tempo-coordinator-k8s charm. As such, it does not live in a charm lib as most other relation endpoint wrappers do.
+As this relation is cluster-internal and not intended for third-party charms to interact with
+`tempo-coordinator-k8s`, its only user will be the tempo-worker-k8s charm. As such,
+it does not live in a charm lib as most other relation endpoint wrappers do.
 """
-
-
+import collections
 import json
 import logging
-from collections import defaultdict
-from enum import Enum
-from typing import Any, Dict, Iterable, List, MutableMapping, Optional, Set
+from enum import Enum, unique
+from typing import Any, Dict, List, MutableMapping, Optional, Set, cast
 
 import ops
 import pydantic
-from ops import Object
+
+# The only reason we need the tracing lib is this enum. Not super nice.
+from charms.tempo_k8s.v2.tracing import ReceiverProtocol
+from ops import EventSource, Object, ObjectEvents
 from pydantic import BaseModel, ConfigDict
 
 log = logging.getLogger("tempo_cluster")
 
 DEFAULT_ENDPOINT_NAME = "tempo-cluster"
 BUILTIN_JUJU_KEYS = {"ingress-address", "private-address", "egress-subnets"}
-TEMPO_CONFIG_FILE = "/etc/tempo/tempo-config.yaml"
-TEMPO_CERT_FILE = "/etc/tempo/server.cert"
-TEMPO_KEY_FILE = "/etc/tempo/private.key"
-TEMPO_CLIENT_CA_FILE = "/etc/tempo/ca.cert"
 
 
+# TODO: inherit enum.StrEnum when jammy is no longer supported.
+# https://docs.python.org/3/library/enum.html#enum.StrEnum
+@unique
 class TempoRole(str, Enum):
-    """Tempo component role names."""
+    """Tempo component role names.
+
+    References:
+     arch:
+      -> https://grafana.com/docs/tempo/latest/operations/architecture/
+     config:
+      -> https://grafana.com/docs/tempo/latest/configuration/#server
+    """
+
+    # scalable-single-binary is a bit too long to type
+    all = "scalable-single-binary"  # default, meta-role.
 
-    overrides_exporter = "overrides-exporter"
-    query_scheduler = "query-scheduler"
-    flusher = "flusher"
-    query_frontend = "query-frontend"
     querier = "querier"
-    store_gateway = "store-gateway"
+    query_frontend = "query-frontend"
     ingester = "ingester"
     distributor = "distributor"
-    ruler = "ruler"
-    alertmanager = "alertmanager"
     compactor = "compactor"
+    metrics_generator = "metrics-generator"
 
-    # meta-roles
-    read = "read"
-    write = "write"
-    backend = "backend"
-    all = "all"
-
-
-META_ROLES = {
-    TempoRole.read: (TempoRole.query_frontend, TempoRole.querier),
-    TempoRole.write: (TempoRole.distributor, TempoRole.ingester),
-    TempoRole.backend: (
-        TempoRole.store_gateway,
-        TempoRole.compactor,
-        TempoRole.ruler,
-        TempoRole.alertmanager,
-        TempoRole.query_scheduler,
-        TempoRole.overrides_exporter,
-    ),
-    TempoRole.all: list(TempoRole),
-}
-
-
-def expand_roles(roles: Iterable[TempoRole]) -> Set[TempoRole]:
-    """Expand any meta roles to their 'atomic' equivalents."""
-    expanded_roles = set()
-    for role in roles:
-        if role in META_ROLES:
-            expanded_roles.update(META_ROLES[role])
-        else:
-            expanded_roles.add(role)
-    return expanded_roles
+    @property
+    def all_nonmeta(self):
+        return (
+            TempoRole.querier,
+            TempoRole.query_frontend,
+            TempoRole.ingester,
+            TempoRole.distributor,
+            TempoRole.compactor,
+            TempoRole.metrics_generator,
+        )
 
 
-class TempoClusterProvider(Object):
-    """``tempo-cluster`` provider endpoint wrapper."""
+class ConfigReceivedEvent(ops.EventBase):
+    """Event emitted when the "tempo-cluster" provider has shared a new tempo config."""
 
-    def __init__(
-        self,
-        charm: ops.CharmBase,
-        key: Optional[str] = None,
-        endpoint: str = DEFAULT_ENDPOINT_NAME,
-    ):
-        super().__init__(charm, key)
-        self._charm = charm
-        self._relations = self.model.relations[endpoint]
+    config: Dict[str, Any]
+    """The tempo config."""
 
-    def publish_data(
-        self,
-        tempo_config: Dict[str, Any],
-        loki_endpoints: Optional[Dict[str, str]] = None,
-    ) -> None:
-        """Publish the tempo config and loki endpoints to all related tempo worker clusters."""
-        for relation in self._relations:
-            if relation:
-                local_app_databag = TempoClusterProviderAppData(
-                    tempo_config=tempo_config, loki_endpoints=loki_endpoints
-                )
-                local_app_databag.dump(relation.data[self.model.app])
+    def __init__(self, handle: ops.framework.Handle, config: Dict[str, Any]):
+        super().__init__(handle)
+        self.config = config
 
-    def gather_roles(self) -> Dict[TempoRole, int]:
-        """Go through the worker's app databags and sum the available application roles."""
-        data = {}
-        for relation in self._relations:
-            if relation.app:
-                remote_app_databag = relation.data[relation.app]
-                try:
-                    worker_roles: List[TempoRole] = TempoClusterRequirerAppData.load(
-                        remote_app_databag
-                    ).roles
-                except DataValidationError as e:
-                    log.info(f"invalid databag contents: {e}")
-                    worker_roles = []
+    def snapshot(self) -> Dict[str, Any]:
+        """Used by the framework to serialize the event to disk.
 
-                # the number of units with each role is the number of remote units
-                role_n = len(relation.units)  # exclude this unit
+        Not meant to be called by charm code.
+        """
+        return {"config": json.dumps(self.config)}
 
-                for role in expand_roles(worker_roles):
-                    if role not in data:
-                        data[role] = 0
-                    data[role] += role_n
-        return data
+    def restore(self, snapshot: Dict[str, Any]):
+        """Used by the framework to deserialize the event from disk.
 
-    def gather_addresses_by_role(self) -> Dict[str, Set[str]]:
-        """Go through the worker's unit databags to collect all the addresses published by the units, by role."""
-        data = defaultdict(set)
-        for relation in self._relations:
-            if not relation.app:
-                log.debug(f"skipped {relation} as .app is None")
-                continue
+        Not meant to be called by charm code.
+        """
+        self.relation = json.loads(snapshot["config"])  # noqa
 
-            try:
-                worker_app_data = TempoClusterRequirerAppData.load(relation.data[relation.app])
-                worker_roles = set(worker_app_data.roles)
-            except DataValidationError as e:
-                log.info(f"invalid databag contents: {e}")
-                continue
 
-            for worker_unit in relation.units:
-                try:
-                    worker_data = TempoClusterRequirerUnitData.load(relation.data[worker_unit])
-                    unit_address = worker_data.address
-                    for role in worker_roles:
-                        data[role].add(unit_address)
-                except DataValidationError as e:
-                    log.info(f"invalid databag contents: {e}")
-                    continue
+class TempoClusterError(Exception):
+    """Base class for exceptions raised by this module."""
 
-        return data
 
-    def gather_addresses(self) -> Set[str]:
-        """Go through the worker's unit databags to collect all the addresses published by the units."""
-        data = set()
-        addresses_by_role = self.gather_addresses_by_role()
-        for role, address_set in addresses_by_role.items():
-            data.update(address_set)
+class DataValidationError(TempoClusterError):
+    """Raised when relation databag validation fails."""
 
-        return data
 
-    def get_datasource_address(self) -> Optional[str]:
-        """Get datasource address."""
-        addresses_by_role = self.gather_addresses_by_role()
-        if address_set := addresses_by_role.get("ruler", None):
-            return address_set.pop()
+class DatabagAccessPermissionError(TempoClusterError):
+    """Raised when a follower attempts to write leader settings."""
 
-    def gather_topology(self) -> List[Dict[str, str]]:
-        """Gather Topology."""
-        data = []
-        for relation in self._relations:
-            if not relation.app:
-                continue
 
-            for worker_unit in relation.units:
-                try:
-                    worker_data = TempoClusterRequirerUnitData.load(relation.data[worker_unit])
-                    unit_address = worker_data.address
-                except DataValidationError as e:
-                    log.info(f"invalid databag contents: {e}")
-                    continue
-                worker_topology = {
-                    "unit": worker_unit.name,
-                    "app": worker_unit.app.name,
-                    "address": unit_address,
-                }
-                data.append(worker_topology)
+class JujuTopology(pydantic.BaseModel):
+    """JujuTopology."""
 
-        return data
+    model: str
+    unit: str
+    # ...
 
 
 class DatabagModel(BaseModel):
@@ -200,16 +116,16 @@ class DatabagModel(BaseModel):
         populate_by_name=True,
         # Custom config key: whether to nest the whole datastructure (as json)
         # under a field or spread it out at the toplevel.
-        _NEST_UNDER=None,
+        _NEST_UNDER=None,  # noqa
     )  # type: ignore
     """Pydantic config."""
 
     @classmethod
     def load(cls, databag: MutableMapping[str, str]):
         """Load this model from a Juju databag."""
-        nest_under = cls.model_config.get("_NEST_UNDER")
+        nest_under = cls.model_config.get("_NEST_UNDER")  # noqa
         if nest_under:
-            return cls.parse_obj(json.loads(databag[nest_under]))
+            return cls.parse_obj(json.loads(databag[cast(str, nest_under)]))
 
         try:
             data = {k: json.loads(v) for k, v in databag.items() if k not in BUILTIN_JUJU_KEYS}
@@ -236,7 +152,7 @@ def dump(self, databag: Optional[MutableMapping[str, str]] = None, clear: bool =
 
         if databag is None:
             databag = {}
-        nest_under = self.model_config.get("_NEST_UNDER")
+        nest_under = self.model_config.get("_NEST_UNDER")  # noqa
         if nest_under:
             databag[nest_under] = self.json()
 
@@ -247,12 +163,17 @@ def dump(self, databag: Optional[MutableMapping[str, str]] = None, clear: bool =
         return databag
 
 
-class JujuTopology(pydantic.BaseModel):
-    """JujuTopology."""
+class TempoClusterRequirerAppData(DatabagModel):
+    """TempoClusterRequirerAppData."""
 
-    model: str
-    unit: str
-    # ...
+    role: TempoRole
+
+
+class TempoClusterRequirerUnitData(DatabagModel):
+    """TempoClusterRequirerUnitData."""
+
+    juju_topology: JujuTopology
+    address: str
 
 
 class TempoClusterProviderAppData(DatabagModel):
@@ -260,28 +181,160 @@ class TempoClusterProviderAppData(DatabagModel):
 
     tempo_config: Dict[str, Any]
     loki_endpoints: Optional[Dict[str, str]] = None
+    ca_cert: Optional[str] = None
+    server_cert: Optional[str] = None
+    privkey_secret_id: Optional[str] = None
+    tempo_receiver: Optional[Dict[ReceiverProtocol, str]] = None
 
 
-class TempoClusterRequirerAppData(DatabagModel):
-    """TempoClusterRequirerAppData."""
+class TempoClusterChangedEvent(ops.EventBase):
+    """Event emitted when any "tempo-cluster" relation event fires."""
 
-    roles: List[TempoRole]
 
+class TempoClusterProviderEvents(ObjectEvents):
+    """Events emitted by the TempoClusterProvider "tempo-cluster" endpoint wrapper."""
 
-class TempoClusterRequirerUnitData(DatabagModel):
-    """TempoClusterRequirerUnitData."""
+    changed = EventSource(TempoClusterChangedEvent)
 
-    juju_topology: JujuTopology
-    address: str
 
+class TempoClusterProvider(Object):
+    """``tempo-cluster`` provider endpoint wrapper."""
 
-class TempoClusterError(Exception):
-    """Base class for exceptions raised by this module."""
+    on = TempoClusterProviderEvents()  # type: ignore
 
+    def __init__(
+        self,
+        charm: ops.CharmBase,
+        key: Optional[str] = None,
+        endpoint: str = DEFAULT_ENDPOINT_NAME,
+    ):
+        super().__init__(charm, key or endpoint)
+        self._charm = charm
+        self.juju_topology = {"unit": self.model.unit.name, "model": self.model.name}
+
+        # filter out common unhappy relation states
+        self._relations: List[ops.Relation] = [
+            rel for rel in self.model.relations[endpoint] if (rel.app and rel.data)
+        ]
+
+        # we coalesce all tempo-cluster-relation-* events into a single cluster-changed API.
+        # the coordinator uses a common exit hook reconciler, that's why.
+        self.framework.observe(
+            self._charm.on[endpoint].relation_joined, self._on_tempo_cluster_changed
+        )
+        self.framework.observe(
+            self._charm.on[endpoint].relation_changed, self._on_tempo_cluster_changed
+        )
+        self.framework.observe(
+            self._charm.on[endpoint].relation_departed, self._on_tempo_cluster_changed
+        )
+        self.framework.observe(
+            self._charm.on[endpoint].relation_broken, self._on_tempo_cluster_changed
+        )
+
+    def _on_tempo_cluster_changed(self, _):
+        self.on.changed.emit()
+
+    def publish_privkey(self, label: str) -> str:
+        """Grant the secret containing the privkey to all relations, and return the secret ID."""
+        secret = self.model.get_secret(label=label)
+        for relation in self._relations:
+            secret.grant(relation)
+        # can't return secret.id because secret was obtained by label, and so
+        # we don't have an ID unless we fetch it
+        return secret.get_info().id
 
-class DataValidationError(TempoClusterError):
-    """Raised when relation databag validation fails."""
+    def publish_data(
+        self,
+        tempo_config: Dict[str, Any],
+        tempo_receiver: Dict[str, Any] = None,
+        ca_cert: str = None,
+        server_cert: str = None,
+        privkey_secret_id: str = None,
+        loki_endpoints: Optional[Dict[str, str]] = None,
+    ) -> None:
+        """Publish the tempo config to all related tempo worker clusters."""
+        for relation in self._relations:
+            if relation:
+                local_app_databag = TempoClusterProviderAppData(
+                    tempo_config=tempo_config,
+                    loki_endpoints=loki_endpoints,
+                    tempo_receiver=tempo_receiver,
+                    ca_cert=ca_cert,
+                    server_cert=server_cert,
+                    privkey_secret_id=privkey_secret_id,
+                )
+                local_app_databag.dump(relation.data[self.model.app])
 
+    @property
+    def has_workers(self) -> bool:
+        """Return whether this tempo coordinator has any connected workers."""
+        # we use the presence of relations instead of addresses, because we want this
+        # check to fail early
+        return bool(self._relations)
 
-class DatabagAccessPermissionError(TempoClusterError):
-    """Raised when a follower attempts to write leader settings."""
+    def gather_addresses_by_role(self) -> Dict[str, Set[str]]:
+        """Go through the worker's unit databags to collect all the addresses published by the units, by role."""
+        data = collections.defaultdict(set)
+        for relation in self._relations:
+
+            if not relation.app:
+                log.debug(f"skipped {relation} as .app is None")
+                continue
+
+            try:
+                worker_app_data = TempoClusterRequirerAppData.load(relation.data[relation.app])
+                worker_roles = set(worker_app_data.roles)
+            except DataValidationError as e:
+                log.info(f"invalid databag contents: {e}")
+                continue
+
+            for worker_unit in relation.units:
+                try:
+                    worker_data = TempoClusterRequirerUnitData.load(relation.data[worker_unit])
+                    unit_address = worker_data.address
+                    for role in worker_roles:
+                        data[role].add(unit_address)
+                except DataValidationError as e:
+                    log.info(f"invalid databag contents: {e}")
+                    continue
+
+        return data
+
+    def gather_addresses(self) -> Set[str]:
+        """Go through the worker's unit databags to collect all the addresses published by the units."""
+        data = set()
+        addresses_by_role = self.gather_addresses_by_role()
+        for role, address_set in addresses_by_role.items():
+            data.update(address_set)
+
+        return data
+
+    def gather_roles(self, base: collections.Counter = None) -> Dict[TempoRole, int]:
+        """Go through the worker's app databags and sum the available application roles."""
+        data = base or collections.Counter()
+        for relation in self._relations:
+            if relation.app:
+                remote_app_databag = relation.data[relation.app]
+                try:
+                    worker_role: TempoRole = TempoClusterRequirerAppData.load(
+                        remote_app_databag
+                    ).role
+                except DataValidationError as e:
+                    log.info(f"invalid databag contents: {e}")
+                    continue
+
+                # the number of units with each role is the number of remote units
+                role_n = len(relation.units)  # exclude this unit
+                if worker_role is TempoRole.all:
+                    for role in [r for r in TempoRole if r is not TempoRole.all]:
+                        data[role] += role_n
+                    continue
+
+                data[worker_role] += role_n
+
+        dct = dict(data)
+        # exclude all roles from the count, if any slipped through
+        if TempoRole.all in data:
+            del data[TempoRole.all]
+        return dct
diff --git a/tests/catan/test_clustering.py b/tests/catan/test_clustering.py
new file mode 100644
index 0000000..72da1e4
--- /dev/null
+++ b/tests/catan/test_clustering.py
@@ -0,0 +1,162 @@
+import json
+import os
+from unittest.mock import patch
+
+import pytest
+import scenario
+from catan import App, Catan
+from scenario import Container, State
+
+os.environ["CHARM_TRACING_ENABLED"] = "0"
+
+
+def get_facade(name="facade"):
+    facade = App.from_path(
+        "/home/pietro/canonical/charm-relation-interfaces/facade_charm",
+        name=name,
+    )
+    return facade
+
+
+@pytest.fixture
+def s3_facade() -> App:
+    return get_facade("s3")
+
+
+def prep_s3_facade(catan: Catan):
+    catan.run_action(
+        scenario.Action(
+            "update",
+            params={
+                "endpoint": json.dumps("provides-s3"),
+                "app_data": json.dumps(
+                    {
+                        "access-key": "key",
+                        "bucket": "tempo",
+                        "endpoint": "http://1.2.3.4:9000",
+                        "secret-key": "soverysecret",
+                    }
+                ),
+            },
+        ),
+        catan.get_app("s3"),
+    )
+
+
+@pytest.fixture
+def ca_facade() -> App:
+    return get_facade("ca")
+
+
+def prep_ca_facade(catan: Catan):
+    catan.run_action(
+        scenario.Action(
+            "update",
+            params={"endpoint": json.dumps("provides-certificates"), "app_data": json.dumps("")},
+        ),
+        catan.get_app("ca"),
+    )
+
+
+@pytest.fixture
+def traefik_facade() -> App:
+    return get_facade("traefik")
+
+
+def prep_traefik_facade(catan: Catan):
+    catan.run_action(
+        scenario.Action(
+            "update",
+            params={
+                "endpoint": json.dumps("provides-traefik_route"),
+                "app_data": json.dumps({"external_host": "1.2.3.4", "scheme": "http"}),
+            },
+        ),
+        catan.get_app("ca"),
+    )
+
+
+@pytest.fixture
+def tempo_coordinator():
+    tempo = App.from_path(
+        "/home/pietro/canonical/tempo-k8s",
+        patches=[
+            patch("charm.KubernetesServicePatch"),
+            patch("lightkube.core.client.GenericSyncClient"),
+            patch("charm.TempoCharm._update_server_cert"),
+            patch("tempo.Tempo.is_ready", new=lambda _: True),
+        ],
+        name="tempo",
+    )
+    yield tempo
+
+
+@pytest.fixture
+def tempo_worker():
+    tempo = App.from_path(
+        "/home/pietro/canonical/tempo-worker-k8s-operator",
+        patches=[patch("charm.KubernetesServicePatch")],
+        name="tempo",
+    )
+    yield tempo
+
+
+@pytest.fixture
+def tempo_coordinator_state():
+    return State(
+        containers=[Container(name="tempo", can_connect=True)],
+    )
+
+
+@pytest.fixture
+def tempo_worker_state():
+    return State(
+        containers=[Container(name="tempo", can_connect=True)],
+    )
+
+
+def assert_active(state: State, message=None):
+    assert state.unit_status.name == "active"
+    if message:
+        assert state.unit_status.message == message
+
+
+def assert_blocked(state: State, message=None):
+    assert state.unit_status.name == "blocked"
+    if message:
+        assert state.unit_status.message == message
+
+
+def test_monolithic_deployment(tempo_coordinator, tempo_coordinator_state):
+    c = Catan()
+    c.deploy(tempo_coordinator, [0], state_template=tempo_coordinator_state)
+
+    ms_out = c.settle()
+
+    state_out = ms_out.get_unit_state(tempo_coordinator, 0)
+    assert_active(state_out, "")
+
+
+@pytest.mark.xfail  # catan doesn't support peer relations yet
+def test_scaling_without_s3_blocks(tempo_coordinator, tempo_coordinator_state, s3_facade):
+    c = Catan()
+    c.deploy(tempo_coordinator, [0, 1], state_template=tempo_coordinator_state)
+
+    ms_out = c.settle()
+
+    assert_blocked(ms_out.get_unit_state(tempo_coordinator, 0))
+    assert_blocked(ms_out.get_unit_state(tempo_coordinator, 1))
+
+
+def test_scaling_with_s3_active(tempo_coordinator, tempo_coordinator_state, s3_facade):
+    c = Catan()
+    c.deploy(tempo_coordinator, [0], state_template=tempo_coordinator_state)
+    c.deploy(s3_facade, [0])
+    prep_s3_facade(c)
+    c.integrate(tempo_coordinator, "s3", s3_facade, "provide-s3")
+    c.add_unit(tempo_coordinator, 1, tempo_coordinator_state)
+
+    ms_out = c.settle()
+
+    assert_active(ms_out.get_unit_state(tempo_coordinator, 0))
+    assert_active(ms_out.get_unit_state(tempo_coordinator, 1))
diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py
index 6c08a60..1accd8d 100644
--- a/tests/integration/conftest.py
+++ b/tests/integration/conftest.py
@@ -1,57 +1,76 @@
-#!/usr/bin/env python3
-# Copyright 2023 Canonical Ltd.
+# Copyright 2021 Canonical Ltd.
 # See LICENSE file for licensing details.
-
-import functools
 import logging
-from collections import defaultdict
-from datetime import datetime
+import os
+import shutil
+from pathlib import Path
 
-import pytest
+import yaml
+from pytest import fixture
+from pytest_operator.plugin import OpsTest
 
 logger = logging.getLogger(__name__)
 
 
-class Store(defaultdict):
-    def __init__(self):
-        super(Store, self).__init__(Store)
+@fixture(scope="module")
+async def tempo_charm(ops_test: OpsTest):
+    """Zinc charm used for integration testing."""
+    charm = await ops_test.build_charm(".")
+    return charm
+
 
-    def __getattr__(self, key):
-        """Override __getattr__ so dot syntax works on keys."""
-        try:
-            return self[key]
-        except KeyError:
-            raise AttributeError(key)
+@fixture(scope="module")
+def tempo_metadata(ops_test: OpsTest):
+    return yaml.safe_load(Path("./metadata.yaml").read_text())
 
-    def __setattr__(self, key, value):
-        """Override __setattr__ so dot syntax works on keys."""
-        self[key] = value
 
+@fixture(scope="module")
+def tempo_oci_image(ops_test: OpsTest, tempo_metadata):
+    return tempo_metadata["resources"]["tempo-image"]["upstream-source"]
 
-store = Store()
 
+@fixture(scope="module", autouse=True)
+def copy_charm_libs_into_tester_charm(ops_test):
+    """Ensure the tester charm has the libraries it uses."""
+    libraries = [
+        "observability_libs/v1/cert_handler.py",
+        "tls_certificates_interface/v3/tls_certificates.py",
+        "tempo_k8s/v1/charm_tracing.py",
+        "tempo_k8s/v2/tracing.py",
+    ]
 
-def timed_memoizer(func):
-    @functools.wraps(func)
-    async def wrapper(*args, **kwargs):
-        fname = func.__qualname__
-        logger.info("Started: %s" % fname)
-        start_time = datetime.now()
-        if fname in store.keys():
-            ret = store[fname]
-        else:
-            logger.info("Return for {} not cached".format(fname))
-            ret = await func(*args, **kwargs)
-            store[fname] = ret
-        logger.info("Finished: {} in: {} seconds".format(fname, datetime.now() - start_time))
-        return ret
+    copies = []
 
-    return wrapper
+    for lib in libraries:
+        install_path = f"tests/integration/tester/lib/charms/{lib}"
+        os.makedirs(os.path.dirname(install_path), exist_ok=True)
+        shutil.copyfile(f"lib/charms/{lib}", install_path)
+        copies.append(install_path)
 
+    yield
 
-@pytest.fixture(scope="module")
-@timed_memoizer
-async def tempo_charm(ops_test):
-    """Tempo charm used for integration testing."""
-    charm = await ops_test.build_charm(".")
-    return charm
+    # cleanup: remove all libs
+    for path in copies:
+        Path(path).unlink()
+
+
+@fixture(scope="module", autouse=True)
+def copy_charm_libs_into_tester_grpc_charm(ops_test):
+    """Ensure the tester GRPC charm has the libraries it uses."""
+    libraries = [
+        "tempo_k8s/v2/tracing.py",
+    ]
+
+    copies = []
+
+    for lib in libraries:
+        install_path = f"tests/integration/tester-grpc/lib/charms/{lib}"
+        os.makedirs(os.path.dirname(install_path), exist_ok=True)
+        shutil.copyfile(f"lib/charms/{lib}", install_path)
+        copies.append(install_path)
+
+    yield
+
+    # cleanup: remove all libs
+    for path in copies:
+        Path(path).unlink()
diff --git a/tests/integration/helpers.py b/tests/integration/helpers.py
index e340345..ef9c6b0 100644
--- a/tests/integration/helpers.py
+++ b/tests/integration/helpers.py
@@ -1,92 +1,159 @@
-# Copyright 2022 Canonical Ltd.
-# See LICENSE file for licensing details.
-
-import logging
 import subprocess
-from pathlib import Path
+from dataclasses import dataclass
+from typing import Dict
 
 import yaml
-from pytest_operator.plugin import OpsTest
-
-logger = logging.getLogger(__name__)
 
-
-async def deploy_literal_bundle(ops_test: OpsTest, bundle: str):
-    run_args = [
-        "juju",
-        "deploy",
-        "--trust",
-        "-m",
-        ops_test.model_name,
-        str(ops_test.render_bundle(bundle)),
+_JUJU_DATA_CACHE = {}
+_JUJU_KEYS = ("egress-subnets", "ingress-address", "private-address")
+
+
+def purge(data: dict):
+    for key in _JUJU_KEYS:
+        if key in data:
+            del data[key]
+
+
+def get_unit_info(unit_name: str, model: str = None) -> dict:
+    """Return unit-info data structure.
+
+     for example:
+
+    traefik-k8s/0:
+      opened-ports: []
+      charm: local:focal/traefik-k8s-1
+      leader: true
+      relation-info:
+      - endpoint: ingress-per-unit
+        related-endpoint: ingress
+        application-data:
+          _supported_versions: '- v1'
+        related-units:
+          prometheus-k8s/0:
+            in-scope: true
+            data:
+              egress-subnets: 10.152.183.150/32
+              ingress-address: 10.152.183.150
+              private-address: 10.152.183.150
+      provider-id: traefik-k8s-0
+      address: 10.1.232.144
+    """
+    cmd = f"juju show-unit {unit_name}".split(" ")
+    if model:
+        cmd.insert(2, "-m")
+        cmd.insert(3, model)
+
+    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+    raw_data = proc.stdout.read().decode("utf-8").strip()
+
+    data = yaml.safe_load(raw_data) if raw_data else None
+
+    if not data:
+        raise ValueError(
+            f"no unit info could be grabbed for {unit_name}; "
+            f"are you sure it's a valid unit name?"
+            f"cmd={' '.join(proc.args)}"
+        )
+
+    if unit_name not in data:
+        raise KeyError(unit_name, f"not in {data!r}")
+
+    unit_data = data[unit_name]
+    _JUJU_DATA_CACHE[unit_name] = unit_data
+    return unit_data
+
+
+def get_relation_by_endpoint(relations, local_endpoint, remote_endpoint, remote_obj):
+    matches = [
+        r
+        for r in relations
+        if (
+            (r["endpoint"] == local_endpoint and r["related-endpoint"] == remote_endpoint)
+            or (r["endpoint"] == remote_endpoint and r["related-endpoint"] == local_endpoint)
+        )
+        and remote_obj in r["related-units"]
     ]
+    if not matches:
+        raise ValueError(
+            f"no matches found with endpoint=="
+            f"{local_endpoint} "
+            f"in {remote_obj} (matches={matches})"
+        )
+    if len(matches) > 1:
+        raise ValueError(
+            "multiple matches found with endpoint=="
+            f"{local_endpoint} "
+            f"in {remote_obj} (matches={matches})"
+        )
+    return matches[0]
+
+
+@dataclass
+class UnitRelationData:
+    unit_name: str
+    endpoint: str
+    leader: bool
+    application_data: Dict[str, str]
+    unit_data: Dict[str, str]
+
+
+def get_content(
+    obj: str, other_obj, include_default_juju_keys: bool = False, model: str = None
+) -> UnitRelationData:
+    """Get the content of the databag of `obj`, as seen from `other_obj`."""
+    unit_name, endpoint = obj.split(":")
+    other_unit_name, other_endpoint = other_obj.split(":")
+
+    unit_data, app_data, leader = get_databags(
+        unit_name, endpoint, other_unit_name, other_endpoint, model
+    )
+
+    if not include_default_juju_keys:
+        purge(unit_data)
+
+    return UnitRelationData(unit_name, endpoint, leader, app_data, unit_data)
+
+
+def get_databags(local_unit, local_endpoint, remote_unit, remote_endpoint, model):
+    """Get the databags of local unit and its leadership status.
+
+    Given a remote unit and the remote endpoint name.
+    """
+    local_data = get_unit_info(local_unit, model)
+    leader = local_data["leader"]
+
+    data = get_unit_info(remote_unit, model)
+    relation_info = data.get("relation-info")
+    if not relation_info:
+        raise RuntimeError(f"{remote_unit} has no relations")
 
-    retcode, stdout, stderr = await ops_test.run(*run_args)
-    assert retcode == 0, f"Deploy failed: {(stderr or stdout).strip()}"
-    logger.info(stdout)
+    raw_data = get_relation_by_endpoint(relation_info, local_endpoint, remote_endpoint, local_unit)
+    unit_data = raw_data["related-units"][local_unit]["data"]
+    app_data = raw_data["application-data"]
+    return unit_data, app_data, leader
 
 
-def oci_image(metadata_file: str, image_name: str) -> str:
-    """Find upstream source for a container image.
+@dataclass
+class RelationData:
+    provider: UnitRelationData
+    requirer: UnitRelationData
 
-    Args:
-        metadata_file: string path of metadata YAML file relative
-            to top level charm directory
-        image_name: OCI container image string name as defined in
-            metadata.yaml file
 
-    Returns:
-        upstream image source
+def get_relation_data(
+    *,
+    provider_endpoint: str,
+    requirer_endpoint: str,
+    include_default_juju_keys: bool = False,
+    model: str = None,
+):
+    """Get relation databags for a juju relation.
 
-    Raises:
-        FileNotFoundError: if metadata_file path is invalid
-        ValueError: if upstream source for image name can not be found
+    >>> get_relation_data('prometheus/0:ingress', 'traefik/1:ingress-per-unit')
     """
-    metadata = yaml.safe_load(Path(metadata_file).read_text())
-
-    resources = metadata.get("resources", {})
-    if not resources:
-        raise ValueError("No resources found")
-
-    image = resources.get(image_name, {})
-    if not image:
-        raise ValueError(f"{image_name} image not found")
-
-    upstream_source = image.get("upstream-source", "")
-    if not upstream_source:
-        raise ValueError("Upstream source not found")
-
-    return upstream_source
-
-
-def get_workload_file(
-    model_name: str, app_name: str, unit_num: int, container_name: str, filepath: str
-) -> bytes:
-    cmd = [
-        "juju",
-        "ssh",
-        "--model",
-        model_name,
-        "--container",
-        container_name,
-        f"{app_name}/{unit_num}",
-        "cat",
-        filepath,
-    ]
-    try:
-        res = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
-    except subprocess.CalledProcessError as e:
-        logger.error(e.stdout.decode())
-        raise e
-    return res.stdout
-
-
-async def run_command(model_name: str, app_name: str, unit_num: int, command: list) -> bytes:
-    cmd = ["juju", "ssh", "--model", model_name, f"{app_name}/{unit_num}", *command]
-    try:
-        res = subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
-        logger.info(res)
-    except subprocess.CalledProcessError as e:
-        logger.error(e.stdout.decode())
-        raise e
-    return res.stdout
+    provider_data = get_content(
+        provider_endpoint, requirer_endpoint, include_default_juju_keys, model
+    )
+    requirer_data = get_content(
+        requirer_endpoint, provider_endpoint, include_default_juju_keys, model
+    )
+    return RelationData(provider=provider_data, requirer=requirer_data)
diff --git a/tests/integration/test_ingressed_tls.py b/tests/integration/test_ingressed_tls.py
new file mode 100644
index 0000000..03e35ce
--- /dev/null
+++ b/tests/integration/test_ingressed_tls.py
@@ -0,0 +1,180 @@
+import asyncio
+import json
+import logging
+import random
+import subprocess
+import tempfile
+from pathlib import Path
+
+import pytest
+import requests
+import yaml
+from pytest_operator.plugin import OpsTest
+from tenacity import retry, stop_after_attempt, wait_exponential
+
+from tempo import Tempo
+from tests.integration.helpers import get_relation_data
+
+METADATA = yaml.safe_load(Path("./charmcraft.yaml").read_text())
+APP_NAME = "tempo"
+SSC = "self-signed-certificates"
+SSC_APP_NAME = "ssc"
+TRAEFIK = "traefik-k8s"
+TRAEFIK_APP_NAME = "trfk"
+TRACEGEN_SCRIPT_PATH = Path() / "scripts" / "tracegen.py"
+
+logger = logging.getLogger(__name__)
+
+
+@pytest.fixture(scope="function")
+def nonce():
+    """Generate an integer nonce for easier trace querying."""
+    return str(random.random())[2:]
+
+
+@pytest.fixture(scope="function")
+def server_cert(ops_test: OpsTest):
+    data = get_relation_data(
+        requirer_endpoint=f"{APP_NAME}/0:certificates",
+        provider_endpoint=f"{SSC_APP_NAME}/0:certificates",
+        model=ops_test.model.name,
+    )
+    cert = json.loads(data.provider.application_data["certificates"])[0]["certificate"]
+
+    with tempfile.NamedTemporaryFile() as f:
+        p = Path(f.name)
+        p.write_text(cert)
+        yield p
+
+
+def get_traces(tempo_host: str, nonce, service_name="tracegen"):
+    req = requests.get(
+        "https://" + tempo_host + ":3200/api/search",
+        params={"service.name": service_name, "nonce": nonce},
+        verify=False,
+    )
+    assert req.status_code == 200
+    return json.loads(req.text)["traces"]
+
+
+@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, min=4, max=10))
+async def get_traces_patiently(tempo_host, nonce):
+    assert get_traces(tempo_host, nonce=nonce)
+
+
+async def get_tempo_host(ops_test: OpsTest):
+    status = await ops_test.model.get_status()
+    app = status["applications"][TRAEFIK_APP_NAME]
+    return app.public_address
+
+
+async def emit_trace(
+    endpoint, ops_test: OpsTest, nonce, proto: str = "http", verbose=0, use_cert=False
+):
+    """Use juju ssh to run tracegen from the tempo charm; to avoid any DNS issues."""
+    cmd = (
+        f"juju ssh -m {ops_test.model_name} {APP_NAME}/0 "
+        f"TRACEGEN_ENDPOINT={endpoint} "
+        f"TRACEGEN_VERBOSE={verbose} "
+        f"TRACEGEN_PROTOCOL={proto} "
+        f"TRACEGEN_CERT={Tempo.server_cert_path if use_cert else ''} "
+        f"TRACEGEN_NONCE={nonce} "
+        "python3 tracegen.py"
+    )
+
+    return subprocess.getoutput(cmd)
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_build_and_deploy(ops_test: OpsTest):
+    tempo_charm = await ops_test.build_charm(".")
+    resources = {
+        "tempo-image": METADATA["resources"]["tempo-image"]["upstream-source"],
+    }
+    await asyncio.gather(
+        ops_test.model.deploy(tempo_charm, resources=resources, application_name=APP_NAME),
+        ops_test.model.deploy(SSC, application_name=SSC_APP_NAME),
+        ops_test.model.deploy(TRAEFIK, application_name=TRAEFIK_APP_NAME, channel="edge"),
+    )
+
+    await asyncio.gather(
+        ops_test.model.wait_for_idle(
+            apps=[APP_NAME, SSC_APP_NAME, TRAEFIK_APP_NAME],
+            status="active",
+            raise_on_blocked=True,
+            timeout=10000,
+            raise_on_error=False,
+        ),
+    )
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_push_tracegen_script_and_deps(ops_test: OpsTest):
+    await ops_test.juju("scp", TRACEGEN_SCRIPT_PATH, f"{APP_NAME}/0:tracegen.py")
+    await ops_test.juju(
+        "ssh",
+        f"{APP_NAME}/0",
+        "python3 -m pip install opentelemetry-exporter-otlp-proto-grpc opentelemetry-exporter-otlp-proto-http",
+    )
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_relate(ops_test: OpsTest):
+    await ops_test.model.integrate(APP_NAME + ":certificates", SSC_APP_NAME + ":certificates")
+    await ops_test.model.integrate(
+        SSC_APP_NAME + ":certificates", TRAEFIK_APP_NAME + ":certificates"
+    )
+    await ops_test.model.integrate(APP_NAME + ":ingress", TRAEFIK_APP_NAME + ":traefik-route")
+    await ops_test.model.wait_for_idle(
+        apps=[APP_NAME, SSC_APP_NAME, TRAEFIK_APP_NAME],
+        status="active",
+        timeout=1000,
+    )
+
+
+@pytest.mark.abort_on_fail
+async def test_verify_ingressed_trace_http_upgrades_to_tls(ops_test: OpsTest, nonce):
+    tempo_host = await get_tempo_host(ops_test)
+    # IF tempo is related to SSC
+    # WHEN we emit an http trace, **unsecured**
+    await emit_trace(
+        f"http://{tempo_host}:4318/v1/traces", nonce=nonce, ops_test=ops_test
+    )  # this should fail
+    # THEN we can verify it's not been ingested
+    assert get_traces_patiently(tempo_host, nonce=nonce)
+
+
+@pytest.mark.abort_on_fail
+async def test_verify_ingressed_trace_http_tls(ops_test: OpsTest, nonce, server_cert):
+    tempo_host = await get_tempo_host(ops_test)
+    await emit_trace(
+        f"https://{tempo_host}:4318/v1/traces", nonce=nonce, ops_test=ops_test, use_cert=True
+    )
+    # THEN we can verify it's been ingested
+    assert get_traces_patiently(tempo_host, nonce=nonce)
+
+
+@pytest.mark.abort_on_fail
+async def test_verify_ingressed_traces_grpc_tls(ops_test: OpsTest, nonce, server_cert):
+    tempo_host = await get_tempo_host(ops_test)
+    await emit_trace(
+        f"{tempo_host}:4317", nonce=nonce, proto="grpc", ops_test=ops_test, use_cert=True
+    )
+    # THEN we can verify it's been ingested
+    assert get_traces_patiently(tempo_host, nonce=nonce)
+
+
+@pytest.mark.teardown
+@pytest.mark.abort_on_fail
+async def test_remove_relation(ops_test: OpsTest):
+    await ops_test.juju(
+        "remove-relation", APP_NAME + ":certificates", SSC_APP_NAME + ":certificates"
+    )
+    await asyncio.gather(
+        ops_test.model.wait_for_idle(
+            apps=[APP_NAME], status="active", raise_on_blocked=True, timeout=1000
+        ),
+    )
diff --git a/tests/integration/test_integration.py b/tests/integration/test_integration.py
new file mode 100644
index 0000000..2fc39a6
--- /dev/null
+++ b/tests/integration/test_integration.py
@@ -0,0 +1,168 @@
+import asyncio
+import json
+import logging
+from pathlib import Path
+
+import pytest
+import yaml
+from pytest_operator.plugin import OpsTest
+
+METADATA = yaml.safe_load(Path("./charmcraft.yaml").read_text())
+APP_NAME = METADATA["name"]
+TESTER_METADATA = yaml.safe_load(Path("./tests/integration/tester/metadata.yaml").read_text())
+TESTER_APP_NAME = TESTER_METADATA["name"]
+TESTER_GRPC_METADATA = yaml.safe_load(
+    Path("./tests/integration/tester-grpc/metadata.yaml").read_text()
+)
+TESTER_GRPC_APP_NAME = TESTER_GRPC_METADATA["name"]
+
+logger = logging.getLogger(__name__)
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_build_and_deploy(ops_test: OpsTest):
+    # Given a fresh build of the charm
+    # When deploying it together with testers
+    # Then applications should eventually be created
+    tempo_charm = await ops_test.build_charm(".")
+    tester_charm = await ops_test.build_charm("./tests/integration/tester/")
+    tester_grpc_charm = await ops_test.build_charm("./tests/integration/tester-grpc/")
+    resources = {
+        "tempo-image": METADATA["resources"]["tempo-image"]["upstream-source"],
+    }
+    resources_tester = {"workload": TESTER_METADATA["resources"]["workload"]["upstream-source"]}
+    resources_tester_grpc = {
+        "workload": TESTER_GRPC_METADATA["resources"]["workload"]["upstream-source"]
+    }
+
+    await asyncio.gather(
+        ops_test.model.deploy(tempo_charm, resources=resources, application_name=APP_NAME),
+        ops_test.model.deploy(
+            tester_charm,
+            resources=resources_tester,
+            application_name=TESTER_APP_NAME,
+            num_units=3,
+        ),
+        ops_test.model.deploy(
+            tester_grpc_charm,
+            resources=resources_tester_grpc,
+            application_name=TESTER_GRPC_APP_NAME,
+            num_units=3,
+        ),
+    )
+
+    await asyncio.gather(
+        ops_test.model.wait_for_idle(
+            apps=[APP_NAME],
+            status="active",
+            raise_on_blocked=True,
+            timeout=10000,
+            raise_on_error=False,
+        ),
+        # for tester, depending on the result of race with tempo it's either waiting or active
+        ops_test.model.wait_for_idle(
+            apps=[TESTER_APP_NAME], raise_on_blocked=True, timeout=1000, raise_on_error=False
+        ),
+        ops_test.model.wait_for_idle(
+            apps=[TESTER_GRPC_APP_NAME], raise_on_blocked=True, timeout=1000, raise_on_error=False
+        ),
+    )
+
+    assert ops_test.model.applications[APP_NAME].units[0].workload_status == "active"
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_relate(ops_test: OpsTest):
+    # given a deployed charm
+    # when relating it together with the tester
+    # then relation should appear
+    await ops_test.model.add_relation(APP_NAME + ":tracing", TESTER_APP_NAME + ":tracing")
+    await ops_test.model.add_relation(APP_NAME + ":tracing", TESTER_GRPC_APP_NAME + ":tracing")
+    await ops_test.model.wait_for_idle(
+        apps=[APP_NAME, TESTER_APP_NAME, TESTER_GRPC_APP_NAME],
+        status="active",
+        timeout=1000,
+    )
+
+
+async def test_verify_traces_http(ops_test: OpsTest):
+    # given a relation between charms
+    # when traces endpoint is queried
+    # then it should contain traces from tester charm
+    status = await ops_test.model.get_status()
+    app = status["applications"][APP_NAME]
+    logger.info(app.public_address)
+    endpoint = app.public_address + ":3200/api/search"
+    cmd = [
+        "curl",
+        endpoint,
+    ]
+    rc, stdout, stderr = await ops_test.run(*cmd)
+    logger.info("%s: %s", endpoint, (rc, stdout, stderr))
+    assert rc == 0, (
+        f"curl exited with rc={rc} for {endpoint}; "
+        f"non-zero return code means curl encountered a >= 400 HTTP code; "
+        f"cmd={cmd}"
+    )
+    traces = json.loads(stdout)["traces"]
+
+    found = False
+    for trace in traces:
+        if trace["rootServiceName"] == APP_NAME and trace["rootTraceName"] == "charm exec":
+            found = True
+
+    assert found, f"There's no trace of charm exec traces in tempo. {json.dumps(traces, indent=2)}"
+
+
+async def test_verify_traces_grpc(ops_test: OpsTest):
+    # the tester-grpc charm emits a single grpc trace in its common exit hook
+    # we verify it's there
+    status = await ops_test.model.get_status()
+    app = status["applications"][APP_NAME]
+    logger.info(app.public_address)
+    endpoint = app.public_address + ":3200/api/search"
+    cmd = [
+        "curl",
+        endpoint,
+    ]
+    rc, stdout, stderr = await ops_test.run(*cmd)
+    logger.info("%s: %s", endpoint, (rc, stdout, stderr))
+    assert rc == 0, (
+        f"curl exited with rc={rc} for {endpoint}; "
+        f"non-zero return code means curl encountered a >= 400 HTTP code; "
+        f"cmd={cmd}"
+    )
+    traces = json.loads(stdout)["traces"]
+
+    found = False
+    for trace in traces:
+        if trace["rootServiceName"] == "TempoTesterGrpcCharm":
+            found = True
+
+    assert (
+        found
+    ), f"There's no trace of generated grpc traces in tempo. {json.dumps(traces, indent=2)}"
+
+
+@pytest.mark.teardown
+@pytest.mark.abort_on_fail
+async def test_remove_relation(ops_test: OpsTest):
+    # given related charms
+    # when relation is removed
+    # then both charms should become active again
+    await ops_test.juju("remove-relation", APP_NAME + ":tracing", TESTER_APP_NAME + ":tracing")
+    await ops_test.juju(
+        "remove-relation", APP_NAME + ":tracing", TESTER_GRPC_APP_NAME + ":tracing"
+    )
+    await asyncio.gather(
+        ops_test.model.wait_for_idle(
+            apps=[APP_NAME], status="active", raise_on_blocked=True, timeout=1000
+        ),
+        # for tester, depending on the result of race with tempo it's either waiting or active
+        ops_test.model.wait_for_idle(apps=[TESTER_APP_NAME], raise_on_blocked=True, timeout=1000),
+        ops_test.model.wait_for_idle(
+            apps=[TESTER_GRPC_APP_NAME], raise_on_blocked=True, timeout=1000
+        ),
+    )
diff --git a/tests/integration/test_scaling_monolithic.py b/tests/integration/test_scaling_monolithic.py
new file mode 100644
index 0000000..60bd1f1
--- /dev/null
+++ b/tests/integration/test_scaling_monolithic.py
@@ -0,0 +1,119 @@
+import json
+import logging
+import shlex
+import tempfile
+from pathlib import Path
+from subprocess import run
+from typing import Dict, Literal
+
+import pytest
+import yaml
+from juju.application import Application
+from pytest_operator.plugin import OpsTest
+
+METADATA = yaml.safe_load(Path("./charmcraft.yaml").read_text())
+APP_NAME = "tempo"
+FACADE = "facade"
+TRACEGEN_SCRIPT_PATH = Path() / "scripts" / "tracegen.py"
+
+logger = logging.getLogger(__name__)
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_deploy_tempo(ops_test: OpsTest):
+    tempo_charm = await ops_test.build_charm(".")
+    resources = {"tempo-image": METADATA["resources"]["tempo-image"]["upstream-source"]}
+    await ops_test.model.deploy(tempo_charm, resources=resources, application_name=APP_NAME),
+
+    await ops_test.model.wait_for_idle(
+        apps=[APP_NAME],
+        # tempo might be in waiting as it waits for tempo workload ready
+        raise_on_blocked=True,
+        timeout=10000,
+        raise_on_error=False,
+    )
+
+
+@pytest.mark.abort_on_fail
+async def test_scale_tempo_up_without_s3_blocks(ops_test: OpsTest):
+    app: Application = ops_test.model.applications[APP_NAME]
+    await app.add_unit(1)
+
+    await ops_test.model.wait_for_idle(
+        apps=[APP_NAME],
+        status="blocked",
+        raise_on_blocked=False,
+        timeout=1000,
+    )
+
+
+def present_facade(
+    interface: str,
+    app_data: Dict = None,
+    unit_data: Dict = None,
+    role: Literal["provide", "require"] = "provide",
+    model: str = None,
+    app: str = "facade",
+):
+    """Set up the facade charm to present this data over the interface ``interface``."""
+    data = {
+        "endpoint": f"{role}-{interface}",
+    }
+    if app_data:
+        data["app_data"] = json.dumps(app_data)
+    if unit_data:
+        data["unit_data"] = json.dumps(unit_data)
+
+    with tempfile.NamedTemporaryFile() as f:
+        fpath = Path(f.name)
+        fpath.write_text(yaml.safe_dump(data))
+
+        _model = f" --model {model}" if model else ""
+        run(shlex.split(f"juju run {app}/0{_model} --params {fpath.absolute()}"))
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_tempo_active_when_deploy_s3_facade(ops_test: OpsTest):
+    await ops_test.model.deploy(FACADE, channel="edge")
+    await ops_test.model.integrate(APP_NAME + ":s3", FACADE + ":provide-s3")
+
+    present_facade(
+        "s3",
+        model=ops_test.model_name,
+        app_data={
+            "access-key": "key",
+            "bucket": "tempo",
+            "endpoint": "http://1.2.3.4:9000",
+            "secret-key": "soverysecret",
+        },
+    )
+
+    await ops_test.model.wait_for_idle(
+        apps=[FACADE],
+        raise_on_blocked=True,
+        status="active",
+        timeout=2000,
+    )
+
+    await ops_test.model.wait_for_idle(
+        apps=[APP_NAME],
+        # we can't raise on blocked as tempo will likely start at blocked
+        raise_on_blocked=False,
+        # we can't wait for a specific status as tempo
+        # might quickly go from waiting to active depending on when the notice comes in
+        timeout=1000,
+    )
+
+
+@pytest.mark.teardown
+async def test_tempo_blocks_if_s3_goes_away(ops_test: OpsTest):
+    app: Application = ops_test.model.applications[FACADE]
+    await app.destroy(destroy_storage=True)
+    await ops_test.model.wait_for_idle(
+        apps=[APP_NAME],
+        raise_on_blocked=False,
+        status="blocked",
+        timeout=1000,
+    )
diff --git a/tests/integration/test_tls.py b/tests/integration/test_tls.py
index 4e83b7a..921f811 100644
--- a/tests/integration/test_tls.py
+++ b/tests/integration/test_tls.py
@@ -1,66 +1,170 @@
-#!/usr/bin/env python3
-# Copyright 2023 Canonical Ltd.
-# See LICENSE file for licensing details.
-
 import asyncio
+import json
 import logging
+import random
+import tempfile
 from pathlib import Path
-from types import SimpleNamespace
+from subprocess import getoutput
 
 import pytest
+import requests
 import yaml
-from helpers import get_workload_file, oci_image
 from pytest_operator.plugin import OpsTest
+from tenacity import retry, stop_after_attempt, wait_exponential
+
+from tempo import Tempo
+from tests.integration.helpers import get_relation_data
 
+METADATA = yaml.safe_load(Path("./charmcraft.yaml").read_text())
+APP_NAME = "tempo"
+SSC = "self-signed-certificates"
+SSC_APP_NAME = "ssc"
+TRACEGEN_SCRIPT_PATH = Path() / "scripts" / "tracegen.py"
 logger = logging.getLogger(__name__)
 
-METADATA = yaml.safe_load(Path("./metadata.yaml").read_text())
-mc = SimpleNamespace(name="mc")
 
-tempo_app_name = "coordinator"
-ca_app_name = "ca"
-app_names = [tempo_app_name, ca_app_name]
+@pytest.fixture(scope="function")
+def nonce():
+    """Generate an integer nonce for easier trace querying."""
+    return str(random.random())[2:]
+
+
+def get_traces(tempo_host: str, nonce):
+    url = "https://" + tempo_host + ":3200/api/search"
+    req = requests.get(
+        url,
+        params={"q": f'{{ .nonce = "{nonce}" }}'},
+        # it would fail to verify as the cert was issued for fqdn, not IP.
+        verify=False,
+    )
+    assert req.status_code == 200
+    return json.loads(req.text)["traces"]
+
+
+@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1, min=4, max=10))
+async def get_traces_patiently(ops_test, nonce):
+    assert get_traces(await get_tempo_ip(ops_test), nonce=nonce)
+
+
+async def get_tempo_ip(ops_test: OpsTest):
+    status = await ops_test.model.get_status()
+    app = status["applications"][APP_NAME]
+    return app.public_address
+
 
+async def get_tempo_internal_host(ops_test: OpsTest):
+    return f"https://{APP_NAME}-0.{APP_NAME}-endpoints.{ops_test.model.name}.svc.cluster.local"
 
-def get_nginx_config(ops_test: OpsTest):
-    return get_workload_file(
-        ops_test.model_name, tempo_app_name, 0, "nginx", "/etc/nginx/nginx.conf"
+
+@pytest.fixture(scope="function")
+def server_cert(ops_test: OpsTest):
+    data = get_relation_data(
+        requirer_endpoint=f"{APP_NAME}/0:certificates",
+        provider_endpoint=f"{SSC_APP_NAME}/0:certificates",
+        model=ops_test.model.name,
+    )
+    cert = json.loads(data.provider.application_data["certificates"])[0]["certificate"]
+
+    with tempfile.NamedTemporaryFile() as f:
+        p = Path(f.name)
+        p.write_text(cert)
+        yield p
+
+
+async def emit_trace(ops_test: OpsTest, nonce, proto: str = "http", verbose=0, use_cert=False):
+    """Use juju ssh to run tracegen from the tempo charm; to avoid any DNS issues."""
+    hostname = await get_tempo_internal_host(ops_test)
+    cmd = (
+        f"juju ssh -m {ops_test.model_name} {APP_NAME}/0 "
+        f"TRACEGEN_ENDPOINT={hostname}:4318/v1/traces "
+        f"TRACEGEN_VERBOSE={verbose} "
+        f"TRACEGEN_PROTOCOL={proto} "
+        f"TRACEGEN_CERT={Tempo.server_cert_path if use_cert else ''} "
+        f"TRACEGEN_NONCE={nonce} "
+        "python3 tracegen.py"
     )
 
+    return getoutput(cmd)
 
+
+@pytest.mark.setup
 @pytest.mark.abort_on_fail
-async def test_nginx_config_has_ssl(ops_test: OpsTest):
+async def test_build_and_deploy(ops_test: OpsTest):
     tempo_charm = await ops_test.build_charm(".")
+    resources = {
+        "tempo-image": METADATA["resources"]["tempo-image"]["upstream-source"],
+    }
     await asyncio.gather(
-        ops_test.model.deploy(
-            tempo_charm,
-            resources={
-                "nginx-image": oci_image("./metadata.yaml", "nginx-image"),
-                "nginx-prometheus-exporter-image": oci_image(
-                    "./metadata.yaml", "nginx-prometheus-exporter-image"
-                ),
-            },
-            application_name="coordinator",
-            trust=True,  # otherwise errors on ghwf (persistentvolumeclaims ... is forbidden)
-        ),
-        ops_test.model.deploy(
-            "ch:self-signed-certificates",
-            application_name="ca",
-            channel="edge",
-            trust=True,
-        ),
+        ops_test.model.deploy(tempo_charm, resources=resources, application_name=APP_NAME),
+        ops_test.model.deploy(SSC, application_name=SSC_APP_NAME),
     )
 
     await asyncio.gather(
-        ops_test.model.wait_for_idle(apps=[tempo_app_name], status="blocked"),
-        ops_test.model.wait_for_idle(apps=[ca_app_name], status="active"),
+        ops_test.model.wait_for_idle(
+            apps=[APP_NAME, SSC_APP_NAME],
+            status="active",
+            raise_on_blocked=True,
+            timeout=10000,
+            raise_on_error=False,
+        ),
     )
-    await ops_test.model.add_relation(tempo_app_name, ca_app_name)
-    await asyncio.gather(
-        ops_test.model.wait_for_idle(apps=[tempo_app_name], status="blocked"),
-        ops_test.model.wait_for_idle(apps=[ca_app_name], status="active"),
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_relate(ops_test: OpsTest):
+    await ops_test.model.integrate(APP_NAME + ":certificates", SSC_APP_NAME + ":certificates")
+    await ops_test.model.wait_for_idle(
+        apps=[APP_NAME, SSC_APP_NAME],
+        status="active",
+        timeout=1000,
+    )
+
+
+@pytest.mark.setup
+@pytest.mark.abort_on_fail
+async def test_push_tracegen_script_and_deps(ops_test: OpsTest):
+    await ops_test.juju("scp", TRACEGEN_SCRIPT_PATH, f"{APP_NAME}/0:tracegen.py")
+    await ops_test.juju(
+        "ssh",
+        f"{APP_NAME}/0",
+        "python3 -m pip install opentelemetry-exporter-otlp-proto-grpc opentelemetry-exporter-otlp-proto-http",
     )
 
-    nginx_config = get_nginx_config(ops_test).decode()
-    assert "ssl_certificate /etc/nginx/certs/server.cert;" in nginx_config
-    assert "ssl_certificate_key /etc/nginx/certs/server.key;" in nginx_config
+
+async def test_verify_trace_http_no_tls_fails(ops_test: OpsTest, server_cert, nonce):
+    # IF tempo is related to SSC
+    # WHEN we emit an http trace, **unsecured**
+    await emit_trace(ops_test, nonce=nonce)  # this should fail
+    # THEN we can verify it's not been ingested
+    tempo_ip = await get_tempo_ip(ops_test)
+    traces = get_traces(tempo_ip, nonce=nonce)
+    assert not traces
+
+
+async def test_verify_trace_http_tls(ops_test: OpsTest, nonce, server_cert):
+    # WHEN we emit a trace secured with TLS
+    await emit_trace(ops_test, nonce=nonce, use_cert=True)
+    # THEN we can verify it's eventually ingested
+    await get_traces_patiently(ops_test, nonce)
+
+
+@pytest.mark.xfail  # expected to fail because in this context the grpc receiver is not enabled
+async def test_verify_traces_grpc_tls(ops_test: OpsTest, nonce, server_cert):
+    # WHEN we emit a trace secured with TLS
+    await emit_trace(ops_test, nonce=nonce, verbose=1, proto="grpc", use_cert=True)
+    # THEN we can verify it's been ingested
+    await get_traces_patiently(ops_test, nonce)
+
+
+@pytest.mark.teardown
+@pytest.mark.abort_on_fail
+async def test_remove_relation(ops_test: OpsTest):
+    await ops_test.juju(
+        "remove-relation", APP_NAME + ":certificates", SSC_APP_NAME + ":certificates"
+    )
+    await asyncio.gather(
+        ops_test.model.wait_for_idle(
+            apps=[APP_NAME], status="active", raise_on_blocked=True, timeout=1000
+        ),
+    )
diff --git a/tests/integration/tester-grpc/README.md b/tests/integration/tester-grpc/README.md
new file mode 100644
index 0000000..c773f1b
--- /dev/null
+++ b/tests/integration/tester-grpc/README.md
@@ -0,0 +1,2 @@
+to deploy:
+`juju deploy /path/to/tester.charm --resource workload=python:slim-buster --config tempo_endpoint:foo.bar:4318`
\ No newline at end of file
diff --git a/tests/integration/tester-grpc/charmcraft.yaml b/tests/integration/tester-grpc/charmcraft.yaml
new file mode 100644
index 0000000..234879e
--- /dev/null
+++ b/tests/integration/tester-grpc/charmcraft.yaml
@@ -0,0 +1,17 @@
+# Learn more about charmcraft.yaml configuration at:
+# https://juju.is/docs/sdk/charmcraft-config
+type: "charm"
+bases:
+  - build-on:
+    - name: "ubuntu"
+      channel: "22.04"
+    run-on:
+    - name: "ubuntu"
+      channel: "22.04"
+parts:
+  charm:
+    charm-binary-python-packages:
+      - "pydantic>=2"
+      - "cryptography"
+      - "jsonschema"
+      - "opentelemetry-exporter-otlp-proto-http==1.21.0"
\ No newline at end of file
diff --git a/tests/integration/tester-grpc/config.yaml b/tests/integration/tester-grpc/config.yaml
new file mode 100644
index 0000000..0547444
--- /dev/null
+++ b/tests/integration/tester-grpc/config.yaml
@@ -0,0 +1,14 @@
+# Copyright 2022 pietro
+# See LICENSE file for licensing details.
+
+options:
+  port:
+    description: port at which the webserver should run
+    type: int
+    default: 3000
+
+  host:
+    description: host at which the webserver should run
+    type: string
+    default: 0.0.0.0
+
diff --git a/tests/integration/tester-grpc/lib/.gitignore b/tests/integration/tester-grpc/lib/.gitignore
new file mode 100644
index 0000000..8e5bbf0
--- /dev/null
+++ b/tests/integration/tester-grpc/lib/.gitignore
@@ -0,0 +1 @@
+*.py
\ No newline at end of file
diff --git a/tests/integration/tester-grpc/lib/charms/observability_libs/v0/.gitkeep b/tests/integration/tester-grpc/lib/charms/observability_libs/v0/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester-grpc/lib/charms/prometheus_k8s/v0/.gitkeep b/tests/integration/tester-grpc/lib/charms/prometheus_k8s/v0/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester-grpc/lib/charms/tempo_k8s/v0/.gitkeep b/tests/integration/tester-grpc/lib/charms/tempo_k8s/v0/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester-grpc/lib/charms/tempo_k8s/v1/.gitkeep b/tests/integration/tester-grpc/lib/charms/tempo_k8s/v1/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester-grpc/lib/charms/tempo_k8s/v2/.gitkeep b/tests/integration/tester-grpc/lib/charms/tempo_k8s/v2/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester-grpc/metadata.yaml b/tests/integration/tester-grpc/metadata.yaml
new file mode 100644
index 0000000..136b1e9
--- /dev/null
+++ b/tests/integration/tester-grpc/metadata.yaml
@@ -0,0 +1,27 @@
+# Copyright 2022 pietro
+# See LICENSE file for licensing details.
+
+name: tester-grpc
+display-name: tester-grpc
+description: tempo tester GRPC charm
+summary: tempo tester GRPC charm
+
+containers:
+  workload:
+    resource: workload
+
+resources:
+  workload:
+    type: oci-image
+    description: OCI image for the workload
+    # Included for simplicity in integration tests
+    upstream-source: python:slim-buster
+
+requires:
+  tracing:
+    interface: tracing
+    limit: 1
+
+peers:
+  replicas:
+    interface: replicas
\ No newline at end of file
diff --git a/tests/integration/tester-grpc/requirements.txt b/tests/integration/tester-grpc/requirements.txt
new file mode 100644
index 0000000..bf39b04
--- /dev/null
+++ b/tests/integration/tester-grpc/requirements.txt
@@ -0,0 +1,2 @@
+ops >= 1.4.0
+opentelemetry-exporter-otlp-proto-grpc>=1.21.0
\ No newline at end of file
diff --git a/tests/integration/tester-grpc/src/charm.py b/tests/integration/tester-grpc/src/charm.py
new file mode 100755
index 0000000..05662d3
--- /dev/null
+++ b/tests/integration/tester-grpc/src/charm.py
@@ -0,0 +1,303 @@
+#!/usr/bin/env python3
+# Copyright 2024 Canonical Inc.
+# See LICENSE file for licensing details.
+
+import logging
+import time
+from pathlib import Path
+from typing import List, Optional
+
+from charms.tempo_k8s.v2.tracing import TracingEndpointRequirer
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
+from ops.charm import CharmBase, PebbleReadyEvent
+from ops.main import main
+from ops.model import (
+    ActiveStatus,
+    Container,
+    MaintenanceStatus,
+    Relation,
+    WaitingStatus,
+)
+from ops.pebble import Layer
+
+logger = logging.getLogger(__name__)
+TRACING_APP_NAME = "TempoTesterGrpcCharm"
+
+
+class TempoTesterGrpcCharm(CharmBase):
+    """Charm the service."""
+
+    _layer_name = _service_name = "tester"
+    _container_name = "workload"
+    _peer_relation_name = "replicas"
+    _address_name = "private-address-ip"
+
+    def __init__(self, *args):
+        super().__init__(*args)
+
+        self.container: Container = self.unit.get_container(self._container_name)
+
+        self.tracing = TracingEndpointRequirer(
+            self, relation_name="tracing", protocols=["otlp_grpc"]
+        )
+
+        # Core lifecycle events
+        self.framework.observe(self.on.config_changed, self._update)
+
+        # Peer relation events
+        self.framework.observe(self.on[self._peer_relation_name].relation_joined, self._update)
+        self.framework.observe(self.on[self._peer_relation_name].relation_changed, self._update)
+        self.framework.observe(self.tracing.on.endpoint_changed, self._update)
+
+    @property
+    def peers(self) -> List[str]:
+        return self._get_peer_addresses()
+
+    def _on_tester_pebble_ready(self, e: PebbleReadyEvent):
+        if e.workload.can_connect():
+            self._setup_container(e.workload)
+        else:
+            self.unit.status = WaitingStatus("waiting for container connectivity...")
+            return e.defer()
+        self.unit.status = ActiveStatus("ready")
+
+    def ensure_container_setup(self):
+        container = self.container
+        if container.list_files("/", pattern="webserver.py"):
+            return
+        self._setup_container(container)
+
+    def _setup_container(self, container: Container):
+        # copy the webserver file to the container. In a production environment,
+        # the workload would typically be an OCI image. Here however we have a
+        # 'bare' python container as base.
+        self.unit.status = MaintenanceStatus("copying over webserver source")
+
+        resources = Path(__file__).parent / "resources"
+        webserver_source_path = resources / "webserver.py"
+        logger.info(webserver_source_path)
+        with open(webserver_source_path, "r") as webserver_source:
+            logger.info("pushing webserver source...")
+            container.push("/webserver.py", webserver_source)
+
+        self.unit.status = MaintenanceStatus("installing software in workload container")
+        # we install the webserver dependencies; in a production environment, these
+        # would typically be baked in the workload OCI image.
+        webserver_dependencies_path = resources / "webserver-dependencies.txt"
+        logger.info(webserver_dependencies_path)
+        with open(webserver_dependencies_path, "r") as dependencies_file:
+            dependencies = dependencies_file.read().split("\n")
+            logger.info(f"installing webserver dependencies {dependencies}...")
+            container.exec(["pip", "install", *dependencies]).wait()
+
+        self.unit.status = MaintenanceStatus("container ready")
+
+    # Actual tester stuff
+    def _tester_layer(self):
+        """Returns a Pebble configuration layer for the tester runtime."""
+        env = {
+            "PORT": self.config["port"],
+            "HOST": self.config["host"],
+            "APP_NAME": self.app.name,
+            "TEMPO_ENDPOINT": str(self.tracing.get_endpoint("otlp_grpc") or ""),
+        }
+        logging.info(f"Initing pebble layer with env: {str(env)}")
+
+        if self.unit.name.split("/")[1] == "0":
+            env["PEERS"] = peers = ";".join(self.peers)
+            env["OVERLORD"] = "1"
+            logging.info(f"Configuring overlord node; Peers: {peers}")
+
+        return Layer(
+            {
+                "summary": "tester layer",
+                "description": "pebble config layer for tester",
+                "services": {
+                    "tester": {
+                        "override": "merge",
+                        "summary": "tester service",
+                        "command": "python webserver.py > webserver.log",
+                        "startup": "enabled",
+                        "environment": env,
+                    }
+                },
+            }
+        )
+
+    # source: https://github.com/canonical/alertmanager-k8s-operator
+    def _restart_service(self) -> bool:
+        """Helper function for restarting the underlying service.
+        Returns:
+            True if restart succeeded; False otherwise.
+        """
+        logger.info("Restarting service %s", self._service_name)
+
+        if not self.container.can_connect():
+            logger.error("Cannot (re)start service: container is not ready.")
+            return False
+
+        # Check if service exists, to avoid ModelError from being raised when the service does
+        # not exist,
+        if not self.container.get_plan().services.get(self._service_name):
+            logger.error("Cannot (re)start service: service does not (yet) exist.")
+            return False
+
+        logger.info(f"pebble env, {self.container.get_plan().services.get('tester').environment}")
+
+        self.container.restart(self._service_name)
+        logger.info(f"restarted {self._service_name}")
+        return True
+
+    def _update_layer(self, restart: bool) -> bool:
+        """Update service layer to reflect changes in peers (replicas).
+        Args:
+          restart: a flag indicating if the service should be restarted if a change was detected.
+        Returns:
+          True if anything changed; False otherwise
+        """
+        overlay = self._tester_layer()
+        plan = self.container.get_plan()
+
+        logger.info("updating layer")
+
+        if self._service_name not in plan.services or overlay.services != plan.services:
+            logger.info("container.add_layer")
+            self.container.add_layer(self._layer_name, overlay, combine=True)
+
+            service_exists = self.container.get_plan().services.get("tester", None) is not None
+            if service_exists and restart:
+                self._restart_service()
+
+            return True
+
+        return False
+
+    @property
+    def peer_relation(self) -> Relation:
+        """Helper function for obtaining the peer relation object.
+        Returns: peer relation object
+        (NOTE: would return None if called too early, e.g. during install).
+        """
+        return self.model.get_relation(self._peer_relation_name)
+
+    @property
+    def private_address(self) -> Optional[str]:
+        """Get the unit's ip address.
+        Technically, receiving a "joined" event guarantees an IP address is available. If this is
+        called beforehand, a None would be returned.
+        When operating a single unit, no "joined" events are visible so obtaining an address is a
+        matter of timing in that case.
+        This function is still needed in Juju 2.9.5 because the "private-address" field in the
+        data bag is being populated by the app IP instead of the unit IP.
+        Also in Juju 2.9.5, ip address may be None even after RelationJoinedEvent, for which
+        "ops.model.RelationDataError: relation data values must be strings" would be emitted.
+        Returns:
+          None if no IP is available (called before unit "joined"); unit's ip address otherwise
+        """
+
+        # if bind_address := check_output(["unit-get", "private-address"]).decode().strip()
+        if bind_address := self.model.get_binding(self._peer_relation_name).network.bind_address:
+            bind_address = str(bind_address)
+        return bind_address
+
+    def update_address_in_relation_data(self, relation):
+        """stores this unit's private IP in the relation databag"""
+        relation.data[self.unit].update({self._address_name: self.private_address})
+        logger.info(f"stored {self.private_address} in relation databag")
+
+    def _update(self, _):
+        """Event handler for all things."""
+        self.ensure_container_setup()
+
+        logger.info("running _update")
+        if not self.container.can_connect():
+            self.unit.status = WaitingStatus("waiting for container connectivity...")
+            return
+
+        # Wait for IP address. IP address is needed for forming tester clusters
+        # and for related apps' config.
+        if not self.private_address:
+            self.unit.status = MaintenanceStatus("waiting for IP address...")
+            return
+
+        if not self.tracing.is_ready():
+            self.unit.status = WaitingStatus("waiting for tracing to be ready...")
+            return
+
+        # In the case of a single unit deployment, no 'RelationJoined' event is emitted, so
+        # setting IP here.
+        # Store private address in unit's peer relation data bucket. This is still needed because
+        # the "private-address" field in the data bag is being populated incorrectly.
+        # Also, ip address may still be None even after RelationJoinedEvent, for which
+        # "ops.model.RelationDataError: relation data values must be strings" would be emitted.
+        if peer_relation := self.peer_relation:
+            self.update_address_in_relation_data(peer_relation)
+        else:
+            logger.info("no peer relation to configure")
+
+        self._update_layer(restart=True)
+        self.unit.status = ActiveStatus("ready")
+
+        self._send_grpc_traces()
+
+    def _get_peer_addresses(self) -> List[str]:
+        """Create a list of addresses of all peer units (all units excluding current).
+        The returned addresses include the port number but do not include scheme (http).
+        If a unit does not have an address, it will be omitted from the list.
+        """
+        addresses = []
+        if pr := self.peer_relation:
+            addresses = [
+                f"{address}"
+                for unit in pr.units
+                # pr.units only holds peers (self.unit is not included)
+                if (address := pr.data[unit].get(self._address_name))
+            ]
+
+        return addresses
+
+    def tempo_otlp_grpc_endpoint(self) -> Optional[str]:
+        """Endpoint at which the charm tracing information will be forwarded."""
+        if self.tracing.is_ready():
+            return self.tracing.get_endpoint("otlp_grpc")
+        else:
+            return None
+
+    @staticmethod
+    def _emit_trace(endpoint: str, log_trace_to_console: bool = False):
+        logger.info(f"Emitting traces to endpoint {endpoint}")
+        span_exporter = OTLPSpanExporter(
+            endpoint=endpoint,
+            insecure=True,
+        )
+        resource = Resource.create(attributes={"service.name": TRACING_APP_NAME})
+        provider = TracerProvider(resource=resource)
+        if log_trace_to_console:
+            processor = BatchSpanProcessor(ConsoleSpanExporter())
+            provider.add_span_processor(processor)
+        span_processor = BatchSpanProcessor(span_exporter)
+        provider.add_span_processor(span_processor)
+        trace.set_tracer_provider(provider)
+
+        tracer = trace.get_tracer(__name__)
+        logger.info(f"tracer instance: {tracer}")
+
+        with tracer.start_as_current_span("foo"):
+            with tracer.start_as_current_span("bar"):
+                with tracer.start_as_current_span("baz"):
+                    time.sleep(0.1)
+                    logger.info("spans emitted")
+
+    def _send_grpc_traces(self):
+        if self.tracing.is_ready():
+            logger.info("tracing is ready")
+            endpoint = self.tempo_otlp_grpc_endpoint()
+            self._emit_trace(endpoint)
+
+
+if __name__ == "__main__":
+    main(TempoTesterGrpcCharm)
diff --git a/tests/integration/tester-grpc/src/resources/webserver-dependencies.txt b/tests/integration/tester-grpc/src/resources/webserver-dependencies.txt
new file mode 100644
index 0000000..6834282
--- /dev/null
+++ b/tests/integration/tester-grpc/src/resources/webserver-dependencies.txt
@@ -0,0 +1,7 @@
+requests
+opentelemetry-exporter-otlp
+opentelemetry-instrumentation-fastapi
+opentelemetry-instrumentation-logging
+fastapi
+uvicorn
+prometheus_client
\ No newline at end of file
diff --git a/tests/integration/tester-grpc/src/resources/webserver.py b/tests/integration/tester-grpc/src/resources/webserver.py
new file mode 100644
index 0000000..81ae884
--- /dev/null
+++ b/tests/integration/tester-grpc/src/resources/webserver.py
@@ -0,0 +1,251 @@
+# Copyright 2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+import logging
+import os
+import random
+import time
+from typing import Tuple
+
+import requests
+import uvicorn as uvicorn
+from fastapi import FastAPI
+from fastapi.openapi.models import Response
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
+from opentelemetry.instrumentation.logging import LoggingInstrumentor
+from opentelemetry.propagate import inject
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from prometheus_client import REGISTRY, Counter, Gauge, Histogram
+from prometheus_client.openmetrics.exposition import (
+    CONTENT_TYPE_LATEST,
+    generate_latest,
+)
+from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
+from starlette.requests import Request
+from starlette.routing import Match
+from starlette.status import HTTP_500_INTERNAL_SERVER_ERROR
+from starlette.types import ASGIApp
+
+REQUESTS_PROCESSING_TIME = Histogram(
+    "fastapi_requests_duration_seconds",
+    "Histogram of requests processing time by path (in seconds)",
+    ["method", "path", "app_name"],
+)
+
+APP_NAME = os.environ.get("APP_NAME").replace("-", "_")
+TEMPO_ENDPOINT = os.environ.get("TEMPO_ENDPOINT", "")
+HOST = os.environ.get("HOST", "0.0.0.0")
+PORT = int(os.environ.get("PORT", 3000))
+
+
+if not TEMPO_ENDPOINT:
+    logging.warning("TEMPO_ENDPOINT not configured; tracing disabled. webserver dead.")
+
+
+INFO = Gauge("fastapi_app_info", "application information.", ["app_name"])
+REQUESTS = Counter(
+    f"{APP_NAME}_requests_total",
+    "Total count of requests by method and path.",
+    ["method", "path", "app_name"],
+)
+RESPONSES = Counter(
+    f"{APP_NAME}_responses_total",
+    "Total count of responses by method, path and status codes.",
+    ["method", "path", "status_code", "app_name"],
+)
+REQUESTS_PROCESSING_TIME = Histogram(
+    f"{APP_NAME}_requests_duration_seconds",
+    "Histogram of requests processing time by path (in seconds)",
+    ["method", "path", "app_name"],
+)
+EXCEPTIONS = Counter(
+    f"{APP_NAME}_exceptions_total",
+    "Total count of exceptions raised by path and exception type",
+    ["method", "path", "exception_type", "app_name"],
+)
+REQUESTS_IN_PROGRESS = Gauge(
+    f"{APP_NAME}_requests_in_progress",
+    "Gauge of requests by method and path currently being processed",
+    ["method", "path", "app_name"],
+)
+
+
+class PrometheusMiddleware(BaseHTTPMiddleware):
+    def __init__(self, app: ASGIApp, app_name: str = APP_NAME) -> None:
+        super().__init__(app)
+        self.app_name = app_name
+        INFO.labels(app_name=self.app_name).inc()
+
+    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
+        method = request.method
+        path, is_handled_path = self.get_path(request)
+
+        if not is_handled_path:
+            return await call_next(request)
+
+        REQUESTS_IN_PROGRESS.labels(method=method, path=path, app_name=self.app_name).inc()
+        REQUESTS.labels(method=method, path=path, app_name=self.app_name).inc()
+        before_time = time.perf_counter()
+        try:
+            response = await call_next(request)
+        except BaseException as e:
+            status_code = HTTP_500_INTERNAL_SERVER_ERROR
+            EXCEPTIONS.labels(
+                method=method, path=path, exception_type=type(e).__name__, app_name=self.app_name
+            ).inc()
+            raise e from None
+        else:
+            status_code = response.status_code
+            after_time = time.perf_counter()
+            # retrieve trace id for exemplar
+            span = trace.get_current_span()
+            trace_id = trace.format_trace_id(span.get_span_context().trace_id)
+
+            REQUESTS_PROCESSING_TIME.labels(
+                method=method, path=path, app_name=self.app_name
+            ).observe(after_time - before_time, exemplar={"TraceID": trace_id})
+        finally:
+            RESPONSES.labels(
+                method=method, path=path, status_code=status_code, app_name=self.app_name
+            ).inc()
+            REQUESTS_IN_PROGRESS.labels(method=method, path=path, app_name=self.app_name).dec()
+
+        return response
+
+    @staticmethod
+    def get_path(request: Request) -> Tuple[str, bool]:
+        for route in request.app.routes:
+            match, child_scope = route.matches(request.scope)
+            if match == Match.FULL:
+                return route.path, True
+
+        return request.url.path, False
+
+
+def init_overlord():
+    logging.debug("Initing as overlord.")
+    # manual instrumentation for overlord node
+    resource = Resource.create(attributes={"service.name": APP_NAME, "compose_service": APP_NAME})
+
+    provider = TracerProvider(resource=resource)
+    processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=TEMPO_ENDPOINT))
+    provider.add_span_processor(processor)
+    trace.set_tracer_provider(provider)
+    tracer = trace.get_tracer(__name__)
+    subordinates = os.environ.get("PEERS").split(";")
+
+    print(f"preparing to send traces to {subordinates}")
+
+    while True:
+        with tracer.start_as_current_span(
+            "query_peers", attributes={"service.name": APP_NAME}
+        ) as span:
+            for i, subordinate in enumerate(subordinates):
+                url = "http://" + subordinate + f":{PORT}/query/{i}"
+
+                print(f"sending {i} to {subordinate}")
+                span.add_event(
+                    "log",
+                    {
+                        "service.name": APP_NAME,
+                        "subordinate.id": i,
+                        "subordinate.root_url": subordinate,
+                        "subordinate.query_url": subordinate,
+                    },
+                )
+
+                with tracer.start_as_current_span(
+                    f"query_peer_{i}", attributes={"service.name": APP_NAME}
+                ) as child_span:
+                    headers = {}
+                    inject(headers)  # inject trace id into request headers.
+
+                    resp = requests.get(url, headers=headers)
+                    print(f"received {resp} from {subordinate}")
+                    try:
+                        id_, value_ = resp.text.split("=")
+                    except:  # noqa
+                        id_ = value_ = None
+                        print(f"error handling response from {url}: invalid text: {resp.text}")
+
+                    child_span.add_event(
+                        "log",
+                        {
+                            "service.name": APP_NAME,
+                            "subordinate.response.id": id_,
+                            "subordinate.response.value": value_,
+                        },
+                    )
+        provider.force_flush()
+
+
+def init_subordinate():
+    logging.debug("Initing as subordinate.")
+    # auto instrumentation for subordinate nodes
+    app = FastAPI()
+    app.add_middleware(PrometheusMiddleware, app_name=APP_NAME)
+
+    @app.get("/metrics")
+    def metrics():
+        return Response(generate_latest(REGISTRY), headers={"Content-Type": CONTENT_TYPE_LATEST})
+
+    @app.get("/query/{var}")
+    async def query(var: str):
+        randn = random.random()
+        time.sleep(randn)
+
+        # add request processing time metric with exemplar
+        span = trace.get_current_span()
+        trace_id = trace.format_trace_id(span.get_span_context().trace_id)
+        REQUESTS_PROCESSING_TIME.labels(
+            method="GET", path="/query/{var}", app_name=APP_NAME
+        ).observe(randn, exemplar={"TraceID": trace_id})
+        return f"{var}={randn}"
+
+    def setup_otlp(
+        app: ASGIApp, app_name: str, endpoint: str, log_correlation: bool = True
+    ) -> None:
+        # Setting OpenTelemetry
+        # set the service name to show in traces
+        resource = Resource.create(
+            attributes={"service.name": app_name, "compose_service": app_name}
+        )
+
+        # set the tracer provider
+        tracer = TracerProvider(resource=resource)
+        trace.set_tracer_provider(tracer)
+
+        tracer.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint)))
+
+        if log_correlation:
+            LoggingInstrumentor().instrument(set_logging_format=True)
+
+        FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer)
+
+    setup_otlp(app, APP_NAME, TEMPO_ENDPOINT)
+    uvicorn.run(app, host=HOST, port=PORT)
+
+
+class EndpointFilter(logging.Filter):
+    # Uvicorn endpoint access log filter
+    def filter(self, record: logging.LogRecord) -> bool:
+        return record.getMessage().find("GET /metrics") == -1
+
+
+# Filter out /endpoint
+logging.getLogger("uvicorn.access").addFilter(EndpointFilter())
+
+
+if __name__ == "__main__":
+    if not TEMPO_ENDPOINT:
+        while True:
+            logging.warning("IDLING. TEMPO_ENDPOINT unavailable.")
+            time.sleep(10)
+
+    if os.environ.get("OVERLORD"):
+        init_overlord()
+    else:
+        init_subordinate()
diff --git a/tests/integration/tester/README.md b/tests/integration/tester/README.md
new file mode 100644
index 0000000..c773f1b
--- /dev/null
+++ b/tests/integration/tester/README.md
@@ -0,0 +1,2 @@
+to deploy:
+`juju deploy /path/to/tester.charm --resource workload=python:slim-buster --config tempo_endpoint:foo.bar:4318`
\ No newline at end of file
diff --git a/tests/integration/tester/charmcraft.yaml b/tests/integration/tester/charmcraft.yaml
new file mode 100644
index 0000000..234879e
--- /dev/null
+++ b/tests/integration/tester/charmcraft.yaml
@@ -0,0 +1,17 @@
+# Learn more about charmcraft.yaml configuration at:
+# https://juju.is/docs/sdk/charmcraft-config
+type: "charm"
+bases:
+  - build-on:
+    - name: "ubuntu"
+      channel: "22.04"
+    run-on:
+    - name: "ubuntu"
+      channel: "22.04"
+parts:
+  charm:
+    charm-binary-python-packages:
+      - "pydantic>=2"
+      - "cryptography"
+      - "jsonschema"
+      - "opentelemetry-exporter-otlp-proto-http==1.21.0"
\ No newline at end of file
diff --git a/tests/integration/tester/config.yaml b/tests/integration/tester/config.yaml
new file mode 100644
index 0000000..0547444
--- /dev/null
+++ b/tests/integration/tester/config.yaml
@@ -0,0 +1,14 @@
+# Copyright 2022 pietro
+# See LICENSE file for licensing details.
+
+options:
+  port:
+    description: port at which the webserver should run
+    type: int
+    default: 3000
+
+  host:
+    description: host at which the webserver should run
+    type: string
+    default: 0.0.0.0
+
diff --git a/tests/integration/tester/lib/.gitignore b/tests/integration/tester/lib/.gitignore
new file mode 100644
index 0000000..8e5bbf0
--- /dev/null
+++ b/tests/integration/tester/lib/.gitignore
@@ -0,0 +1 @@
+*.py
\ No newline at end of file
diff --git a/tests/integration/tester/lib/charms/observability_libs/v0/.gitkeep b/tests/integration/tester/lib/charms/observability_libs/v0/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester/lib/charms/prometheus_k8s/v0/.gitkeep b/tests/integration/tester/lib/charms/prometheus_k8s/v0/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester/lib/charms/tempo_k8s/v0/.gitkeep b/tests/integration/tester/lib/charms/tempo_k8s/v0/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester/lib/charms/tempo_k8s/v1/.gitkeep b/tests/integration/tester/lib/charms/tempo_k8s/v1/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester/lib/charms/tempo_k8s/v2/.gitkeep b/tests/integration/tester/lib/charms/tempo_k8s/v2/.gitkeep
new file mode 100644
index 0000000..e69de29
diff --git a/tests/integration/tester/metadata.yaml b/tests/integration/tester/metadata.yaml
new file mode 100644
index 0000000..438197a
--- /dev/null
+++ b/tests/integration/tester/metadata.yaml
@@ -0,0 +1,27 @@
+# Copyright 2022 pietro
+# See LICENSE file for licensing details.
+
+name: tester
+display-name: tester
+description: tempo tester charm
+summary: tempo tester charm
+
+containers:
+  workload:
+    resource: workload
+
+resources:
+  workload:
+    type: oci-image
+    description: OCI image for the workload
+    # Included for simplicity in integration tests
+    upstream-source: python:slim-buster
+
+requires:
+  tracing:
+    interface: tracing
+    limit: 1
+
+peers:
+  replicas:
+    interface: replicas
\ No newline at end of file
diff --git a/tests/integration/tester/requirements.txt b/tests/integration/tester/requirements.txt
new file mode 100644
index 0000000..18c7453
--- /dev/null
+++ b/tests/integration/tester/requirements.txt
@@ -0,0 +1 @@
+ops >= 1.4.0
\ No newline at end of file
diff --git a/tests/integration/tester/src/charm.py b/tests/integration/tester/src/charm.py
new file mode 100755
index 0000000..72feedd
--- /dev/null
+++ b/tests/integration/tester/src/charm.py
@@ -0,0 +1,259 @@
+#!/usr/bin/env python3
+# Copyright 2022 pietro
+# See LICENSE file for licensing details.
+
+import logging
+from pathlib import Path
+from typing import List, Optional
+
+from charms.tempo_k8s.v1.charm_tracing import trace_charm
+from charms.tempo_k8s.v2.tracing import TracingEndpointRequirer, charm_tracing_config
+from ops.charm import CharmBase, PebbleReadyEvent
+from ops.main import main
+from ops.model import (
+    ActiveStatus,
+    Container,
+    MaintenanceStatus,
+    Relation,
+    WaitingStatus,
+)
+from ops.pebble import Layer
+
+logger = logging.getLogger(__name__)
+TRACING_APP_NAME = "TempoTesterCharm"
+
+
+@trace_charm(tracing_endpoint="tempo_otlp_http_endpoint", service_name=TRACING_APP_NAME)
+class TempoTesterCharm(CharmBase):
+    """Charm the service."""
+
+    _layer_name = _service_name = "tester"
+    _container_name = "workload"
+    _peer_relation_name = "replicas"
+    _address_name = "private-address-ip"
+
+    def __init__(self, *args):
+        super().__init__(*args)
+
+        self.container: Container = self.unit.get_container(self._container_name)
+
+        self.tracing = TracingEndpointRequirer(
+            self, relation_name="tracing", protocols=["otlp_http", "otlp_grpc"]
+        )
+        self.tempo_otlp_http_endpoint = charm_tracing_config(self.tracing, None)
+        # Core lifecycle events
+        self.framework.observe(self.on.config_changed, self._update)
+
+        # Peer relation events
+        self.framework.observe(self.on[self._peer_relation_name].relation_joined, self._update)
+        self.framework.observe(self.on[self._peer_relation_name].relation_changed, self._update)
+        self.framework.observe(self.tracing.on.endpoint_changed, self._update)
+
+    @property
+    def peers(self) -> List[str]:
+        return self._get_peer_addresses()
+
+    def _on_tester_pebble_ready(self, e: PebbleReadyEvent):
+        if e.workload.can_connect():
+            self._setup_container(e.workload)
+        else:
+            self.unit.status = WaitingStatus("waiting for container connectivity...")
+            return e.defer()
+        self.unit.status = ActiveStatus("ready")
+
+    def ensure_container_setup(self):
+        container = self.container
+        if container.list_files("/", pattern="webserver.py"):
+            return
+        self._setup_container(container)
+
+    def _setup_container(self, container: Container):
+        # copy the webserver file to the container. In a production environment,
+        # the workload would typically be an OCI image. Here however we have a
+        # 'bare' python container as base.
+        self.unit.status = MaintenanceStatus("copying over webserver source")
+
+        resources = Path(__file__).parent / "resources"
+        webserver_source_path = resources / "webserver.py"
+        logger.info(webserver_source_path)
+        with open(webserver_source_path, "r") as webserver_source:
+            logger.info("pushing webserver source...")
+            container.push("/webserver.py", webserver_source)
+
+        self.unit.status = MaintenanceStatus("installing software in workload container")
+        # we install the webserver dependencies; in a production environment, these
+        # would typically be baked in the workload OCI image.
+        webserver_dependencies_path = resources / "webserver-dependencies.txt"
+        logger.info(webserver_dependencies_path)
+        with open(webserver_dependencies_path, "r") as dependencies_file:
+            dependencies = dependencies_file.read().split("\n")
+            logger.info(f"installing webserver dependencies {dependencies}...")
+            container.exec(["pip", "install", *dependencies]).wait()
+
+        self.unit.status = MaintenanceStatus("container ready")
+
+    # Actual tester stuff
+    def _tester_layer(self):
+        """Returns a Pebble configuration layer for the tester runtime."""
+        env = {
+            "PORT": self.config["port"],
+            "HOST": self.config["host"],
+            "APP_NAME": self.app.name,
+            "TEMPO_ENDPOINT": str(self.tempo_otlp_http_endpoint or ""),
+        }
+        logging.info(f"Initing pebble layer with env: {str(env)}")
+
+        if self.unit.name.split("/")[1] == "0":
+            env["PEERS"] = peers = ";".join(self.peers)
+            env["OVERLORD"] = "1"
+            logging.info(f"Configuring overlord node; Peers: {peers}")
+
+        return Layer(
+            {
+                "summary": "tester layer",
+                "description": "pebble config layer for tester",
+                "services": {
+                    "tester": {
+                        "override": "merge",
+                        "summary": "tester service",
+                        "command": "python webserver.py > webserver.log",
+                        "startup": "enabled",
+                        "environment": env,
+                    }
+                },
+            }
+        )
+
+    # source: https://github.com/canonical/alertmanager-k8s-operator
+    def _restart_service(self) -> bool:
+        """Helper function for restarting the underlying service.
+        Returns:
+            True if restart succeeded; False otherwise.
+        """
+        logger.info("Restarting service %s", self._service_name)
+
+        if not self.container.can_connect():
+            logger.error("Cannot (re)start service: container is not ready.")
+            return False
+
+        # Check if service exists, to avoid ModelError from being raised when the service does
+        # not exist,
+        if not self.container.get_plan().services.get(self._service_name):
+            logger.error("Cannot (re)start service: service does not (yet) exist.")
+            return False
+
+        logger.info(f"pebble env, {self.container.get_plan().services.get('tester').environment}")
+
+        self.container.restart(self._service_name)
+        logger.info(f"restarted {self._service_name}")
+        return True
+
+    def _update_layer(self, restart: bool) -> bool:
+        """Update service layer to reflect changes in peers (replicas).
+        Args:
+          restart: a flag indicating if the service should be restarted if a change was detected.
+        Returns:
+          True if anything changed; False otherwise
+        """
+        overlay = self._tester_layer()
+        plan = self.container.get_plan()
+
+        logger.info("updating layer")
+
+        if self._service_name not in plan.services or overlay.services != plan.services:
+            logger.info("container.add_layer")
+            self.container.add_layer(self._layer_name, overlay, combine=True)
+
+            service_exists = self.container.get_plan().services.get("tester", None) is not None
+            if service_exists and restart:
+                self._restart_service()
+
+            return True
+
+        return False
+
+    @property
+    def peer_relation(self) -> Relation:
+        """Helper function for obtaining the peer relation object.
+        Returns: peer relation object
+        (NOTE: would return None if called too early, e.g. during install).
+        """
+        return self.model.get_relation(self._peer_relation_name)
+
+    @property
+    def private_address(self) -> Optional[str]:
+        """Get the unit's ip address.
+        Technically, receiving a "joined" event guarantees an IP address is available. If this is
+        called beforehand, a None would be returned.
+        When operating a single unit, no "joined" events are visible so obtaining an address is a
+        matter of timing in that case.
+        This function is still needed in Juju 2.9.5 because the "private-address" field in the
+        data bag is being populated by the app IP instead of the unit IP.
+        Also in Juju 2.9.5, ip address may be None even after RelationJoinedEvent, for which
+        "ops.model.RelationDataError: relation data values must be strings" would be emitted.
+        Returns:
+          None if no IP is available (called before unit "joined"); unit's ip address otherwise
+        """
+
+        # if bind_address := check_output(["unit-get", "private-address"]).decode().strip()
+        if bind_address := self.model.get_binding(self._peer_relation_name).network.bind_address:
+            bind_address = str(bind_address)
+        return bind_address
+
+    def update_address_in_relation_data(self, relation):
+        """stores this unit's private IP in the relation databag"""
+        relation.data[self.unit].update({self._address_name: self.private_address})
+        logger.info(f"stored {self.private_address} in relation databag")
+
+    def _update(self, _):
+        """Event handler for all things."""
+        self.ensure_container_setup()
+
+        logger.info("running _update")
+        if not self.container.can_connect():
+            self.unit.status = WaitingStatus("waiting for container connectivity...")
+            return
+
+        # Wait for IP address. IP address is needed for forming tester clusters
+        # and for related apps' config.
+        if not self.private_address:
+            self.unit.status = MaintenanceStatus("waiting for IP address...")
+            return
+
+        if not (self.tracing.is_ready()):
+            self.unit.status = WaitingStatus("waiting for tracing to be ready...")
+            return
+
+        # In the case of a single unit deployment, no 'RelationJoined' event is emitted, so
+        # setting IP here.
+        # Store private address in unit's peer relation data bucket. This is still needed because
+        # the "private-address" field in the data bag is being populated incorrectly.
+        # Also, ip address may still be None even after RelationJoinedEvent, for which
+        # "ops.model.RelationDataError: relation data values must be strings" would be emitted.
+        if peer_relation := self.peer_relation:
+            self.update_address_in_relation_data(peer_relation)
+        else:
+            logger.info("no peer relation to configure")
+
+        self._update_layer(restart=True)
+        self.unit.status = ActiveStatus("ready")
+
+    def _get_peer_addresses(self) -> List[str]:
+        """Create a list of addresses of all peer units (all units excluding current).
+        The returned addresses include the port number but do not include scheme (http).
+        If a unit does not have an address, it will be omitted from the list.
+        """
+        addresses = []
+        if pr := self.peer_relation:
+            addresses = [
+                f"{address}"
+                for unit in pr.units
+                # pr.units only holds peers (self.unit is not included)
+                if (address := pr.data[unit].get(self._address_name))
+            ]
+
+        return addresses
+
+
+if __name__ == "__main__":
+    main(TempoTesterCharm)
diff --git a/tests/integration/tester/src/resources/webserver-dependencies.txt b/tests/integration/tester/src/resources/webserver-dependencies.txt
new file mode 100644
index 0000000..6834282
--- /dev/null
+++ b/tests/integration/tester/src/resources/webserver-dependencies.txt
@@ -0,0 +1,7 @@
+requests
+opentelemetry-exporter-otlp
+opentelemetry-instrumentation-fastapi
+opentelemetry-instrumentation-logging
+fastapi
+uvicorn
+prometheus_client
\ No newline at end of file
diff --git a/tests/integration/tester/src/resources/webserver.py b/tests/integration/tester/src/resources/webserver.py
new file mode 100644
index 0000000..9bb42b2
--- /dev/null
+++ b/tests/integration/tester/src/resources/webserver.py
@@ -0,0 +1,251 @@
+# Copyright 2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+import logging
+import os
+import random
+import time
+from typing import Tuple
+
+import requests
+import uvicorn as uvicorn
+from fastapi import FastAPI
+from fastapi.openapi.models import Response
+from opentelemetry import trace
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
+from opentelemetry.instrumentation.logging import LoggingInstrumentor
+from opentelemetry.propagate import inject
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from prometheus_client import REGISTRY, Counter, Gauge, Histogram
+from prometheus_client.openmetrics.exposition import (
+    CONTENT_TYPE_LATEST,
+    generate_latest,
+)
+from starlette.middleware.base import BaseHTTPMiddleware, RequestResponseEndpoint
+from starlette.requests import Request
+from starlette.routing import Match
+from starlette.status import HTTP_500_INTERNAL_SERVER_ERROR
+from starlette.types import ASGIApp
+
+REQUESTS_PROCESSING_TIME = Histogram(
+    "fastapi_requests_duration_seconds",
+    "Histogram of requests processing time by path (in seconds)",
+    ["method", "path", "app_name"],
+)
+
+APP_NAME = os.environ.get("APP_NAME")
+TEMPO_ENDPOINT = os.environ.get("TEMPO_ENDPOINT", "")
+HOST = os.environ.get("HOST", "0.0.0.0")
+PORT = int(os.environ.get("PORT", 3000))
+
+
+if not TEMPO_ENDPOINT:
+    logging.warning("TEMPO_ENDPOINT not configured; tracing disabled. webserver dead.")
+
+
+INFO = Gauge("fastapi_app_info", "application information.", ["app_name"])
+REQUESTS = Counter(
+    f"{APP_NAME}_requests_total",
+    "Total count of requests by method and path.",
+    ["method", "path", "app_name"],
+)
+RESPONSES = Counter(
+    f"{APP_NAME}_responses_total",
+    "Total count of responses by method, path and status codes.",
+    ["method", "path", "status_code", "app_name"],
+)
+REQUESTS_PROCESSING_TIME = Histogram(
+    f"{APP_NAME}_requests_duration_seconds",
+    "Histogram of requests processing time by path (in seconds)",
+    ["method", "path", "app_name"],
+)
+EXCEPTIONS = Counter(
+    f"{APP_NAME}_exceptions_total",
+    "Total count of exceptions raised by path and exception type",
+    ["method", "path", "exception_type", "app_name"],
+)
+REQUESTS_IN_PROGRESS = Gauge(
+    f"{APP_NAME}_requests_in_progress",
+    "Gauge of requests by method and path currently being processed",
+    ["method", "path", "app_name"],
+)
+
+
+class PrometheusMiddleware(BaseHTTPMiddleware):
+    def __init__(self, app: ASGIApp, app_name: str = APP_NAME) -> None:
+        super().__init__(app)
+        self.app_name = app_name
+        INFO.labels(app_name=self.app_name).inc()
+
+    async def dispatch(self, request: Request, call_next: RequestResponseEndpoint) -> Response:
+        method = request.method
+        path, is_handled_path = self.get_path(request)
+
+        if not is_handled_path:
+            return await call_next(request)
+
+        REQUESTS_IN_PROGRESS.labels(method=method, path=path, app_name=self.app_name).inc()
+        REQUESTS.labels(method=method, path=path, app_name=self.app_name).inc()
+        before_time = time.perf_counter()
+        try:
+            response = await call_next(request)
+        except BaseException as e:
+            status_code = HTTP_500_INTERNAL_SERVER_ERROR
+            EXCEPTIONS.labels(
+                method=method, path=path, exception_type=type(e).__name__, app_name=self.app_name
+            ).inc()
+            raise e from None
+        else:
+            status_code = response.status_code
+            after_time = time.perf_counter()
+            # retrieve trace id for exemplar
+            span = trace.get_current_span()
+            trace_id = trace.format_trace_id(span.get_span_context().trace_id)
+
+            REQUESTS_PROCESSING_TIME.labels(
+                method=method, path=path, app_name=self.app_name
+            ).observe(after_time - before_time, exemplar={"TraceID": trace_id})
+        finally:
+            RESPONSES.labels(
+                method=method, path=path, status_code=status_code, app_name=self.app_name
+            ).inc()
+            REQUESTS_IN_PROGRESS.labels(method=method, path=path, app_name=self.app_name).dec()
+
+        return response
+
+    @staticmethod
+    def get_path(request: Request) -> Tuple[str, bool]:
+        for route in request.app.routes:
+            match, child_scope = route.matches(request.scope)
+            if match == Match.FULL:
+                return route.path, True
+
+        return request.url.path, False
+
+
+def init_overlord():
+    logging.debug("Initing as overlord.")
+    # manual instrumentation for overlord node
+    resource = Resource.create(attributes={"service.name": APP_NAME, "compose_service": APP_NAME})
+
+    provider = TracerProvider(resource=resource)
+    processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=TEMPO_ENDPOINT))
+    provider.add_span_processor(processor)
+    trace.set_tracer_provider(provider)
+    tracer = trace.get_tracer(__name__)
+    subordinates = os.environ.get("PEERS").split(";")
+
+    print(f"preparing to send traces to {subordinates}")
+
+    while True:
+        with tracer.start_as_current_span(
+            "query_peers", attributes={"service.name": APP_NAME}
+        ) as span:
+            for i, subordinate in enumerate(subordinates):
+                url = "http://" + subordinate + f":{PORT}/query/{i}"
+
+                print(f"sending {i} to {subordinate}")
+                span.add_event(
+                    "log",
+                    {
+                        "service.name": APP_NAME,
+                        "subordinate.id": i,
+                        "subordinate.root_url": subordinate,
+                        "subordinate.query_url": subordinate,
+                    },
+                )
+
+                with tracer.start_as_current_span(
+                    f"query_peer_{i}", attributes={"service.name": APP_NAME}
+                ) as child_span:
+                    headers = {}
+                    inject(headers)  # inject trace id into request headers.
+
+                    resp = requests.get(url, headers=headers)
+                    print(f"received {resp} from {subordinate}")
+                    try:
+                        id_, value_ = resp.text.split("=")
+                    except:  # noqa
+                        id_ = value_ = None
+                        print(f"error handling response from {url}: invalid text: {resp.text}")
+
+                    child_span.add_event(
+                        "log",
+                        {
+                            "service.name": APP_NAME,
+                            "subordinate.response.id": id_,
+                            "subordinate.response.value": value_,
+                        },
+                    )
+        provider.force_flush()
+
+
+def init_subordinate():
+    logging.debug("Initing as subordinate.")
+    # auto instrumentation for subordinate nodes
+    app = FastAPI()
+    app.add_middleware(PrometheusMiddleware, app_name=APP_NAME)
+
+    @app.get("/metrics")
+    def metrics():
+        return Response(generate_latest(REGISTRY), headers={"Content-Type": CONTENT_TYPE_LATEST})
+
+    @app.get("/query/{var}")
+    async def query(var: str):
+        randn = random.random()
+        time.sleep(randn)
+
+        # add request processing time metric with exemplar
+        span = trace.get_current_span()
+        trace_id = trace.format_trace_id(span.get_span_context().trace_id)
+        REQUESTS_PROCESSING_TIME.labels(
+            method="GET", path="/query/{var}", app_name=APP_NAME
+        ).observe(randn, exemplar={"TraceID": trace_id})
+        return f"{var}={randn}"
+
+    def setup_otlp(
+        app: ASGIApp, app_name: str, endpoint: str, log_correlation: bool = True
+    ) -> None:
+        # Setting OpenTelemetry
+        # set the service name to show in traces
+        resource = Resource.create(
+            attributes={"service.name": app_name, "compose_service": app_name}
+        )
+
+        # set the tracer provider
+        tracer = TracerProvider(resource=resource)
+        trace.set_tracer_provider(tracer)
+
+        tracer.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint)))
+
+        if log_correlation:
+            LoggingInstrumentor().instrument(set_logging_format=True)
+
+        FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer)
+
+    setup_otlp(app, APP_NAME, TEMPO_ENDPOINT)
+    uvicorn.run(app, host=HOST, port=PORT)
+
+
+class EndpointFilter(logging.Filter):
+    # Uvicorn endpoint access log filter
+    def filter(self, record: logging.LogRecord) -> bool:
+        return record.getMessage().find("GET /metrics") == -1
+
+
+# Filter out /endpoint
+logging.getLogger("uvicorn.access").addFilter(EndpointFilter())
+
+
+if __name__ == "__main__":
+    if not TEMPO_ENDPOINT:
+        while True:
+            logging.warning("IDLING. TEMPO_ENDPOINT unavailable.")
+            time.sleep(10)
+
+    if os.environ.get("OVERLORD"):
+        init_overlord()
+    else:
+        init_subordinate()
diff --git a/tests/interface/conftest.py b/tests/interface/conftest.py
new file mode 100644
index 0000000..4941189
--- /dev/null
+++ b/tests/interface/conftest.py
@@ -0,0 +1,52 @@
+# Copyright 2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+from unittest.mock import patch
+
+import pytest
+from charms.tempo_k8s.v1.charm_tracing import charm_tracing_disabled
+from interface_tester import InterfaceTester
+from ops.pebble import Layer
+from scenario.state import Container, State
+
+from charm import TempoCharm
+
+
+# Interface tests are centrally hosted at https://github.com/canonical/charm-relation-interfaces.
+# this fixture is used by the test runner of charm-relation-interfaces to test tempo's compliance
+# with the interface specifications.
+# DO NOT MOVE OR RENAME THIS FIXTURE! If you need to, you'll need to open a PR on
+# https://github.com/canonical/charm-relation-interfaces and change tempo's test configuration
+# to include the new identifier/location.
+@pytest.fixture
+def interface_tester(interface_tester: InterfaceTester):
+    with patch("charm.KubernetesServicePatch"):
+        with charm_tracing_disabled():
+            interface_tester.configure(
+                charm_type=TempoCharm,
+                state_template=State(
+                    leader=True,
+                    containers=[
+                        Container(
+                            name="tempo",
+                            can_connect=True,
+                            layers={
+                                "foo": Layer(
+                                    {
+                                        "summary": "foo",
+                                        "description": "bar",
+                                        "services": {
+                                            "tempo": {
+                                                "startup": "enabled",
+                                                "current": "active",
+                                                "name": "tempo",
+                                            }
+                                        },
+                                        "checks": {},
+                                    }
+                                )
+                            },
+                        )
+                    ],
+                ),
+            )
+            yield interface_tester
diff --git a/tests/interface/test_tracing.py b/tests/interface/test_tracing.py
new file mode 100644
index 0000000..806664a
--- /dev/null
+++ b/tests/interface/test_tracing.py
@@ -0,0 +1,19 @@
+# Copyright 2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+from interface_tester import InterfaceTester
+
+
+def test_tracing_v0_interface(interface_tester: InterfaceTester):
+    interface_tester.configure(
+        interface_name="tracing",
+        interface_version=0,
+    )
+    interface_tester.run()
+
+
+def test_tracing_v2_interface(interface_tester: InterfaceTester):
+    interface_tester.configure(
+        interface_name="tracing",
+        interface_version=2,
+    )
+    interface_tester.run()
diff --git a/tests/manual/facades/README.md b/tests/manual/facades/README.md
new file mode 100644
index 0000000..439051e
--- /dev/null
+++ b/tests/manual/facades/README.md
@@ -0,0 +1,10 @@
+This directory contains some files to be used in conjunction with the facade charm.
+
+For example, to simulate s3:
+
+```
+juju deploy facade s3-facade
+juju relate s3-facade:provide-s3 tempo:s3
+juju run s3-facade/0 update --params ./tests/manual/facades/s3.yaml
+```
+
diff --git a/tests/manual/facades/s3.yaml b/tests/manual/facades/s3.yaml
new file mode 100644
index 0000000..a68a580
--- /dev/null
+++ b/tests/manual/facades/s3.yaml
@@ -0,0 +1,8 @@
+endpoint: provide-s3
+app_data: |
+  {
+    "access-key": "mykey",
+    "bucket": "BuckyMcBucket",
+    "endpoint": "http://1.2.3.4:9000",
+    "secret-key": "soverysecret"
+  }
\ No newline at end of file
diff --git a/tests/scenario/conftest.py b/tests/scenario/conftest.py
new file mode 100644
index 0000000..6591149
--- /dev/null
+++ b/tests/scenario/conftest.py
@@ -0,0 +1,19 @@
+from unittest.mock import patch
+
+import pytest
+from scenario import Context
+
+from charm import TempoCharm
+
+
+@pytest.fixture
+def tempo_charm():
+    with patch("charm.KubernetesServicePatch"):
+        with patch("lightkube.core.client.GenericSyncClient"):
+            with patch("charm.TempoCharm._update_server_cert"):
+                yield TempoCharm
+
+
+@pytest.fixture(scope="function")
+def context(tempo_charm):
+    return Context(charm_type=tempo_charm)
diff --git a/tests/scenario/helpers.py b/tests/scenario/helpers.py
new file mode 100644
index 0000000..644911d
--- /dev/null
+++ b/tests/scenario/helpers.py
@@ -0,0 +1,12 @@
+from pathlib import Path
+
+import scenario
+import yaml
+
+from tempo import Tempo
+
+
+def get_tempo_config(container: scenario.Container, context: scenario.Context):
+    fs = container.get_filesystem(context)
+    cfg_path = Path(str(fs) + Tempo.config_path)
+    return yaml.safe_load(cfg_path.read_text())
diff --git a/tests/scenario/test_charm.py b/tests/scenario/test_charm.py
new file mode 100644
index 0000000..b213043
--- /dev/null
+++ b/tests/scenario/test_charm.py
@@ -0,0 +1,181 @@
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import pytest
+import yaml
+from charms.tempo_k8s.v1.charm_tracing import charm_tracing_disabled
+from charms.tempo_k8s.v2.tracing import TracingRequirerAppData
+from ops import pebble
+from scenario import Container, Mount, Relation, State
+from scenario.sequences import check_builtin_sequences
+
+from tempo import Tempo
+from tests.scenario.helpers import get_tempo_config
+
+TEMPO_CHARM_ROOT = Path(__file__).parent.parent.parent
+
+
+@pytest.fixture(params=(True, False))
+def base_state(request):
+    return State(
+        leader=request.param,
+        containers=[Container("tempo", can_connect=True)],
+    )
+
+
+def test_builtin_sequences(tempo_charm, base_state):
+    with charm_tracing_disabled():
+        check_builtin_sequences(tempo_charm, template_state=base_state)
+
+
+def test_start(context, base_state):
+    # verify the charm runs at all with and without leadership
+    with charm_tracing_disabled():
+        context.run("start", base_state)
+
+
+@pytest.mark.parametrize("requested_protocol", ("otlp_grpc", "zipkin"))
+def test_tempo_restart_on_ingress_v2_changed(context, tmp_path, requested_protocol):
+    # GIVEN
+    # an initial configuration with an otlp_http receiver
+    container, tempo = _tempo_mock_with_initial_config(tmp_path)
+
+    # the remote end requests an otlp_grpc endpoint
+    ingress = Relation(
+        "tracing",
+        remote_app_data=TracingRequirerAppData(receivers=[requested_protocol]).dump(),
+    )
+
+    # WHEN
+    # the charm receives an ingress(v2) relation-changed requesting an otlp_grpc receiver
+    state = State(leader=True, containers=[tempo], relations=[ingress])
+    context.run(ingress.changed_event, state)
+
+    # THEN
+    # Tempo pushes a new config to the container filesystem
+    new_config = get_tempo_config(tempo, context)
+    expected_config = Tempo(container).generate_config(
+        ["otlp_http", requested_protocol],
+    )
+    assert new_config == expected_config
+    # AND restarts the pebble service.
+    assert (
+        context.output_state.get_container("tempo").service_status["tempo"]
+        is pebble.ServiceStatus.ACTIVE
+    )
+
+
+def _tempo_mock_with_initial_config(tmp_path):
+    tempo_config = tmp_path / "tempo.yaml"
+    container = MagicMock()
+    container.can_connect = lambda: True
+    # prevent tls_ready from reporting True
+    container.exists = lambda path: (
+        False if path in [Tempo.tls_cert_path, Tempo.tls_key_path, Tempo.tls_ca_path] else True
+    )
+    initial_config = Tempo(container).generate_config(["otlp_http"])
+    tempo_config.write_text(yaml.safe_dump(initial_config))
+    tempo = Container(
+        "tempo",
+        can_connect=True,
+        layers={
+            "tempo": pebble.Layer(
+                {
+                    "summary": "tempo layer",
+                    "description": "foo",
+                    "services": {
+                        "tempo": {"startup": "enabled"},
+                        "tempo-ready": {"startup": "disabled"},
+                    },
+                },
+            ),
+        },
+        service_status={
+            # we don't have a way to check if the service has been restarted: all that scenario does ATM is set it to
+            # 'active'.
+            # so as a way to check that it's been restarted, we must set it to inactive here.
+            "tempo": pebble.ServiceStatus.INACTIVE,
+        },
+        mounts={
+            "data": Mount("/etc/tempo/tempo.yaml", tempo_config),
+        },
+    )
+    return container, tempo
+
+
+def test_tempo_tracing_created_before_pebble_ready(context, tmp_path):
+    # GIVEN there is no plan yet
+    tempo = Container(
+        "tempo",
+        can_connect=True,
+    )
+
+    # WHEN
+    # the charm receives a tracing-relation-created requesting an otlp_grpc receiver
+    tracing = Relation(
+        "tracing",
+        remote_app_data={"receivers": '["otlp_http"]'},
+        local_app_data={
+            "receivers": '[{"protocol": {"name": "otlp_grpc", "type": "grpc"} , "url": "foo.com:10"}, '
+            '{"protocol": {"name": "otlp_http", "type": "http"}, "url": "http://foo.com:11"}, ',
+        },
+    )
+    state = State(leader=True, containers=[tempo], relations=[tracing])
+    state_out = context.run(tracing.created_event, state)
+
+    # THEN
+    # tempo still has no services
+    tempo_out = state_out.get_container("tempo")
+    assert not tempo_out.services
+
+
+def test_tracing_storage_is_configured_to_local_without_relation(context, tmp_path):
+    # GIVEN tempo mock
+    container, tempo = _tempo_mock_with_initial_config(tmp_path)
+
+    # WHEN any event comes in
+    state = State(leader=True, containers=[tempo], relations=[])
+    context.run("update-status", state)
+
+    # THEN tempo's config has a local storage configured
+    config = get_tempo_config(tempo, context)
+    expected_config = Tempo(container).generate_config(["otlp_http"])
+    assert config == expected_config
+    assert config["storage"]["trace"]["backend"] == "local"
+
+
+@pytest.mark.parametrize(
+    "relation_data",
+    (
+        {},
+        {
+            "access-key": "key",
+            "bucket": "tempo",
+            "endpoint": "http://1.2.3.4:9000",
+            "secret-key": "soverysecret",
+        },
+    ),
+)
+def test_tracing_storage_is_configured_to_s3_if_s3_relation_filled(
+    context,
+    tmp_path,
+    relation_data,
+):
+    # GIVEN tempo mock
+    container, tempo = _tempo_mock_with_initial_config(tmp_path)
+
+    # WHEN a charm receives an s3 relation
+    s3_relation = Relation(
+        "s3",
+        remote_app_data=relation_data,
+        local_app_data={"bucket": "tempo"},
+    )
+
+    state = State(leader=True, containers=[tempo], relations=[s3_relation])
+    context.run(s3_relation.changed_event, state)
+
+    # THEN
+    # Tempo's config contains the data from the relation
+    new_config = get_tempo_config(tempo, context)
+    expected_config = Tempo(container).generate_config(["otlp_http"], relation_data)
+    assert new_config == expected_config
diff --git a/tests/scenario/test_charm_scaling.py b/tests/scenario/test_charm_scaling.py
new file mode 100644
index 0000000..3c2284e
--- /dev/null
+++ b/tests/scenario/test_charm_scaling.py
@@ -0,0 +1,80 @@
+from unittest.mock import patch
+
+import ops
+import pytest
+from scenario import Container, PeerRelation, Relation, State
+
+
+def test_monolithic_status_no_s3(context):
+    state_out = context.run(
+        "start", State(unit_status=ops.ActiveStatus(), containers=[Container("tempo")])
+    )
+    assert state_out.unit_status.name == "waiting"
+
+
+def test_scaled_status_no_s3(context):
+    state_out = context.run(
+        "start",
+        State(
+            relations=[PeerRelation("tempo-peers", peers_data={1: {}, 2: {}})],
+            unit_status=ops.ActiveStatus(),
+            containers=[Container("tempo")],
+        ),
+    )
+    assert state_out.unit_status.name == "blocked"
+
+
+def test_scaled_status_with_s3(context):
+    state_out = context.run(
+        "start",
+        State(
+            relations=[
+                PeerRelation("tempo-peers", peers_data={1: {}, 2: {}}),
+                Relation(
+                    "s3",
+                    remote_app_data={
+                        "access-key": "key",
+                        "bucket": "tempo",
+                        "endpoint": "http://1.2.3.4:9000",
+                        "secret-key": "soverysecret",
+                    },
+                    local_unit_data={"bucket": "tempo"},
+                ),
+            ],
+            unit_status=ops.ActiveStatus(),
+            containers=[Container("tempo")],
+        ),
+    )
+    assert state_out.unit_status.name == "waiting"
+
+
+@pytest.mark.parametrize("leader", (True, False))
+@patch("tempo.Tempo.is_ready", new=lambda _: True)
+def test_scaled_status_with_s3_and_container_ready(context, leader):
+    container = Container(
+        name="tempo",
+        can_connect=True,
+    )
+    state_out = context.run(
+        container.pebble_ready_event,
+        State(
+            leader=leader,
+            relations=[
+                PeerRelation("tempo-peers", peers_data={1: {}, 2: {}}),
+                Relation(
+                    "s3",
+                    remote_app_data={
+                        "access-key": "key",
+                        "bucket": "tempo",
+                        "endpoint": "http://1.2.3.4:9000",
+                        "secret-key": "soverysecret",
+                    },
+                    local_unit_data={"bucket": "tempo"},
+                ),
+            ],
+            unit_status=ops.ActiveStatus(),
+            containers=[container],
+        ),
+    )
+    assert state_out.unit_status.name == "active"
+    assert state_out.get_container("tempo").services["tempo"].is_running()
diff --git a/tests/scenario/test_charm_tracing.py b/tests/scenario/test_charm_tracing.py
new file mode 100644
index 0000000..e67bba5
--- /dev/null
+++ b/tests/scenario/test_charm_tracing.py
@@ -0,0 +1,558 @@
+import logging
+import os
+from unittest.mock import patch
+
+import pytest
+import scenario
+from charms.tempo_k8s.v1.charm_tracing import CHARM_TRACING_ENABLED
+from charms.tempo_k8s.v1.charm_tracing import _autoinstrument as autoinstrument
+from charms.tempo_k8s.v2.tracing import (
+    ProtocolNotRequestedError,
+    ProtocolType,
+    Receiver,
+    TracingEndpointRequirer,
+    TracingProviderAppData,
+    TracingRequirerAppData,
+)
+from ops import EventBase, EventSource, Framework
+from ops.charm import CharmBase, CharmEvents
+from scenario import Context, State
+
+from lib.charms.tempo_k8s.v1.charm_tracing import get_current_span, trace
+
+os.environ[CHARM_TRACING_ENABLED] = "1"
+
+logger = logging.getLogger(__name__)
+
+
+@pytest.fixture(autouse=True)
+def cleanup():
+    # if any other test module disabled it...
+    os.environ[CHARM_TRACING_ENABLED] = "1"
+
+    def patched_set_tracer_provider(tracer_provider, log):
+        import opentelemetry
+
+        opentelemetry.trace._TRACER_PROVIDER = tracer_provider
+
+    with patch("opentelemetry.trace._set_tracer_provider", new=patched_set_tracer_provider):
+        yield
+
+
+class MyCharmSimple(CharmBase):
+    META = {"name": "frank"}
+
+    @property
+    def tempo(self):
+        return "foo.bar:80"
+
+
+autoinstrument(MyCharmSimple, MyCharmSimple.tempo)
+
+
+def test_base_tracer_endpoint(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmSimple, meta=MyCharmSimple.META)
+        ctx.run("start", State())
+        assert "Setting up span exporter to endpoint: foo.bar:80" in caplog.text
+        assert "Starting root trace with id=" in caplog.text
+        span = f.call_args_list[0].args[0][0]
+        assert span.resource.attributes["service.name"] == "frank"
+        assert span.resource.attributes["compose_service"] == "frank"
+        assert span.resource.attributes["charm_type"] == "MyCharmSimple"
+
+
+class SubObject:
+    def foo(self):
+        return "bar"
+
+
+class MyCharmSubObject(CharmBase):
+    META = {"name": "frank"}
+
+    def __init__(self, framework: Framework):
+        super().__init__(framework)
+        self.subobj = SubObject()
+        framework.observe(self.on.start, self._on_start)
+
+    def _on_start(self, _):
+        self.subobj.foo()
+
+    @property
+    def tempo(self):
+        return "foo.bar:80"
+
+
+autoinstrument(MyCharmSubObject, MyCharmSubObject.tempo, extra_types=[SubObject])
+
+
+def test_subobj_tracer_endpoint(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmSubObject, meta=MyCharmSubObject.META)
+        ctx.run("start", State())
+        spans = f.call_args_list[0].args[0]
+        assert spans[0].name == "method call: SubObject.foo"
+
+
+class MyCharmInitAttr(CharmBase):
+    META = {"name": "frank"}
+
+    def __init__(self, framework: Framework):
+        super().__init__(framework)
+        self._tempo = "foo.bar:80"
+
+    @property
+    def tempo(self):
+        return self._tempo
+
+
+autoinstrument(MyCharmInitAttr, MyCharmInitAttr.tempo)
+
+
+def test_init_attr(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmInitAttr, meta=MyCharmInitAttr.META)
+        ctx.run("start", State())
+        assert "Setting up span exporter to endpoint: foo.bar:80" in caplog.text
+        span = f.call_args_list[0].args[0][0]
+        assert span.resource.attributes["service.name"] == "frank"
+        assert span.resource.attributes["compose_service"] == "frank"
+        assert span.resource.attributes["charm_type"] == "MyCharmInitAttr"
+
+
+class MyCharmSimpleDisabled(CharmBase):
+    META = {"name": "frank"}
+
+    @property
+    def tempo(self):
+        return None
+
+
+autoinstrument(MyCharmSimpleDisabled, MyCharmSimpleDisabled.tempo)
+
+
+def test_base_tracer_endpoint_disabled(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmSimpleDisabled, meta=MyCharmSimpleDisabled.META)
+        ctx.run("start", State())
+
+        assert "quietly disabling charm_tracing for the run." in caplog.text
+        assert not f.called
+
+
+@trace
+def _my_fn(foo):
+    return foo + 1
+
+
+class MyCharmSimpleEvent(CharmBase):
+    META = {"name": "frank"}
+
+    def __init__(self, fw):
+        super().__init__(fw)
+        span = get_current_span()
+        assert span is None  # can't do that in init.
+        fw.observe(self.on.start, self._on_start)
+
+    def _on_start(self, _):
+        span = get_current_span()
+        span.add_event(
+            "log",
+            {
+                "foo": "bar",
+                "baz": "qux",
+            },
+        )
+        _my_fn(2)
+
+    @property
+    def tempo(self):
+        return "foo.bar:80"
+
+
+autoinstrument(MyCharmSimpleEvent, MyCharmSimpleEvent.tempo)
+
+
+def test_base_tracer_endpoint_event(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmSimpleEvent, meta=MyCharmSimpleEvent.META)
+        ctx.run("start", State())
+
+        spans = f.call_args_list[0].args[0]
+        span0, span1, span2, span3 = spans
+        assert span0.name == "function call: _my_fn"
+
+        assert span1.name == "method call: MyCharmSimpleEvent._on_start"
+
+        assert span2.name == "event: start"
+        evt = span2.events[0]
+        assert evt.name == "start"
+
+        assert span3.name == "charm exec"
+
+        for span in spans:
+            assert span.resource.attributes["service.name"] == "frank"
+
+
+def test_juju_topology_injection(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmSimpleEvent, meta=MyCharmSimpleEvent.META)
+        state = ctx.run("start", State())
+
+        spans = f.call_args_list[0].args[0]
+
+        for span in spans:
+            # topology
+            assert span.resource.attributes["juju_unit"] == "frank/0"
+            assert span.resource.attributes["juju_application"] == "frank"
+            assert span.resource.attributes["juju_model"] == state.model.name
+            assert span.resource.attributes["juju_model_uuid"] == state.model.uuid
+
+
+class MyCharmWithMethods(CharmBase):
+    META = {"name": "frank"}
+
+    def __init__(self, fw):
+        super().__init__(fw)
+        fw.observe(self.on.start, self._on_start)
+
+    def _on_start(self, _):
+        self.a()
+        self.b()
+        self.c()
+
+    def a(self):
+        pass
+
+    def b(self):
+        pass
+
+    def c(self):
+        pass
+
+    @property
+    def tempo(self):
+        return "foo.bar:80"
+
+
+autoinstrument(MyCharmWithMethods, MyCharmWithMethods.tempo)
+
+
+def test_base_tracer_endpoint_methods(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmWithMethods, meta=MyCharmWithMethods.META)
+        ctx.run("start", State())
+
+        spans = f.call_args_list[0].args[0]
+        span_names = [span.name for span in spans]
+        assert span_names == [
+            "method call: MyCharmWithMethods.a",
+            "method call: MyCharmWithMethods.b",
+            "method call: MyCharmWithMethods.c",
+            "method call: MyCharmWithMethods._on_start",
+            "event: start",
+            "charm exec",
+        ]
+
+
+class Foo(EventBase):
+    pass
+
+
+class MyEvents(CharmEvents):
+    foo = EventSource(Foo)
+
+
+class MyCharmWithCustomEvents(CharmBase):
+    on = MyEvents()
+
+    META = {"name": "frank"}
+
+    def __init__(self, fw):
+        super().__init__(fw)
+        fw.observe(self.on.start, self._on_start)
+        fw.observe(self.on.foo, self._on_foo)
+
+    def _on_start(self, _):
+        self.on.foo.emit()
+
+    def _on_foo(self, _):
+        pass
+
+    @property
+    def tempo(self):
+        return "foo.bar:80"
+
+
+autoinstrument(MyCharmWithCustomEvents, MyCharmWithCustomEvents.tempo)
+
+
+def test_base_tracer_endpoint_custom_event(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmWithCustomEvents, meta=MyCharmWithCustomEvents.META)
+        ctx.run("start", State())
+
+        spans = f.call_args_list[0].args[0]
+        span_names = [span.name for span in spans]
+        assert span_names == [
+            "method call: MyCharmWithCustomEvents._on_foo",
+            "event: foo",
+            "method call: MyCharmWithCustomEvents._on_start",
+            "event: start",
+            "charm exec",
+        ]
+        # only the charm exec span is a root
+        assert not spans[-1].parent
+        for span in spans[:-1]:
+            assert span.parent
+            assert span.parent.trace_id
+        assert len({(span.parent.trace_id if span.parent else 0) for span in spans}) == 2
+
+
+class MyRemoteCharm(CharmBase):
+    META = {"name": "charlie", "requires": {"tracing": {"interface": "tracing", "limit": 1}}}
+    _request = True
+
+    def __init__(self, framework: Framework):
+        super().__init__(framework)
+        self.tracing = TracingEndpointRequirer(
+            self, "tracing", protocols=(["otlp_http"] if self._request else [])
+        )
+
+    def tempo(self):
+        return self.tracing.get_endpoint("otlp_http")
+
+
+autoinstrument(MyRemoteCharm, MyRemoteCharm.tempo)
+
+
+@pytest.mark.parametrize("leader", (True, False))
+def test_tracing_requirer_remote_charm_request_response(leader):
+    # IF the leader unit (whoever it is) did request the endpoint to be activated
+    MyRemoteCharm._request = True
+    ctx = Context(MyRemoteCharm, meta=MyRemoteCharm.META)
+    # WHEN you get any event AND the remote unit has already replied
+    tracing = scenario.Relation(
+        "tracing",
+        # if we're not leader, assume the leader did its part already
+        local_app_data=(
+            TracingRequirerAppData(receivers=["otlp_http"]).dump() if not leader else {}
+        ),
+        remote_app_data=TracingProviderAppData(
+            host="foo.com",
+            receivers=[
+                Receiver(
+                    url="http://foo.com:80", protocol=ProtocolType(name="otlp_http", type="http")
+                )
+            ],
+        ).dump(),
+    )
+    with ctx.manager("start", State(leader=leader, relations=[tracing])) as mgr:
+        # THEN you're good
+        assert mgr.charm.tempo() == "http://foo.com:80"
+
+
+@pytest.mark.parametrize("leader", (True, False))
+def test_tracing_requirer_remote_charm_no_request_but_response(leader):
+    # IF the leader did NOT request the endpoint to be activated
+    MyRemoteCharm._request = False
+    ctx = Context(MyRemoteCharm, meta=MyRemoteCharm.META)
+    # WHEN you get any event AND the remote unit has already replied
+    tracing = scenario.Relation(
+        "tracing",
+        # empty local app data
+        remote_app_data=TracingProviderAppData(
+            # but the remote end has sent the data you need
+            receivers=[
+                Receiver(
+                    url="http://foo.com:80", protocol=ProtocolType(name="otlp_http", type="http")
+                )
+            ],
+        ).dump(),
+    )
+    with ctx.manager("start", State(leader=leader, relations=[tracing])) as mgr:
+        # THEN you're lucky, but you're good
+        assert mgr.charm.tempo() == "http://foo.com:80"
+
+
+@pytest.mark.parametrize("relation", (True, False))
+@pytest.mark.parametrize("leader", (True, False))
+def test_tracing_requirer_remote_charm_no_request_no_response(leader, relation):
+    """Verify that the charm successfully executes (with charm_tracing disabled) if the tempo() call raises."""
+    # IF the leader did NOT request the endpoint to be activated
+    MyRemoteCharm._request = False
+    ctx = Context(MyRemoteCharm, meta=MyRemoteCharm.META)
+    # WHEN you get any event
+    if relation:
+        # AND you have an empty relation
+        tracing = scenario.Relation(
+            "tracing",
+            # empty local and remote app data
+        )
+        relations = [tracing]
+    else:
+        # OR no relation at all
+        relations = []
+
+    # THEN you're not totally good: self.tempo() will raise, but charm exec will still exit 0
+    with ctx.manager("start", State(leader=leader, relations=relations)) as mgr:
+        with pytest.raises(ProtocolNotRequestedError):
+            assert mgr.charm.tempo() is None
+
+
+class MyRemoteBorkyCharm(CharmBase):
+    META = {"name": "charlie", "requires": {"tracing": {"interface": "tracing", "limit": 1}}}
+    _borky_return_value = None
+
+    def tempo(self):
+        return self._borky_return_value
+
+
+autoinstrument(MyRemoteBorkyCharm, MyRemoteBorkyCharm.tempo)
+
+
+@pytest.mark.parametrize("borky_return_value", (True, 42, object(), 0.2, [], (), {}))
+def test_borky_tempo_return_value(borky_return_value, caplog):
+    """Verify that the charm exits 0 (with charm_tracing disabled) if the tempo() call returns bad values."""
+    # IF the charm's tempo endpoint getter returns anything but None or str
+    MyRemoteBorkyCharm._borky_return_value = borky_return_value
+    ctx = Context(MyRemoteBorkyCharm, meta=MyRemoteBorkyCharm.META)
+    # WHEN you get any event
+    # THEN you're not totally good: self.tempo() will raise, but charm exec will still exit 0
+
+    ctx.run("start", State())
+    # traceback from the TypeError raised by _get_tracing_endpoint
+    assert "should return a tempo endpoint" in caplog.text
+    # logger.exception in _setup_root_span_initializer
+    assert "exception retrieving the tracing endpoint from" in caplog.text
+    assert "proceeding with charm_tracing DISABLED." in caplog.text
+
+
+class MyCharmStaticMethods(CharmBase):
+    META = {"name": "jolene"}
+
+    def __init__(self, fw):
+        super().__init__(fw)
+        fw.observe(self.on.start, self._on_start)
+        fw.observe(self.on.update_status, self._on_update_status)
+
+    def _on_start(self, _):
+        for o in (OtherObj(), OtherObj):
+            for meth in ("_staticmeth", "_staticmeth1", "_staticmeth2"):
+                assert getattr(o, meth)(1) == 2
+
+    def _on_update_status(self, _):
+        # super-ugly edge cases
+        OtherObj()._staticmeth3(OtherObj())
+        OtherObj()._staticmeth4(OtherObj())
+        OtherObj._staticmeth3(OtherObj())
+        OtherObj._staticmeth4(OtherObj(), foo=2)
+
+    @property
+    def tempo(self):
+        return "foo.bar:80"
+
+
+class OtherObj:
+    @staticmethod
+    def _staticmeth(i: int, *args, **kwargs):
+        return 1 + i
+
+    @staticmethod
+    def _staticmeth1(i: int):
+        return 1 + i
+
+    @staticmethod
+    def _staticmeth2(i: int, foo="bar"):
+        return 1 + i
+
+    @staticmethod
+    def _staticmeth3(abc: "OtherObj", foo="bar"):
+        return 1 + 1
+
+    @staticmethod
+    def _staticmeth4(abc: int, foo="bar"):
+        return 1 + 1
+
+
+autoinstrument(MyCharmStaticMethods, MyCharmStaticMethods.tempo, extra_types=[OtherObj])
+
+
+def test_trace_staticmethods(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmStaticMethods, meta=MyCharmStaticMethods.META)
+        ctx.run("start", State())
+
+        spans = f.call_args_list[0].args[0]
+
+        span_names = [span.name for span in spans]
+        assert span_names == [
+            "method call: OtherObj._staticmeth",
+            "method call: OtherObj._staticmeth1",
+            "method call: OtherObj._staticmeth2",
+            "method call: OtherObj._staticmeth",
+            "method call: OtherObj._staticmeth1",
+            "method call: OtherObj._staticmeth2",
+            "method call: MyCharmStaticMethods._on_start",
+            "event: start",
+            "charm exec",
+        ]
+
+        for span in spans:
+            assert span.resource.attributes["service.name"] == "jolene"
+
+
+def test_trace_staticmethods_bork(caplog):
+    import opentelemetry
+
+    with patch(
+        "opentelemetry.exporter.otlp.proto.http.trace_exporter.OTLPSpanExporter.export"
+    ) as f:
+        f.return_value = opentelemetry.sdk.trace.export.SpanExportResult.SUCCESS
+        ctx = Context(MyCharmStaticMethods, meta=MyCharmStaticMethods.META)
+        ctx.run("update-status", State())
diff --git a/tests/scenario/test_clustering_consistency.py b/tests/scenario/test_clustering_consistency.py
new file mode 100644
index 0000000..9865b3c
--- /dev/null
+++ b/tests/scenario/test_clustering_consistency.py
@@ -0,0 +1,267 @@
+from typing import Dict
+from unittest.mock import patch
+
+import pytest
+import scenario
+from scenario.state import next_relation_id
+
+from coordinator import MINIMAL_DEPLOYMENT, RECOMMENDED_DEPLOYMENT
+from tempo_cluster import (
+    JujuTopology,
+    TempoClusterRequirerAppData,
+    TempoClusterRequirerUnitData,
+    TempoRole,
+)
+
+SCALABLE_MODE = "scalable-single-binary"
+MONOLITH_MODE = "all"
+
+
+def assert_tempo_running_with_target(target: str, state: scenario.State):
+    tempo_layer_cmd = state.get_container("tempo").plan.to_dict()["services"]["tempo"]["command"]
+    assert f"-target {target}" in tempo_layer_cmd
+
+
+def assert_tempo_not_running(state: scenario.State):
+    tempo_svc = state.get_container("tempo").services.get("tempo")
+    assert not tempo_svc or not tempo_svc.is_running()
+
+
+def assert_status(state: scenario.State, unit: str = None, app: str = None, message: str = None):
+    if unit:
+        status = state.unit_status
+        assert status.name == unit
+        if message:
+            assert status.message == message
+
+    if app:
+        status = state.app_status
+        assert status.name == app
+        if message:
+            assert status.message == message
+
+
+@pytest.fixture
+def peers():
+    return scenario.PeerRelation("tempo-peers", peers_data={0: {}, 1: {}})
+
+
+@pytest.fixture
+def cluster():
+    return scenario.Relation("tempo-cluster")
+
+
+@pytest.fixture
+def s3():
+    return scenario.Relation(
+        "s3",
+        remote_app_data={
+            "access-key": "key",
+            "bucket": "tempo",
+            "endpoint": "http://1.2.3.4:9000",
+            "secret-key": "soverysecret",
+        },
+        local_unit_data={"bucket": "tempo"},
+    )
+
+
+@pytest.fixture
+def mock_tempo_workload_ready():
+    with patch("tempo.Tempo.is_ready", new=lambda _: True):
+        yield
+
+
+@pytest.fixture
+def tempo_container_ready(context, mock_tempo_workload_ready):
+    tempo = scenario.Container("tempo", can_connect=True)
+    return context.run(
+        tempo.pebble_ready_event, scenario.State(leader=True, containers=[tempo])
+    ).get_container("tempo")
+
+
+def test_monolithic(context, tempo_container_ready):
+    state = scenario.State(leader=True, containers=[tempo_container_ready])
+    state_out = context.run("update-status", state)
+
+    assert_tempo_running_with_target(MONOLITH_MODE, state_out)
+    assert_status(state_out, unit="active")
+
+
+def test_scaling_no_s3_inconsistent(context, tempo_container_ready, peers):
+    state = scenario.State(leader=True, containers=[tempo_container_ready], relations=[peers])
+
+    # it doesn't actually matter what event we're running here
+    state_out = context.run("update-status", state)
+    assert_tempo_not_running(state_out)
+    assert_status(state_out, unit="blocked")
+
+
+def test_clustered_and_scaled_coordinator_no_s3_inconsistent(
+    context, peers, cluster, tempo_container_ready
+):
+    state = scenario.State(
+        leader=True, containers=[tempo_container_ready], relations=[peers, cluster]
+    )
+
+    # it doesn't actually matter what event we're running here
+    state_out = context.run("update-status", state)
+    assert_tempo_not_running(state_out)
+    assert_status(state_out, unit="blocked")
+
+
+def test_clustered_coordinator_no_s3_inconsistent(context, tempo_container_ready, cluster):
+    state = scenario.State(leader=True, containers=[tempo_container_ready], relations=[cluster])
+    state_out = context.run("update-status", state)
+    assert_tempo_not_running(state_out)
+    assert_status(state_out, unit="blocked")
+
+
+def test_single_coordinator_no_s3_no_worker_inconsistent(context, tempo_container_ready, cluster):
+    state = scenario.State(
+        leader=True,
+        config={"coordinator_runs_workload_when_clustered": False},
+        containers=[tempo_container_ready],
+        relations=[cluster],
+    )
+    state_out = context.run("update-status", state)
+    assert_tempo_not_running(state_out)
+    assert_status(state_out, unit="blocked")
+
+
+@pytest.fixture
+def s3_ready_state(context, tempo_container_ready, peers, s3):
+    state = scenario.State(leader=True, containers=[tempo_container_ready], relations=[s3])
+    # we need to run s3-changed to have tempo push the configs to disk and be ready to
+    # run in scalable/distributed mode
+    return context.run(s3.changed_event, state)
+
+
+def test_scaling_with_s3_consistent(context, s3_ready_state, tempo_container_ready, s3, peers):
+    state = s3_ready_state.replace(relations=[peers, s3])
+    with context.manager(peers.joined_event, state) as mgr:
+        assert mgr.charm._is_consistent
+        state_out = mgr.run()
+    assert_tempo_running_with_target(SCALABLE_MODE, state_out)
+    assert_status(state_out, unit="active")
+
+
+def test_clustered_scaling_with_s3_consistent(
+    context, s3_ready_state, tempo_container_ready, s3, peers, cluster
+):
+    state = s3_ready_state.replace(relations=[peers, s3, cluster])
+
+    # doesn't matter what event we're processing...
+    for event in [cluster.joined_event, peers.joined_event]:
+        state_out = context.run(event, state)
+        assert_tempo_running_with_target(SCALABLE_MODE, state_out)
+        assert_status(state_out, unit="active")
+
+
+def test_clustered_with_s3_consistent(context, s3_ready_state, tempo_container_ready, s3, cluster):
+    state = s3_ready_state.replace(relations=[s3, cluster])
+
+    state_out = context.run(cluster.joined_event, state)
+    assert_tempo_running_with_target(SCALABLE_MODE, state_out)
+    assert_status(state_out, unit="active")
+
+
+def test_coord_only_scaling_with_s3_consistent(
+    context, s3_ready_state, tempo_container_ready, s3, peers
+):
+    state = s3_ready_state.replace(relations=[peers, s3])
+
+    with context.manager(s3.changed_event, state) as mgr:
+        assert not mgr.charm.tempo_cluster.has_workers
+        # coord is also worker because we don't have a cluster relation
+        assert mgr.charm.coordinator._is_worker
+
+        # still coherent, because we're running the worker!
+        assert mgr.charm.coordinator.is_coherent
+        assert mgr.charm._is_consistent
+        state_out = mgr.run()
+
+    assert_tempo_running_with_target(SCALABLE_MODE, state_out)
+    assert_status(state_out, unit="active")
+
+
+def test_coord_only_clustered_scaling_with_s3_inconsistent(
+    context, tempo_container_ready, s3, peers, cluster, s3_ready_state
+):
+    state = s3_ready_state.replace(
+        config={"coordinator_runs_workload_when_clustered": False}, relations=[peers, s3, cluster]
+    )
+
+    # doesn't quite matter what event we're processing
+    with context.manager("update-status", state) as mgr:
+        # we have a cluster relation -> we have a worker
+        assert mgr.charm.tempo_cluster.has_workers
+
+        # coord is NOT worker because we have a cluster relation
+        # we're configured to stop worker when we have a cluster
+        # fixme do we have a good reason to only stop the worker when we're clustered,
+        #  instead of when we're clustered AND coherent?
+        assert not mgr.charm.coordinator._is_worker
+
+        # not coherent, because we're running the worker and the cluster doesn't have all roles
+        assert not mgr.charm.coordinator.is_coherent
+        assert not mgr.charm._is_consistent
+        state_out = mgr.run()
+
+    assert_tempo_not_running(state_out)
+    assert_status(state_out, unit="blocked")
+
+
+def test_coord_only_clustered_with_s3_inconsistent(
+    context, tempo_container_ready, s3, cluster, s3_ready_state
+):
+    state = s3_ready_state.replace(
+        config={"coordinator_runs_workload_when_clustered": False}, relations=[s3, cluster]
+    )
+
+    # doesn't quite matter what event we're processing
+    state_out = context.run("update-status", state)
+    assert_tempo_not_running(state_out)
+    assert_status(state_out, unit="blocked")
+
+
+def generate_cluster_relations(deployment_spec: Dict[TempoRole, int], template: scenario.Relation):
+    cluster_relations = []
+    for role, n in deployment_spec.items():
+        c = template.replace(
+            relation_id=next_relation_id(),
+            remote_app_data=TempoClusterRequirerAppData(role=role).dump(),
+            remote_units_data={
+                i: TempoClusterRequirerUnitData(
+                    juju_topology=JujuTopology(model="model-name", unit=str(i)),
+                    address=f"0.0.0.{i}",
+                ).dump()
+                for i in range(n)
+            },
+        )
+        cluster_relations.append(c)
+    return cluster_relations
+
+
+def test_full_clustering_consistent(context, tempo_container_ready, s3, cluster, s3_ready_state):
+    """The happy path of all happy paths."""
+    state = s3_ready_state.replace(
+        config={"coordinator_runs_workload_when_clustered": False},
+        relations=[s3, *generate_cluster_relations(MINIMAL_DEPLOYMENT, cluster)],
+    )
+
+    # doesn't quite matter what event we're processing
+    state_out = context.run("update-status", state)
+    assert_tempo_running_with_target(SCALABLE_MODE, state_out)
+    assert_status(state_out, unit="active", message="[coordinator] degraded")
+
+
+def test_full_clustering_recommended(context, tempo_container_ready, s3, cluster, s3_ready_state):
+    state = s3_ready_state.replace(
+        config={"coordinator_runs_workload_when_clustered": False},
+        relations=[s3, *generate_cluster_relations(RECOMMENDED_DEPLOYMENT, cluster)],
+    )
+
+    # doesn't quite matter what event we're processing
+    state_out = context.run("update-status", state)
+    assert_tempo_running_with_target(SCALABLE_MODE, state_out)
+    assert_status(state_out, unit="active", message="")
diff --git a/tests/scenario/test_ingressed_tracing.py b/tests/scenario/test_ingressed_tracing.py
new file mode 100644
index 0000000..ffb8027
--- /dev/null
+++ b/tests/scenario/test_ingressed_tracing.py
@@ -0,0 +1,101 @@
+from unittest.mock import patch
+
+import pytest
+import yaml
+from charms.tempo_k8s.v1.charm_tracing import charm_tracing_disabled
+from scenario import Container, Relation, State
+
+from tempo import Tempo
+
+
+@pytest.fixture
+def base_state():
+    return State(leader=True, containers=[Container("tempo", can_connect=False)])
+
+
+def test_external_url_present(context, base_state):
+    # WHEN ingress is related with external_host
+    tracing = Relation("tracing", remote_app_data={"receivers": "[]"})
+    ingress = Relation("ingress", remote_app_data={"external_host": "1.2.3.4", "scheme": "http"})
+    state = base_state.replace(relations=[tracing, ingress])
+
+    with charm_tracing_disabled():
+        out = context.run(getattr(tracing, "created_event"), state)
+
+    # THEN external_url is present in tracing relation databag
+    tracing_out = out.get_relations(tracing.endpoint)[0]
+    assert tracing_out.local_app_data == {
+        "receivers": '[{"protocol": {"name": "otlp_http", "type": "http"}, "url": "http://1.2.3.4:4318"}]',
+    }
+
+
+@patch("socket.getfqdn", lambda: "1.2.3.4")
+def test_ingress_relation_set_with_dynamic_config(context, base_state):
+    # WHEN ingress is related with external_host
+    ingress = Relation("ingress", remote_app_data={"external_host": "1.2.3.4", "scheme": "http"})
+    state = base_state.replace(relations=[ingress])
+
+    with patch.object(Tempo, "is_ready", lambda _: False):
+        out = context.run(ingress.joined_event, state)
+
+    expected_rel_data = {
+        "http": {
+            "routers": {
+                f"juju-{state.model.name}-tempo-k8s-jaeger-thrift-http": {
+                    "entryPoints": ["jaeger-thrift-http"],
+                    "rule": "ClientIP(`0.0.0.0/0`)",
+                    "service": f"juju-{state.model.name}-tempo-k8s-service-jaeger-thrift-http",
+                },
+                f"juju-{state.model.name}-tempo-k8s-otlp-http": {
+                    "entryPoints": ["otlp-http"],
+                    "rule": "ClientIP(`0.0.0.0/0`)",
+                    "service": f"juju-{state.model.name}-tempo-k8s-service-otlp-http",
+                },
+                f"juju-{state.model.name}-tempo-k8s-tempo-http": {
+                    "entryPoints": ["tempo-http"],
+                    "rule": "ClientIP(`0.0.0.0/0`)",
+                    "service": f"juju-{state.model.name}-tempo-k8s-service-tempo-http",
+                },
+                f"juju-{state.model.name}-tempo-k8s-zipkin": {
+                    "entryPoints": ["zipkin"],
+                    "rule": "ClientIP(`0.0.0.0/0`)",
+                    "service": f"juju-{state.model.name}-tempo-k8s-service-zipkin",
+                },
+                f"juju-{state.model.name}-tempo-k8s-otlp-grpc": {
+                    "entryPoints": ["otlp-grpc"],
+                    "rule": "ClientIP(`0.0.0.0/0`)",
+                    "service": f"juju-{state.model.name}-tempo-k8s-service-otlp-grpc",
+                },
+                f"juju-{state.model.name}-tempo-k8s-tempo-grpc": {
+                    "entryPoints": ["tempo-grpc"],
+                    "rule": "ClientIP(`0.0.0.0/0`)",
+                    "service": f"juju-{state.model.name}-tempo-k8s-service-tempo-grpc",
+                },
+            },
+            "services": {
+                f"juju-{state.model.name}-tempo-k8s-service-jaeger-thrift-http": {
+                    "loadBalancer": {"servers": [{"url": "http://1.2.3.4:14268"}]}
+                },
+                f"juju-{state.model.name}-tempo-k8s-service-otlp-http": {
+                    "loadBalancer": {"servers": [{"url": "http://1.2.3.4:4318"}]}
+                },
+                f"juju-{state.model.name}-tempo-k8s-service-tempo-http": {
+                    "loadBalancer": {"servers": [{"url": "http://1.2.3.4:3200"}]}
+                },
+                f"juju-{state.model.name}-tempo-k8s-service-zipkin": {
+                    "loadBalancer": {"servers": [{"url": "http://1.2.3.4:9411"}]}
+                },
+                f"juju-{state.model.name}-tempo-k8s-service-otlp-grpc": {
+                    "loadBalancer": {"servers": [{"url": "h2c://1.2.3.4:4317"}]},
+                },
+                f"juju-{state.model.name}-tempo-k8s-service-tempo-grpc": {
+                    "loadBalancer": {"servers": [{"url": "h2c://1.2.3.4:9096"}]}
+                },
+            },
+        },
+    }
+
+    # THEN dynamic config is present in ingress relation
+    ingress_out = out.get_relations(ingress.endpoint)[0]
+    assert ingress_out.local_app_data
+    assert yaml.safe_load(ingress_out.local_app_data["config"]) == expected_rel_data
diff --git a/tests/scenario/test_tempo_clustered.py b/tests/scenario/test_tempo_clustered.py
new file mode 100644
index 0000000..f29e5bd
--- /dev/null
+++ b/tests/scenario/test_tempo_clustered.py
@@ -0,0 +1,119 @@
+import datetime
+from unittest.mock import patch
+
+import pytest
+import scenario
+from charms.tls_certificates_interface.v3.tls_certificates import ProviderCertificate
+
+from charm import TempoCharm
+from tempo_cluster import TempoClusterProviderAppData
+
+
+@pytest.fixture
+def cluster():
+    return scenario.Relation("tempo-cluster")
+
+
+@pytest.fixture
+def s3():
+    return scenario.Relation(
+        "s3",
+        remote_app_data={
+            "access-key": "key",
+            "bucket": "tempo",
+            "endpoint": "http://1.2.3.4:9000",
+            "secret-key": "soverysecret",
+        },
+        local_unit_data={"bucket": "tempo"},
+    )
+
+
+@pytest.fixture
+def certs_relation():
+    return scenario.Relation("certificates")
+
+
+#
+# @pytest.fixture
+# def certs_secret():
+#     return scenario.Secret(
+#         "certificates-secret-id",
+#         owner="app",
+#         contents={0:
+#             {
+#                 "private-key": "supersecretkey",
+#                 "ca-cert": "CA_CERT-foo",
+#                 "server-cert": "SERVER_CERT-foo",
+#             }},
+#     )
+
+
+@pytest.fixture
+def mock_tempo_workload_ready():
+    with patch("tempo.Tempo.is_ready", new=lambda _: True):
+        yield
+
+
+@pytest.fixture
+def tempo_container_ready(context, mock_tempo_workload_ready):
+    tempo = scenario.Container("tempo", can_connect=True)
+    return context.run(
+        tempo.pebble_ready_event, scenario.State(leader=True, containers=[tempo])
+    ).get_container("tempo")
+
+
+MOCK_SERVER_CERT = "SERVER_CERT-foo"
+MOCK_CA_CERT = "CA_CERT-foo"
+
+
+@pytest.fixture(autouse=True)
+def patch_certs():
+    cert = ProviderCertificate(
+        relation_id=42,
+        application_name="tempo",
+        csr="CSR",
+        certificate=MOCK_SERVER_CERT,
+        ca=MOCK_CA_CERT,
+        chain=[],
+        revoked=False,
+        expiry_time=datetime.datetime(2050, 4, 1),
+    )
+    with patch(
+        "charms.observability_libs.v1.cert_handler.CertHandler.get_cert", new=lambda _: cert
+    ):
+        yield
+
+
+@pytest.fixture
+def state_with_certs(context, tempo_container_ready, s3, certs_relation):
+    return context.run(
+        certs_relation.joined_event,
+        scenario.State(
+            leader=True, containers=[tempo_container_ready], relations=[s3, certs_relation]
+        ),
+    )
+
+
+def test_certs_ready(context, state_with_certs):
+    with context.manager("update-status", state_with_certs) as mgr:
+        charm: TempoCharm = mgr.charm
+        assert charm.cert_handler.server_cert == MOCK_SERVER_CERT
+        assert charm.cert_handler.ca_cert == MOCK_CA_CERT
+        assert charm.cert_handler.private_key
+
+
+def test_cluster_relation(context, state_with_certs, cluster):
+    clustered_state = state_with_certs.replace(relations=state_with_certs.relations + [cluster])
+
+    state_out = context.run(cluster.joined_event, clustered_state)
+    cluster_out = state_out.get_relations(cluster.endpoint)[0]
+    local_app_data = TempoClusterProviderAppData.load(cluster_out.local_app_data)
+
+    assert local_app_data.ca_cert == MOCK_CA_CERT
+    assert local_app_data.server_cert == MOCK_SERVER_CERT
+    secret = [s for s in state_out.secrets if s.id == local_app_data.privkey_secret_id][0]
+
+    # certhandler's vault uses revision 0 to store an uninitialized-vault marker
+    assert secret.contents[1]["private-key"]
+
+    assert local_app_data.tempo_config
diff --git a/tests/scenario/test_tls.py b/tests/scenario/test_tls.py
new file mode 100644
index 0000000..3753548
--- /dev/null
+++ b/tests/scenario/test_tls.py
@@ -0,0 +1,133 @@
+import socket
+from unittest.mock import patch
+
+import pytest
+from charms.tempo_k8s.v1.charm_tracing import charm_tracing_disabled
+from charms.tempo_k8s.v2.tracing import TracingProviderAppData, TracingRequirerAppData
+from scenario import Container, Relation, State
+
+from charm import Tempo
+
+
+@pytest.fixture
+def base_state():
+    return State(leader=True, containers=[Container("tempo", can_connect=False)])
+
+
+def update_relations_tls_and_verify(
+    base_state,
+    context,
+    has_ingress,
+    local_has_tls,
+    local_scheme,
+    relations,
+    remote_scheme,
+    tracing,
+):
+    state = base_state.replace(relations=relations)
+    with charm_tracing_disabled(), patch.object(Tempo, "tls_ready", local_has_tls):
+        out = context.run(tracing.changed_event, state)
+    tracing_provider_app_data = TracingProviderAppData.load(
+        out.get_relations(tracing.endpoint)[0].local_app_data
+    )
+    actual_url = tracing_provider_app_data.receivers[0].url
+    expected_url = f"{remote_scheme if has_ingress else local_scheme}://{socket.getfqdn() if not has_ingress else 'foo.com.org'}:4318"
+    assert actual_url == expected_url
+    return out
+
+
+@pytest.mark.parametrize("remote_has_tls", (True, False))
+@pytest.mark.parametrize("local_has_tls", (True, False))
+@pytest.mark.parametrize("has_ingress", (True, False))
+def test_tracing_endpoints_with_tls(
+    context, base_state, has_ingress, local_has_tls, remote_has_tls
+):
+    tracing = Relation(
+        "tracing",
+        remote_app_data=TracingRequirerAppData(receivers=["otlp_http"]).dump(),
+    )
+    relations = [tracing]
+
+    local_scheme = "https" if local_has_tls else "http"
+    remote_scheme = "https" if remote_has_tls else "http"
+
+    if has_ingress:
+        relations.append(
+            Relation(
+                "ingress",
+                remote_app_data={"scheme": remote_scheme, "external_host": "foo.com.org"},
+            )
+        )
+
+    update_relations_tls_and_verify(
+        base_state,
+        context,
+        has_ingress,
+        local_has_tls,
+        local_scheme,
+        relations,
+        remote_scheme,
+        tracing,
+    )
+
+
+@pytest.mark.parametrize("has_ingress", (True, False))
+def test_tracing_endpoints_tls_added_then_removed(context, base_state, has_ingress):
+    tracing = Relation(
+        "tracing",
+        remote_app_data=TracingRequirerAppData(receivers=["otlp_http"]).dump(),
+    )
+    relations = [tracing]
+
+    local_scheme = "http"
+    remote_scheme = "http"
+
+    if has_ingress:
+        relations.append(
+            Relation(
+                "ingress",
+                remote_app_data={"scheme": remote_scheme, "external_host": "foo.com.org"},
+            )
+        )
+
+    result_state = update_relations_tls_and_verify(
+        base_state, context, has_ingress, False, local_scheme, relations, remote_scheme, tracing
+    )
+
+    # then we check the scenario where TLS gets enabled
+
+    local_scheme = "https"
+    remote_scheme = "https"
+
+    if has_ingress:
+        # as remote_scheme changed, we need to update the ingress relation
+        relations.pop()
+        relations.append(
+            Relation(
+                "ingress",
+                remote_app_data={"scheme": remote_scheme, "external_host": "foo.com.org"},
+            )
+        )
+
+    result_state = update_relations_tls_and_verify(
+        result_state, context, has_ingress, True, local_scheme, relations, remote_scheme, tracing
+    )
+
+    # then we again remove TLS and compare the same thing
+
+    local_scheme = "http"
+    remote_scheme = "http"
+
+    if has_ingress:
+        # as remote_scheme changed, we need to update the ingress relation
+        relations.pop()
+        relations.append(
+            Relation(
+                "ingress",
+                remote_app_data={"scheme": remote_scheme, "external_host": "foo.com.org"},
+            )
+        )
+
+    update_relations_tls_and_verify(
+        result_state, context, has_ingress, False, local_scheme, relations, remote_scheme, tracing
+    )
diff --git a/tests/scenario/test_tracing_legacy.py b/tests/scenario/test_tracing_legacy.py
new file mode 100644
index 0000000..b3674bc
--- /dev/null
+++ b/tests/scenario/test_tracing_legacy.py
@@ -0,0 +1,26 @@
+import socket
+
+import pytest
+from charms.tempo_k8s.v1.charm_tracing import charm_tracing_disabled
+from scenario import Container, Relation, State
+
+
+@pytest.fixture
+def base_state():
+    return State(leader=True, containers=[Container("tempo", can_connect=False)])
+
+
+@pytest.mark.parametrize("evt_name", ("changed", "created", "joined"))
+def test_tracing_v2_endpoint_published(context, evt_name, base_state):
+    tracing = Relation("tracing", remote_app_data={"receivers": "[]"})
+    state = base_state.replace(relations=[tracing])
+
+    with charm_tracing_disabled():
+        with context.manager(getattr(tracing, f"{evt_name}_event"), state) as mgr:
+            assert len(mgr.charm._requested_receivers()) == 1
+            out = mgr.run()
+
+    tracing_out = out.get_relations(tracing.endpoint)[0]
+    assert tracing_out.local_app_data == {
+        "receivers": f'[{{"protocol": {{"name": "otlp_http", "type": "http"}}, "url": "http://{socket.getfqdn()}:4318"}}]',
+    }
diff --git a/tests/scenario/test_tracing_provider.py b/tests/scenario/test_tracing_provider.py
new file mode 100644
index 0000000..6342d5a
--- /dev/null
+++ b/tests/scenario/test_tracing_provider.py
@@ -0,0 +1,48 @@
+import pytest
+from charms.tempo_k8s.v1.charm_tracing import charm_tracing_disabled
+from charms.tempo_k8s.v2.tracing import ProtocolType, TracingProviderAppData
+from scenario import Container, Context, Relation, State
+
+
+@pytest.fixture
+def context(tempo_charm):
+    return Context(
+        charm_type=tempo_charm,
+    )
+
+
+def test_receivers_removed_on_relation_broken(context):
+    tracing_grpc = Relation(
+        "tracing",
+        remote_app_data={"receivers": '["otlp_grpc"]'},
+        local_app_data={
+            "receivers": '[{"protocol": {"name": "otlp_grpc", "type": "grpc"} , "url": "foo.com:10"}, '
+            '{"protocol": {"name": "otlp_http", "type": "http"}, "url": "http://foo.com:11"}, '
+        },
+    )
+    tracing_http = Relation(
+        "tracing",
+        remote_app_data={"receivers": '["otlp_http"]'},
+        local_app_data={
+            "receivers": '[{"protocol": {"name": "otlp_grpc", "type": "grpc"} , "url": "foo.com:10"}, '
+            '{"protocol": {"name": "otlp_http", "type": "http"}, "url": "http://foo.com:11"}, '
+        },
+    )
+
+    state = State(
+        leader=True,
+        relations=[tracing_grpc, tracing_http],
+        containers=[Container("tempo", can_connect=False)],
+    )
+
+    with charm_tracing_disabled():
+        with context.manager(tracing_grpc.broken_event, state) as mgr:
+            charm = mgr.charm
+            assert charm._requested_receivers() == ("otlp_http",)
+
+    state_out = mgr.output
+    r_out = [r for r in state_out.relations if r.relation_id == tracing_http.relation_id][0]
+    # "otlp_grpc" is gone from the databag
+    assert [r.protocol for r in TracingProviderAppData.load(r_out.local_app_data).receivers] == [
+        ProtocolType(name="otlp_http", type="http")
+    ]
diff --git a/tests/scenario/test_tracing_requirer.py b/tests/scenario/test_tracing_requirer.py
new file mode 100644
index 0000000..517811e
--- /dev/null
+++ b/tests/scenario/test_tracing_requirer.py
@@ -0,0 +1,201 @@
+import socket
+
+import pytest
+from charms.tempo_k8s.v1.charm_tracing import charm_tracing_disabled
+from charms.tempo_k8s.v2.tracing import (
+    EndpointChangedEvent,
+    EndpointRemovedEvent,
+    ProtocolNotRequestedError,
+    TracingEndpointRequirer,
+)
+from ops import CharmBase, Framework, RelationBrokenEvent, RelationChangedEvent
+from scenario import Context, Relation, State
+
+from tempo import Tempo
+
+
+class MyCharm(CharmBase):
+    def __init__(self, framework: Framework):
+        super().__init__(framework)
+        self.tracing = TracingEndpointRequirer(self, protocols=["otlp_grpc"])
+        framework.observe(self.tracing.on.endpoint_changed, self._on_endpoint_changed)
+
+    def _on_endpoint_changed(self, e):
+        pass
+
+
+@pytest.fixture
+def context():
+    return Context(
+        charm_type=MyCharm,
+        meta={"name": "jolly", "requires": {"tracing": {"interface": "tracing", "limit": 1}}},
+    )
+
+
+def test_requirer_api(context):
+    host = socket.getfqdn()
+    tracing = Relation(
+        "tracing",
+        remote_app_data={
+            "receivers": f'[{{"protocol": {{"name": "otlp_grpc", "type": "grpc"}}, "url": "{host}:4317"}}, '
+            f'{{"protocol": {{"name": "otlp_http", "type": "http"}}, "url": "http://{host}:4318"}}, '
+            f'{{"protocol": {{"name": "zipkin", "type": "http"}}, "url": "http://{host}:9411" }}]',
+        },
+    )
+    state = State(leader=True, relations=[tracing])
+
+    with charm_tracing_disabled():
+        with context.manager(tracing.changed_event, state) as mgr:
+            charm = mgr.charm
+            assert charm.tracing.get_endpoint("otlp_grpc") == f"{host}:4317"
+            assert charm.tracing.get_endpoint("otlp_http") == f"http://{host}:4318"
+            assert charm.tracing.get_endpoint("zipkin") == f"http://{host}:9411"
+
+            rel = charm.model.get_relation("tracing")
+            assert charm.tracing.is_ready(rel)
+
+    rchanged, epchanged = context.emitted_events
+    assert isinstance(epchanged, EndpointChangedEvent)
+    assert epchanged.receivers[0].protocol.name == "otlp_grpc"
+    assert epchanged.receivers[1].protocol.name == "otlp_http"
+    assert epchanged.receivers[2].protocol.name == "zipkin"
+
+
+def test_requirer_api_with_internal_scheme(context):
+    host = socket.getfqdn()
+    tracing = Relation(
+        "tracing",
+        remote_app_data={
+            "receivers": f'[{{"protocol": {{"name": "otlp_grpc", "type": "grpc"}} , "url": "{host}:4317"}}, '
+            f'{{"protocol": {{"name": "otlp_http", "type": "http"}}, "url": "https://{host}:4318"}}, '
+            f'{{"protocol": {{"name": "zipkin", "type": "http"}}, "url":  "https://{host}:9411"}}]',
+        },
+    )
+    state = State(leader=True, relations=[tracing])
+
+    with charm_tracing_disabled():
+        with context.manager(tracing.changed_event, state) as mgr:
+            charm = mgr.charm
+            assert charm.tracing.get_endpoint("otlp_grpc") == f"{host}:4317"
+            assert charm.tracing.get_endpoint("otlp_http") == f"https://{host}:4318"
+            assert charm.tracing.get_endpoint("zipkin") == f"https://{host}:9411"
+
+            rel = charm.model.get_relation("tracing")
+            assert charm.tracing.is_ready(rel)
+
+    rchanged, epchanged = context.emitted_events
+    assert isinstance(epchanged, EndpointChangedEvent)
+    assert epchanged.receivers[0].protocol.name == "otlp_grpc"
+
+
+def test_ingressed_requirer_api(context):
+    # WHEN external_url is present in remote app databag
+    external_url = "http://1.2.3.4"
+    tracing = Relation(
+        "tracing",
+        remote_app_data={
+            "receivers": f'[{{"protocol": {{"name": "otlp_grpc", "type": "grpc"}}, "url": "{external_url.split("://")[1]}:4317" }}, '
+            f'{{"protocol": {{"name": "otlp_http", "type": "http"}} , "url": "{external_url}:4318" }}, '
+            f'{{"protocol": {{"name": "zipkin", "type": "http"}} , "url": "{external_url}:9411" }}]',
+        },
+    )
+    state = State(leader=True, relations=[tracing])
+
+    # THEN get_endpoint uses external URL instead of the host
+    with charm_tracing_disabled():
+        with context.manager(tracing.changed_event, state) as mgr:
+            charm = mgr.charm
+            assert (
+                charm.tracing.get_endpoint("otlp_grpc")
+                == f"{external_url.split('://')[1]}:{Tempo.receiver_ports['otlp_grpc']}"
+            )
+            for proto in ["otlp_http", "zipkin"]:
+                assert (
+                    charm.tracing.get_endpoint(proto)
+                    == f"{external_url}:{Tempo.receiver_ports[proto]}"
+                )
+
+            rel = charm.model.get_relation("tracing")
+            assert charm.tracing.is_ready(rel)
+
+    rchanged, epchanged = context.emitted_events
+    assert isinstance(epchanged, EndpointChangedEvent)
+    assert epchanged.receivers[0].protocol.name == "otlp_grpc"
+
+
+@pytest.mark.parametrize(
+    "data",
+    (
+        {
+            "ingesters": '[{"protocol": "otlp_grpc", "port": 9999}]',
+            "bar": "baz",
+        },
+        {
+            "host": "foo.com",
+            "bar": "baz",
+        },
+        {
+            "ingesters": '[{"burp": "barp", "port": 3200}]',
+            "host": "foo.com",
+        },
+        {
+            "ingesters": '[{"protocol": "tempo", "burp": "borp"}]',
+            "host": "foo.com",
+        },
+    ),
+)
+def test_invalid_data(context, data):
+    tracing = Relation(
+        "tracing",
+        remote_app_data=data,
+    )
+    state = State(leader=True, relations=[tracing])
+
+    def post_event(charm: MyCharm):
+        rel = charm.model.get_relation("tracing")
+        assert not charm.tracing.is_ready(rel)
+
+    with charm_tracing_disabled():
+        context.run(tracing.changed_event, state, post_event=post_event)
+
+    emitted_events = context.emitted_events
+    assert len(emitted_events) == 2
+    rchanged, rremoved = emitted_events
+    assert isinstance(rchanged, RelationChangedEvent)
+    assert isinstance(rremoved, EndpointRemovedEvent)
+
+
+def test_broken(context):
+    tracing = Relation("tracing")
+    state = State(leader=True, relations=[tracing])
+
+    with charm_tracing_disabled():
+        context.run(tracing.broken_event, state)
+
+    emitted_events = context.emitted_events
+    assert len(emitted_events) == 2
+    rchanged, ebroken = emitted_events
+    assert isinstance(rchanged, RelationBrokenEvent)
+    assert isinstance(ebroken, EndpointRemovedEvent)
+
+
+def test_requested_not_yet_replied(context):
+    tracing = Relation("tracing")
+    state = State(leader=True, relations=[tracing])
+
+    with charm_tracing_disabled():
+        with context.manager(tracing.created_event, state) as mgr:
+            charm = mgr.charm
+            charm.tracing.request_protocols(["otlp_http"])
+            charm.tracing.get_endpoint("otlp_http")
+
+
+def test_not_requested_raises(context):
+    tracing = Relation("tracing")
+    state = State(leader=True, relations=[tracing])
+
+    with charm_tracing_disabled():
+        with context.manager(tracing.created_event, state) as mgr:
+            charm = mgr.charm
+            with pytest.raises(ProtocolNotRequestedError):
+                charm.tracing.get_endpoint("otlp_http")
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
index 630bb80..79c0c6f 100644
--- a/tests/unit/test_charm.py
+++ b/tests/unit/test_charm.py
@@ -1,24 +1,68 @@
-# Copyright 2023 Ubuntu
+# Copyright 2024 Canonical Ltd.
 # See LICENSE file for licensing details.
-#
-# Learn more about testing at: https://juju.is/docs/sdk/testing
 
-import os
+import logging
+import socket
 import unittest
+from unittest.mock import patch
 
-from charm import TempoCoordinatorK8SOperatorCharm
-from ops.model import BlockedStatus
+from ops.model import ActiveStatus
 from ops.testing import Harness
 
+from charm import TempoCharm
 
-class TestCharm(unittest.TestCase):
+CONTAINER_NAME = "tempo"
+
+
+class TestTempoCharm(unittest.TestCase):
+    @patch("charm.KubernetesServicePatch", lambda x, y: None)
     def setUp(self):
-        os.environ["JUJU_VERSION"] = "3.0.3"
-        self.harness = Harness(TempoCoordinatorK8SOperatorCharm)
-        self.harness.set_can_connect("nginx", True)
+        self.harness = Harness(TempoCharm)
+        self.harness.set_model_name("testmodel")
         self.addCleanup(self.harness.cleanup)
+        self.harness.set_leader(True)
         self.harness.begin_with_initial_hooks()
+        self.maxDiff = None  # we're comparing big traefik configs in tests
+
+    def test_tempo_pebble_ready(self):
+        service = self._container.get_service("tempo")
+        self.assertTrue(service.is_running())
+        self.assertEqual(self.harness.model.unit.status, ActiveStatus())
+
+    def test_entrypoints_are_generated_with_sanitized_names(self):
+        expected_entrypoints = {
+            "entryPoints": {
+                "tempo-http": {"address": ":3200"},
+                "tempo-grpc": {"address": ":9096"},
+                "zipkin": {"address": ":9411"},
+                "otlp-grpc": {"address": ":4317"},
+                "otlp-http": {"address": ":4318"},
+                "jaeger-thrift-http": {"address": ":14268"},
+            }
+        }
+        self.assertEqual(self.harness.charm._static_ingress_config, expected_entrypoints)
+
+    def test_tracing_relation_updates_protocols_as_requested(self):
+        self.harness.set_leader(True)
+        self.harness.container_pebble_ready("tempo")
+
+        tracing_rel_id = self.harness.add_relation("tracing", "grafana")
+        self.harness.add_relation_unit(tracing_rel_id, "grafana/0")
+        self.harness.update_relation_data(
+            tracing_rel_id, "grafana", {"receivers": '["otlp_http"]'}
+        )
+
+        rel_data = self.harness.get_relation_data(tracing_rel_id, self.harness.charm.app.name)
+        logging.warning(rel_data)
+        self.assertEqual(
+            rel_data["receivers"],
+            f'[{{"protocol": {{"name": "otlp_http", "type": "http"}}, "url": "http://{socket.getfqdn()}:4318"}}]',
+        )
+
+    @property
+    def _container(self):
+        return self.harness.model.unit.get_container(CONTAINER_NAME)
 
-    def test_simple(self):
-        self.harness.evaluate_status()
-        self.assertIsInstance(self.harness.charm.unit.status, BlockedStatus)
+    @property
+    def _plan(self):
+        return self.harness.get_container_pebble_plan(CONTAINER_NAME)
diff --git a/tests/unit/test_tempo.py b/tests/unit/test_tempo.py
new file mode 100644
index 0000000..91596a6
--- /dev/null
+++ b/tests/unit/test_tempo.py
@@ -0,0 +1,50 @@
+from unittest.mock import patch
+
+import pytest
+
+from tempo import Tempo
+
+
+@pytest.mark.parametrize(
+    "protocols, expected_config",
+    (
+        (
+            (
+                "otlp_grpc",
+                "otlp_http",
+                "zipkin",
+                "tempo",
+                "jaeger_http_thrift",
+                "jaeger_grpc",
+                "jaeger_thrift_http",
+                "jaeger_thrift_http",
+            ),
+            {
+                "jaeger": {
+                    "protocols": {
+                        "grpc": None,
+                        "thrift_http": None,
+                    }
+                },
+                "zipkin": None,
+                "otlp": {"protocols": {"http": None, "grpc": None}},
+            },
+        ),
+        (
+            ("otlp_http", "zipkin", "tempo", "jaeger_thrift_http"),
+            {
+                "jaeger": {
+                    "protocols": {
+                        "thrift_http": None,
+                    }
+                },
+                "zipkin": None,
+                "otlp": {"protocols": {"http": None}},
+            },
+        ),
+        ([], {}),
+    ),
+)
+def test_tempo_receivers_config(protocols, expected_config):
+    with patch.object(Tempo, "tls_ready", False):
+        assert Tempo(None)._build_receivers_config(protocols) == expected_config
diff --git a/tox.ini b/tox.ini
index bbfeec4..cada980 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,20 +1,21 @@
-# Copyright 2023 Ubuntu
+# Copyright 2022 Canonical Ltd.
 # See LICENSE file for licensing details.
 
 [tox]
 skipsdist=True
 skip_missing_interpreters = True
-envlist = lint, static-{charm,lib}, unit, scenario
+envlist = lint, unit, scenario, static-charm, static-lib
 
 [vars]
 src_path = {toxinidir}/src/
 tst_path = {toxinidir}/tests/
-all_path = {[vars]src_path} {[vars]tst_path}
+lib_path = {toxinidir}/lib/charms/tempo_k8s
+all_path = {[vars]src_path} {[vars]tst_path} {[vars]lib_path}
 
 [testenv]
 setenv =
   PYTHONPATH = {toxinidir}:{toxinidir}/lib:{[vars]src_path}
-  PYTHONBREAKPOINT=pdb.set_trace
+  PYTHONBREAKPOINT=ipdb.set_trace
   PY_COLORS=1
 passenv =
   PYTHONPATH
@@ -26,95 +27,83 @@ description = Apply coding style standards to code
 deps =
     black
     ruff
+    isort
 commands =
+    isort {[vars]all_path}
     black {[vars]all_path}
-    ruff --fix {[vars]all_path}
+    ruff check {[vars]all_path} --fix
 
 [testenv:lint]
 description = Check code against coding style standards
 deps =
     black
     ruff
-    codespell
+    codespell<2.3.0 # https://github.com/codespell-project/codespell/issues/3430
 commands =
-    # uncomment the following line if this charm owns a lib
-    # codespell {[vars]lib_path}
-    codespell {toxinidir} \
-              --skip {toxinidir}/.git \
-              --skip {toxinidir}/.tox \
-              --skip {toxinidir}/build \
-              --skip {toxinidir}/prime \  # bug in charmcraft 2.5.3 does not clean up build dirs
-              --skip {toxinidir}/stage \
-              --skip {toxinidir}/parts \
-              --skip {toxinidir}/lib \
-              --skip {toxinidir}/venv \
-              --skip {toxinidir}/.mypy_cache \
-              --skip {toxinidir}/.ruff_cache \
-              --skip {toxinidir}/icon.svg \
-              --skip {toxinidir}/src/grafana_dashboards \
-              --skip {toxinidir}/tox.ini \
-              -L aNULL,anull
-
-    ruff {[vars]all_path}
+    codespell {[vars]all_path}
+    ruff check {[vars]all_path}
     black --check --diff {[vars]all_path}
 
-[testenv:static-{charm, lib}]
-description = Run static analysis checks
-deps =
-  pyright==1.1.344
-  -r{toxinidir}/requirements.txt
-commands =
-  charm: pyright {[vars]src_path} {posargs}
-
 [testenv:unit]
 description = Run unit tests
 deps =
-    pytest
+    pytest<8.2.0 # https://github.com/pytest-dev/pytest/issues/12263
     coverage[toml]
-    deepdiff
+    .[lib_pydeps]
     -r{toxinidir}/requirements.txt
-    
-    # Binary deps from from charmcraft.yaml
-    cryptography
-    jsonschema
 commands =
     coverage run --source={[vars]src_path} \
-                 -m pytest \
-                 --tb native \
-                 -v \
-                 -s \
-                 {[vars]tst_path}unit \
-                 {posargs}
+        -m pytest  -v --tb native -s {posargs} {[vars]tst_path}unit
     coverage report
 
 [testenv:scenario]
 description = Run scenario tests
 deps =
-    pytest
+    pytest<8.2.0 # https://github.com/pytest-dev/pytest/issues/12263
     coverage[toml]
-    ops-scenario>=3.0
+    ops-scenario>=4.0.3
+    .[lib_pydeps]
     -r{toxinidir}/requirements.txt
 commands =
     coverage run --source={[vars]src_path} \
-                 -m pytest \
-                 --tb native \
-                 -v \
-                 -s \
-                 {[vars]tst_path}scenario \
-                 {posargs}
+        -m pytest  -v --tb native -s {posargs} {[vars]tst_path}scenario
     coverage report
 
 [testenv:integration]
 description = Run integration tests
 deps =
-    pytest
-    juju
+    pytest<8.2.0 # https://github.com/pytest-dev/pytest/issues/12263
+    # see https://github.com/juju/python-libjuju/issues/1025
+    juju<=3.3.0,>=3.0
     pytest-operator
+    requests
+    -r{toxinidir}/requirements.txt
+    # tracegen
+    opentelemetry-exporter-otlp-proto-grpc
+commands =
+    pytest -v --tb native --log-cli-level=INFO {[vars]tst_path}integration -s {posargs}
+
+[testenv:interface]
+description = Run interface tests
+deps =
+    pytest<8.2.0 # https://github.com/pytest-dev/pytest/issues/12263
+    -r{toxinidir}/requirements.txt
+    .[lib_pydeps]
+    pytest-interface-tester
+    # git+https://github.com/canonical/pytest-interface-tester
+commands =
+    pytest -v --tb native --log-cli-level=INFO -s {posargs} {[vars]tst_path}interface
+
+
+[testenv:static-{charm,lib}]
+description = Static code checking
+deps =
+    # pydeps
+    .[lib_pydeps]
+    pyright==1.1.340
     -r{toxinidir}/requirements.txt
 commands =
-    pytest -v \
-           -s \
-           --tb native \
-           --log-cli-level=INFO \
-           {[vars]tst_path}integration \
-           {posargs}
+    charm: pyright --pythonversion 3.8 {[vars]src_path}
+    lib: pyright --pythonversion 3.8 {[vars]lib_path}
+    lib: /usr/bin/env sh -c 'for m in $(git diff main --name-only {[vars]lib_path}); do if git ls-tree -r HEAD --name-only | grep -q "^$m$"; then  if ! git diff main $m | grep -q "+LIBPATCH\|+LIBAPI"; then echo "You forgot to bump the version on $m!"; exit 1; fi; fi; done'
+allowlist_externals = /usr/bin/env