Add LogForwarder to coordinator

[mmkay]

Issue

Although coordinator was passing loki endpoints to workers, it wasn't sending its own workload logs.

Solution

Add a LogForwarder instance for sending nginx logs.

Context

Testing Instructions

Pack one of the coordinator charms (I tested with tempo-coordinator-k8s) with the following changes:

requirements.txt:

# lib/charms/prometheus_k8s/v0/prometheus_scrape.py
cosl@git+https://github.com/canonical/cos-lib.git@coordinator-log-forwarder

charmcraft.yaml:

parts:
  charm:
    build-packages:
      - "git"

Then deploy the following bundle:

bundle: kubernetes
applications:
  alertmanager:
    charm: alertmanager-k8s
    channel: latest/edge
    revision: 135
    base: [email protected]/stable
    resources:
      alertmanager-image: 98
    scale: 1
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
    trust: true
  catalogue:
    charm: catalogue-k8s
    channel: latest/edge
    revision: 63
    base: [email protected]/stable
    resources:
      catalogue-image: 34
    scale: 1
    options:
      description: "Canonical Observability Stack Lite, or COS Lite, is a light-weight,
        highly-integrated, \nJuju-based observability suite running on Kubernetes.\n"
      tagline: Model-driven Observability Stack deployed with a single command.
      title: Canonical Observability Stack
    constraints: arch=amd64
    trust: true
  grafana:
    charm: grafana-k8s
    channel: latest/edge
    revision: 119
    base: [email protected]/stable
    resources:
      grafana-image: 70
      litestream-image: 45
    scale: 1
    constraints: arch=amd64
    storage:
      database: kubernetes,1,1024M
    trust: true
  loki:
    charm: loki-k8s
    channel: latest/edge
    revision: 172
    base: [email protected]/stable
    resources:
      loki-image: 100
      node-exporter-image: 3
    scale: 1
    constraints: arch=amd64
    storage:
      active-index-directory: kubernetes,1,1024M
      loki-chunks: kubernetes,1,1024M
    trust: true
  minio:
    charm: minio
    channel: latest/edge
    revision: 376
    base: [email protected]/stable
    resources:
      oci-image: 545
    scale: 1
    options:
      access-key: accesskey
      secret-key: secretkey
    constraints: arch=amd64
    storage:
      minio-data: kubernetes,1,10240M
    trust: true
  prometheus:
    charm: prometheus-k8s
    channel: latest/edge
    revision: 212
    base: [email protected]/stable
    resources:
      prometheus-image: 150
    scale: 1
    constraints: arch=amd64
    storage:
      database: kubernetes,1,1024M
    trust: true
  s3:
    charm: s3-integrator
    channel: latest/edge
    revision: 59
    scale: 1
    options:
      bucket: tempo
      endpoint: minio-0.minio-endpoints.test-nginx-logs.svc.cluster.local:9000
    constraints: arch=amd64
    trust: true
  tempo-coordinator-k8s:
    charm: local:tempo-coordinator-k8s-1
    scale: 1
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
    trust: true
  tempo-worker-k8s:
    charm: tempo-worker-k8s
    channel: latest/edge
    revision: 34
    resources:
      tempo-image: 3
    scale: 1
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
    trust: true
  traefik:
    charm: traefik-k8s
    channel: latest/edge
    revision: 211
    base: [email protected]/stable
    resources:
      traefik-image: 161
    scale: 1
    constraints: arch=amd64
    storage:
      configurations: kubernetes,1,1024M
    trust: true
relations:
- - traefik:ingress-per-unit
  - prometheus:ingress
- - traefik:ingress-per-unit
  - loki:ingress
- - traefik:traefik-route
  - grafana:ingress
- - traefik:ingress
  - alertmanager:ingress
- - prometheus:alertmanager
  - alertmanager:alerting
- - grafana:grafana-source
  - prometheus:grafana-source
- - grafana:grafana-source
  - loki:grafana-source
- - grafana:grafana-source
  - alertmanager:grafana-source
- - loki:alertmanager
  - alertmanager:alerting
- - prometheus:metrics-endpoint
  - traefik:metrics-endpoint
- - prometheus:metrics-endpoint
  - alertmanager:self-metrics-endpoint
- - prometheus:metrics-endpoint
  - loki:metrics-endpoint
- - prometheus:metrics-endpoint
  - grafana:metrics-endpoint
- - grafana:grafana-dashboard
  - loki:grafana-dashboard
- - grafana:grafana-dashboard
  - prometheus:grafana-dashboard
- - grafana:grafana-dashboard
  - alertmanager:grafana-dashboard
- - catalogue:ingress
  - traefik:ingress
- - catalogue:catalogue
  - grafana:catalogue
- - catalogue:catalogue
  - prometheus:catalogue
- - catalogue:catalogue
  - alertmanager:catalogue
- - catalogue:catalogue
  - loki:catalogue
- - loki:logging
  - traefik:logging
- - minio:grafana-dashboard
  - grafana:grafana-dashboard
- - minio:metrics-endpoint
  - prometheus:metrics-endpoint
- - traefik:grafana-dashboard
  - grafana:grafana-dashboard
- - loki:logging
  - tempo-coordinator-k8s:logging
- - prometheus:receive-remote-write
  - tempo-coordinator-k8s:send-remote-write
- - s3:s3-credentials
  - tempo-coordinator-k8s:s3
- - tempo-coordinator-k8s:tracing
  - alertmanager:tracing
- - tempo-coordinator-k8s:tracing
  - catalogue:tracing
- - tempo-coordinator-k8s:grafana-dashboard
  - grafana:grafana-dashboard
- - tempo-coordinator-k8s:grafana-source
  - grafana:grafana-source
- - tempo-coordinator-k8s:tracing
  - grafana:tracing
- - tempo-coordinator-k8s:tracing
  - loki:tracing
- - tempo-coordinator-k8s:metrics-endpoint
  - prometheus:metrics-endpoint
- - tempo-coordinator-k8s:tracing
  - prometheus:tracing
- - tempo-coordinator-k8s:tempo-cluster
  - tempo-worker-k8s:tempo-cluster
- - tempo-coordinator-k8s:tracing
  - traefik:tracing
- - traefik:traefik-route
  - tempo-coordinator-k8s:ingress

Observe that nginx logs appear in Loki.

Upgrade Notes

[michaeldmitry]

changes look good to me