Add tls overlay #80

sed-i · 2023-08-14T20:02:13Z

This PR adds a TLS overlay.

Fixes #78.

Depends on:

Testing

First, deploy the testing overlay:

$ tox -e render-edge
$ juju deploy ./bundle.yaml --trust \
  --overlay overlays/tls-overlay.yaml \
  --overlay overlays/testing-overlay.yaml

We do not have the receive-ca-cert relation yet in cos charms, so the external ca would need to be copied in manually:

juju run external-ca/0 get-ca-certificate --format json | jq -r '."external-ca/0".results."ca-certificate"' > external-ca.crt
openssl x509 -noout -text -in external-ca.crt

echo | openssl s_client -showcerts -servername 10.43.8.206 -connect 10.43.8.206:443 | openssl x509 -text -noout

juju scp --container prometheus external-ca.crt prometheus/0:/usr/local/share/ca-certificates
juju ssh --container prometheus prometheus/0 update-ca-certificates --fresh

juju scp --container grafana external-ca.crt grafana/0:/usr/local/share/ca-certificates
juju ssh --container grafana grafana/0 update-ca-certificates --fresh

Then:

Attempt to manually curl each cos component: juju show-unit catalogue/0 | grep url.
Make sure all scrape targets are healthy: curl -k https://10.43.8.206/bndl-prometheus-0/api/v1/targets | jq | grep -E "scrapeUrl|health".
Make sure the watchdog alert from avalanche is firing:
- In prometheus: curl <prometheus>/api/v1/alerts.
- In alertmanager: curl <alertmanager>/api/v2/alerts.
Make sure grafana has all the dashboards.

lucabello · 2023-08-23T10:04:57Z

I just tried deploying the bundle with the TLS overlay after the Catalogue issue has been closed, and I can conferm that specific problem is solved; Catalogue is now idle/active 🎉

lucabello · 2023-08-23T10:24:07Z

Everything eventually settles to active idle, which is good.

As per the other test conditions:

✅ I can curl the listed components (prometheus, grafana, alertmanager); ❌ I cannot however curl catalogue, as I get Bad Gateway on both HTTP and HTTPS; (turns out I can curl catalogue directly, however not through traefik; it seems like we don't really use the traefik-provided URL in catalogue at all)
✅ All the scrape targets in Prometheus are healthy
❌ There's no alerts in that alertmanager endpoint; however, prometheus has some (two) which are related to avalanche (AlwaysFiringDueToAbsentMetric/DueToNumericValue); not sure if there's supposed to be an actual Watchdog alert like there is for alertmanager;
❓ Cannot check this due to the unreachable web UI bug

overlays/tls-overlay.yaml

overlays/testing-overlay.yaml

lucabello · 2023-08-24T08:08:12Z

Following up on my previous comment:

The error when curling catalogue was due to us using an outdated image for the workload, so that's solved
The alerts missing from Alertmanager are caused from this bug: Race in alertmanager configuration alertmanager-k8s-operator#189

lucabello · 2023-08-29T07:27:43Z

@sed-i The race in alertmanager configuration issue could be fixed now (as my guess is that this PR solved it).

You should try deploying it and check! :D

PietroPasotti · 2023-09-07T14:14:44Z

interestingly, loki metrics report down because of a None in the url:

no alert seems to be firing after adding avalanche

PietroPasotti · 2023-09-07T14:25:51Z

something wrong with loki's metrics-endpoint

PietroPasotti · 2023-09-07T14:47:47Z

update for posterity: apparently fixed in edge

PietroPasotti · 2023-09-12T13:13:57Z

for my own future benefit:

curl -k https://$TRAEFIK_IP/$MODEL_NAME-prometheus-0/api/v1/targets | jq .status # 'success'
curl -k https://$TRAEFIK_IP/$MODEL_NAME-alertmanager/api/v2/alerts | jq .status # []
curl -k https://192.168.0.105/tls-bundle3-catalogue # ??? Bad Gateway
curl -k https://192.168.0.105/tls-bundle3-grafana # <a href="/tls-bundle3-grafana/login">Found</a>.

PietroPasotti · 2023-09-12T13:49:58Z

result of my day of sweat:

after deploy, all datasources in grafana are dead
I add to /etc/hosts: <traefik ip> traefik.local and `juju config traefik external_hostname="traefik.local"
log into grafana; when testing the datasources I get:
- some error telling me it can't route to traefik
- I juju ssh to grafana and add to /etc/hosts: <traefik ip> traefik.local
- error changes to: Error reading Prometheus: Post "https://traefik.local/api/v1/query": x509: certificate is valid for 5c7c9fdf4db7d6d9269c72bc1453af14.33c28df6f153744b27cff01be4a94ca2.traefik.default, not traefik.local
no alerts are visible in the alertmanager ui

sed-i · 2023-09-13T21:38:47Z

Deployed just now,

$ tox -e render-edge
$ juju deploy ./bundle.yaml --trust \
  --overlay overlays/tls-overlay.yaml \
  --overlay overlays/testing-overlay.yaml

And noticed a few issues with the scrape targets:

curl -sk https://10.43.8.206/bndl-prometheus-0/api/v1/targets | jq | grep -E "scrapeUrl|health"
        "scrapeUrl": "http://10.43.8.206:80/bndl-alertmanager/metrics",
        "health": "down",
        "scrapeUrl": "http://10.1.166.74:9001/metrics",
        "health": "up",
        "scrapeUrl": "http://10.1.166.82:9001/metrics",
        "health": "up",
        "scrapeUrl": "https://grafana-0.grafana-endpoints.bndl.svc.cluster.local:3000/metrics",
        "health": "up",
        "scrapeUrl": "http://10.43.8.206:80/bndl-loki-0/metrics",
        "health": "down",
        "scrapeUrl": "http://traefik-0.traefik-endpoints.bndl.svc.cluster.local:8082/metrics",
        "health": "up",
        "scrapeUrl": "https://prometheus-0.prometheus-endpoints.bndl.svc.cluster.local:9090/metrics",
        "health": "up",

Alertmanager scrape URL has http instead of https, and the port is :80 instead of blank (or :443). (Manually curling https://10.43.8.206/bndl-alertmanager/metrics works.)
Avalanche is not related to traefik, but it is reporting unit IP instead of fqdn. (Low priority)
Loki scrape URL has http instead of https, and the port is :80 instead of blank (or :443). In addition, loki tells traefik over relation data that it's http instead of https, so the metrics endpoint isn't reachable via ingress URL at all.

sed-i · 2023-09-14T04:44:07Z

Heads up, @PietroPasotti:

I added another relation to the tls overlay between traefik and a new charm, "external-ca". That's how it should be.
The problem is that it made things worse.
If before, when traefik was related to ca:certificates, curl prometheus worked, now it returns Internal Server Error%.
Need to figure out why.

PietroPasotti · 2023-09-15T12:04:38Z

seems to go away after adding the https redirect block to traefik config and restarting a couple of times

PietroPasotti · 2023-09-19T09:56:30Z

results from testing with traefik's [fix-tls branch](https://github.com/canonical/traefik-k8s-o perator/pull/245#event-10405698061):

whole bundle goes active/idle 💯
all metrics up except loki: 🤔

        "globalUrl": "https://192.168.0.105:443/clite-tls-test-loki-0/metrics",                                                               
        "lastError": "Get \"https://192.168.0.105:443/clite-tls-test-loki-0/metrics\": x509: cannot validate certificate for 192.168.0.105 bec
ause it doesn't contain any IP SANs",                                                                                                         
        "lastScrape": "2023-09-19T09:40:23.227981499Z",                                                                                       
        "lastScrapeDuration": 0.002484311,                                                                                                    
        "health": "down",

prometheus shows avalance alert 👌
alertmanager does not: 💥

❯ curl -k  https://192.168.0.105/clite-tls-test-alertmanager/api/v2/alerts   
[]

after copying over the cert as described in traefik's pr description:

sed-i · 2023-09-19T14:28:42Z

Yeah, IP for SAN doesn't work yet, I suspect because of canonical/tls-certificates-interface#71.
With external hostname such as trfk.local curl worked for me.

tests/integration/test_bundle.py

+            *"run external-ca/0 get-ca-certificate --format=json --no-color".split()
+        )
+        cert = json.loads(stdout)["external-ca/0"]["results"]["ca-certificate"]
+        cert_path.write_text(cert)


Abuelodelanada

After deploying the bundle as indicated in the testing instructions, I have the following:

$ jst                                                            
Model     Controller  Cloud/Region        Version  SLA          Timestamp
cos-lite  microk8s    microk8s/localhost  3.1.5    unsupported  15:36:51-03:00

App           Version  Status  Scale  Charm                     Channel  Rev  Address         Exposed  Message
alertmanager  0.25.0   active      2  alertmanager-k8s          edge      92  10.152.183.146  no       
avalanche              active      2  avalanche-k8s             edge      35  10.152.183.89   no       
ca                     active      1  self-signed-certificates  edge      37  10.152.183.75   no       
catalogue              active      1  catalogue-k8s             edge      27  10.152.183.204  no       
external-ca            active      1  self-signed-certificates  edge      37  10.152.183.217  no       
grafana       9.2.1    active      1  grafana-k8s               edge      92  10.152.183.218  no       
loki          2.7.4    active      1  loki-k8s                  edge      99  10.152.183.38   no       
prometheus    2.46.0   active      1  prometheus-k8s            edge     150  10.152.183.123  no       
traefik       2.10.4   active      1  traefik-k8s               edge     157  192.168.1.250   no       

Unit             Workload  Agent  Address      Ports  Message
alertmanager/0   active    idle   10.1.38.87          
alertmanager/1*  active    idle   10.1.38.98          
avalanche/0      active    idle   10.1.38.76          
avalanche/1*     active    idle   10.1.38.122         
ca/0*            active    idle   10.1.38.78          
catalogue/0*     active    idle   10.1.38.80          
external-ca/0*   active    idle   10.1.38.102         
grafana/0*       active    idle   10.1.38.109         
loki/0*          active    idle   10.1.38.71          
prometheus/0*    active    idle   10.1.38.70          
traefik/0*       active    idle   10.1.38.73          

Relation provider                   Requirer                     Interface              Type     Message
alertmanager:alerting               loki:alertmanager            alertmanager_dispatch  regular  
alertmanager:alerting               prometheus:alertmanager      alertmanager_dispatch  regular  
alertmanager:grafana-dashboard      grafana:grafana-dashboard    grafana_dashboard      regular  
alertmanager:grafana-source         grafana:grafana-source       grafana_datasource     regular  
alertmanager:replicas               alertmanager:replicas        alertmanager_replica   peer     
alertmanager:self-metrics-endpoint  prometheus:metrics-endpoint  prometheus_scrape      regular  
avalanche:metrics-endpoint          prometheus:metrics-endpoint  prometheus_scrape      regular  
avalanche:replicas                  avalanche:replicas           avalanche_replica      peer     
ca:certificates                     alertmanager:certificates    tls-certificates       regular  
ca:certificates                     catalogue:certificates       tls-certificates       regular  
ca:certificates                     grafana:certificates         tls-certificates       regular  
ca:certificates                     loki:certificates            tls-certificates       regular  
ca:certificates                     prometheus:certificates      tls-certificates       regular  
ca:send-ca-cert                     traefik:receive-ca-cert      certificate_transfer   regular  
catalogue:catalogue                 alertmanager:catalogue       catalogue              regular  
catalogue:catalogue                 grafana:catalogue            catalogue              regular  
catalogue:catalogue                 prometheus:catalogue         catalogue              regular  
catalogue:replicas                  catalogue:replicas           catalogue_replica      peer     
external-ca:certificates            traefik:certificates         tls-certificates       regular  
grafana:grafana                     grafana:grafana              grafana_peers          peer     
grafana:metrics-endpoint            prometheus:metrics-endpoint  prometheus_scrape      regular  
grafana:replicas                    grafana:replicas             grafana_replicas       peer     
loki:grafana-dashboard              grafana:grafana-dashboard    grafana_dashboard      regular  
loki:grafana-source                 grafana:grafana-source       grafana_datasource     regular  
loki:metrics-endpoint               prometheus:metrics-endpoint  prometheus_scrape      regular  
loki:replicas                       loki:replicas                loki_replica           peer     
prometheus:grafana-dashboard        grafana:grafana-dashboard    grafana_dashboard      regular  
prometheus:grafana-source           grafana:grafana-source       grafana_datasource     regular  
prometheus:prometheus-peers         prometheus:prometheus-peers  prometheus_peers       peer     
traefik:ingress                     alertmanager:ingress         ingress                regular  
traefik:ingress                     catalogue:ingress            ingress                regular  
traefik:ingress-per-unit            loki:ingress                 ingress_per_unit       regular  
traefik:ingress-per-unit            prometheus:ingress           ingress_per_unit       regular  
traefik:metrics-endpoint            prometheus:metrics-endpoint  prometheus_scrape      regular  
traefik:peers                       traefik:peers                traefik_peers          peer     
traefik:traefik-route               grafana:ingress              traefik_route          regular

Then:

Attempt to manually curl each cos component: juju show-unit catalogue/0 | grep url.

╭─ubuntu@charm-dev-juju-31 ~/repos/cos-lite-bundle ‹feature/tls-overlay●› [microk8s:cos-lite]
╰─$ for i in $(juju show-unit catalogue/0 | grep url: | awk '{print $2}'); do echo $i; done
https://192.168.1.250/cos-lite-catalogue
https://192.168.1.250/cos-lite-grafana
https://192.168.1.250/cos-lite-prometheus-0
http://192.168.1.250/cos-lite-alertmanager
╭─ubuntu@charm-dev-juju-31 ~/repos/cos-lite-bundle ‹feature/tls-overlay●› [microk8s:cos-lite]
╰─$ for i in $(juju show-unit catalogue/0 | grep url: | awk '{print $2}'); do echo "-->" $i: `curl -o /dev/null -sk -w '%{http_code}' $i`; done
--> https://192.168.1.250/cos-lite-catalogue: 502
--> https://192.168.1.250/cos-lite-grafana: 302
--> https://192.168.1.250/cos-lite-prometheus-0: 302
--> https://192.168.1.250/cos-lite-alertmanager: 200

Catalogue URL is not working
Alertmanager is using HTTP instead of HTTPS
Grafana and Prometheus works OK

Make sure all scrape targets are healthy: curl -k https://10.43.8.206/bndl-prometheus-0/api/v1/targets | jq | grep -E "scrapeUrl|health".

Everything is OK:

$ curl -sk https://192.168.1.250/cos-lite-prometheus-0/api/v1/targets | jq | grep -E "scrapeUrl|health"
        "scrapeUrl": "https://alertmanager-1.alertmanager-endpoints.cos-lite.svc.cluster.local:9093/metrics",
        "health": "up",
        "scrapeUrl": "http://10.1.38.76:9001/metrics",
        "health": "up",
        "scrapeUrl": "http://10.1.38.122:9001/metrics",
        "health": "up",
        "scrapeUrl": "https://grafana-0.grafana-endpoints.cos-lite.svc.cluster.local:3000/metrics",
        "health": "up",
        "scrapeUrl": "https://loki-0.loki-endpoints.cos-lite.svc.cluster.local:3100/metrics",
        "health": "up",
        "scrapeUrl": "http://traefik-0.traefik-endpoints.cos-lite.svc.cluster.local:8082/metrics",
        "health": "up",
        "scrapeUrl": "https://prometheus-0.prometheus-endpoints.cos-lite.svc.cluster.local:9090/metrics",
        "health": "up",

Make sure the watchdog alert from avalanche is firing:

In prometheus: curl /api/v1/alerts.

In alertmanager: curl /api/v2/alerts.

Alerts are firing:

Prometheus:

Alertmanager:

Make sure grafana has all the dashboards.

Dashboards in Grafana:

sed-i · 2023-10-03T19:08:22Z

Confirmed findings by @Abuelodelanada:

After https is enabled in am & trfk, catalogue still has http for am.
Curling the catalogue ingress url (with https) gives "bad gateway" (catalogue has a certs relation in place).

Abuelodelanada · 2023-10-06T19:42:01Z

@sed-i Manually tested again. Everything works OK!!

Add tls overlay

b33fad9

sed-i mentioned this pull request Aug 19, 2023

Charm is in blocked status, "Failed to restart Catalogue" canonical/catalogue-k8s-operator#31

Closed

simskij approved these changes Aug 23, 2023

View reviewed changes

overlays/tls-overlay.yaml Outdated Show resolved Hide resolved

overlays/testing-overlay.yaml Outdated Show resolved Hide resolved

Apply suggestions from code review

3100251

sed-i added 4 commits September 8, 2023 10:36

Use juju 3.1/stable

5cc9826

Fix tests

489c680

Scale down am to 1 unit

ccb1863

Limit matrix test to juju 3.x

810ddcd

sed-i mentioned this pull request Sep 8, 2023

Fix handling of tls config canonical/prometheus-k8s-operator#524

Merged

3 tasks

Relate traefik to external CA

f4ad144

Abuelodelanada mentioned this pull request Sep 14, 2023

TLS errors with Prometheus scraping. canonical/loki-k8s-operator#305

Closed

sed-i mentioned this pull request Sep 15, 2023

TLS fixes canonical/traefik-k8s-operator#245

Merged

Set distinct common names to both CAs

f57874a

sed-i mentioned this pull request Sep 19, 2023

Split SANs into DNS and IP canonical/observability-libs#60

Merged

This was referenced Sep 20, 2023

Uncaught exception ops.pebble.ConnectionError canonical/traefik-k8s-operator#248

Closed

FIX scrape target and certificate created for loki canonical/loki-k8s-operator#308

Merged

sed-i added 2 commits September 23, 2023 00:41

Lint

19694c0

Fix itest/CI

72ef7ff

sed-i mentioned this pull request Sep 25, 2023

Replace juju 2.9 with 3.1 in matrix.yaml #81

Closed

sed-i added 4 commits September 27, 2023 14:26

Fix itest

2cdf42d

Add missing dep

c341610

Drop sh dep

416dc3c

Fix e2e test

1a4a3a6

github-advanced-security bot found potential problems Sep 28, 2023

View reviewed changes

Lint

3a0c4a8

This was referenced Sep 29, 2023

The workload is on https, but traefik still thinks it's http canonical/prometheus-k8s-operator#533

Closed

Update ingress after cert is written canonical/prometheus-k8s-operator#534

Merged

sed-i added 2 commits September 29, 2023 01:16

Fix itest

e0dc9a2

Fix itest

09a35b3

sed-i mentioned this pull request Sep 29, 2023

cannot start service: fork/exec /bin/go-runner: no such file or directory jnsgruk/zinc-k8s-operator#207

Closed

sed-i added 3 commits October 2, 2023 15:08

Fix itest

288e5fd

Lint

6d37d50

CI updates

3133870

sed-i marked this pull request as ready for review October 3, 2023 15:53

sed-i requested review from Abuelodelanada, lucabello, PietroPasotti and dstathis as code owners October 3, 2023 15:53

Abuelodelanada reviewed Oct 3, 2023

View reviewed changes

sed-i mentioned this pull request Oct 3, 2023

The refresh_event has no effect because CatalogueItem doens't take a callable for url canonical/catalogue-k8s-operator#41

Closed

Abuelodelanada approved these changes Oct 6, 2023

View reviewed changes

Use safer hostname

1bc6f7c

sed-i merged commit fd3297b into main Oct 7, 2023

sed-i deleted the feature/tls-overlay branch October 7, 2023 00:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tls overlay #80

Add tls overlay #80

sed-i commented Aug 14, 2023 •

edited

Loading

lucabello commented Aug 23, 2023

lucabello commented Aug 23, 2023 •

edited

Loading

lucabello commented Aug 24, 2023

lucabello commented Aug 29, 2023

PietroPasotti commented Sep 7, 2023

PietroPasotti commented Sep 7, 2023

PietroPasotti commented Sep 7, 2023

PietroPasotti commented Sep 12, 2023

PietroPasotti commented Sep 12, 2023 •

edited

Loading

sed-i commented Sep 13, 2023

sed-i commented Sep 14, 2023

PietroPasotti commented Sep 15, 2023

PietroPasotti commented Sep 19, 2023

sed-i commented Sep 19, 2023

Abuelodelanada left a comment •

edited

Loading

sed-i commented Oct 3, 2023 •

edited

Loading

Abuelodelanada commented Oct 6, 2023

Add tls overlay #80

Add tls overlay #80

Conversation

sed-i commented Aug 14, 2023 • edited Loading

Testing

lucabello commented Aug 23, 2023

lucabello commented Aug 23, 2023 • edited Loading

lucabello commented Aug 24, 2023

lucabello commented Aug 29, 2023

PietroPasotti commented Sep 7, 2023

PietroPasotti commented Sep 7, 2023

PietroPasotti commented Sep 7, 2023

PietroPasotti commented Sep 12, 2023

PietroPasotti commented Sep 12, 2023 • edited Loading

sed-i commented Sep 13, 2023

sed-i commented Sep 14, 2023

PietroPasotti commented Sep 15, 2023

PietroPasotti commented Sep 19, 2023

sed-i commented Sep 19, 2023

Abuelodelanada left a comment • edited Loading

Choose a reason for hiding this comment

Then:

sed-i commented Oct 3, 2023 • edited Loading

Abuelodelanada commented Oct 6, 2023

sed-i commented Aug 14, 2023 •

edited

Loading

lucabello commented Aug 23, 2023 •

edited

Loading

PietroPasotti commented Sep 12, 2023 •

edited

Loading

Abuelodelanada left a comment •

edited

Loading

sed-i commented Oct 3, 2023 •

edited

Loading