Chainlink Prometheus/Grafana TLS monitoring

Indroduction

This documentation is a guide for full Chainlink node monitoring and alerting including the following deployments:

Prometheus server with TLS & basic-auth
Prometheus node exporter with TLS & basic-auth
Grafana with TLS & basic-auth
Loki & Promtail
Full monitoring Chainlink dashboard
Alerts and Telegram notification setup

Comment

For the example deployment we used the created "Kovan" docker network. Every container needs to be in the same network like the Chainlink node to ensure the communication between them.
For the creation of the files we used nano, you can also do it with vim
You need to copy the files from this repository to your system. Just copy the source code after you've created the file by following the guide.

Create directories

Create the directories for all necessary files

mkdir ~/.monitoring
mkdir ~/.monitoring/.tls
mkdir ~/.monitoring/.tls/.prometheus
mkdir ~/.monitoring/.tls/.grafana
mkdir ~/.monitoring/.tls/.node-exporter

TLS certificates

The TLS certificates are created via openssl and stored in the created directories

Prometheus

cd ~/.monitoring/.tls/.prometheus && openssl req -x509 -out   ~/.monitoring/.tls/.prometheus/prometheus.crt  -keyout  ~/.monitoring/.tls/.prometheus/prometheus.key -newkey rsa:2048 -nodes -sha256 -days 365 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

Node exporter

cd ~/.monitoring/.tls/.node-exporter && openssl req -x509 -out   ~/.monitoring/.tls/.node-exporter/node-exporter.crt  -keyout  ~/.monitoring/.tls/.node-exporter/node-exporter.key -newkey rsa:2048 -nodes -sha256 -days 365 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

Grafana

cd ~/.monitoring/.tls/.grafana && openssl req -x509 -out   ~/.monitoring/.tls/.grafana/grafana.crt  -keyout  ~/.monitoring/.tls/.grafana/grafana.key -newkey rsa:2048 -nodes -sha256 -days 365 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

Authentication

A .htpasswd file is used for the protection of the Prometheus credentials using HTTP authentication and is implemented into a .htaccess file.

Install HTPASSWD

yum install httpd-tools

or

sudo apt-get install httpd-tools

Prometheus auth

htpasswd -nBC 10 "" | tr -d ':\n'

you need to save this value for the prometheusweb.yml

Node exporter auth

htpasswd -nBC 10 "" | tr -d ':\n'

you need to save this value for the exporterweb.yml

Node exporter

Create web.yml

cd ~/.monitoring && nano exporterweb.yml

copy the code of the exportweb.yml and just change the username of the basic-auth and the HTPASWD token

Run node-exporter

cd ~/.monitoring && docker run -d -p 9100:9100 --name node-exporter --restart unless-stopped --network kovan --user root -v "/:/hostfs" -v /home/<USER>/.monitoring/exporterweb.yml:/hostfs/web.yml -v /home/<USER>/.monitoring/.tls/.node-exporter/node-exporter.key:/tls/node-exporter.key -v /home/<USER>/.monitoring/.tls/.node-exporter/node-exporter.crt:/tls/node-exporter.crt prom/node-exporter --path.rootfs=/hostfs --web.config=/hostfs/web.yml

You need to change the to your user name in order to gain access. This will point the initialisation to the created and required files and directories.

Prometheus server

Create web.yml

cd ~/.monitoring && nano prometheusweb.yml

Create prometheus.yml

cd ~/.monitoring && nano prometheus.yml

Run prometheus-server

cd ~/.monitoring && sudo docker run --name prometheus --network kovan --restart=unless-stopped --user root -d -p 9090:9090 -v /home/<USER>/.monitoring/prometheus.yml:/etc/prometheus/prometheus.yml -v /home/<USER>/.monitoring/.tls/.prometheus/prometheus.key:/tls/prometheus.key -v /home/<USER>/.monitoring/.tls/.prometheus/prometheus.crt:/tls/prometheus.crt -v /home/<USER>/.monitoring/prometheusweb.yml:/etc/prometheus/web.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml --web.config.file=/etc/prometheus/web.yml

You need to change the to your user name in order to gain access. This will point the initialisation to the created and required files and directories.

To check if Prometheus scrapes all metrics, you need to check your targets in the Prometheus GUI: https://localhost:9090/targets

Loki

Create loki.yml

cd ~/.monitoring && nano loki.yml

Run Loki

cd ~/.monitoring && sudo docker run -d -p 3100:3100 --name loki --network kovan --restart unless-stopped -v /home/<USER>/.monitoring/loki.yml:/mnt/config/loki.yml grafana/loki:2.2.0 -config.file=/mnt/config/loki.yml

Promtail

Create promtail.yml

cd ~/.monitoring && nano promtail.yml

Run promtail

cd ~/.monitoring && sudo docker run -d --name promtail --network kovan --restart unless-stopped --user root -v /home/<USER>/.monitoring/promtail.yml:/mnt/config/promtail.yml -v /var/log:/var/log -v /var/lib/docker:/var/lib/docker grafana/promtail:2.2.0 -config.file=/mnt/config/promtail.yml

Grafana

Create default.ini

cd ~/.monitoring && nano grafana.ini

Run Grafana

cd ~/.monitoring && docker run -d -p 3000:3000 --name grafana --network kovan --restart unless-stopped --user root -v /home/<USER>/.monitoring/.tls/.grafana/grafana.key:/tls/grafana.key -v /home/<USER>/.monitoring/.tls/.grafana/grafana.crt:/tls/grafana.crt -v /home/<USER>/.monitoring/grafana.ini:/etc/grafana/grafana.ini -e GF_PATHS_CONFIG=/etc/grafana/grafana.ini grafana/grafana:latest

Data source Integration

Open your Grafana GUI in your explorer https://localhost:3000
Fill in your username and password
Add a new data source

Prometheus

Target: https://<PROMETHEUS_CONTAINER_ID>:9090
Enable: Basic_Auth, Credentials , CA_Cert, Skip_TLS_VERIFY
SAVE & TEST

LOKI

Target: http://<LOKI_CONTAINER_ID>:3100
SAVE & TEST

Grafana dashboards

Chainlink dashboard:

Click on Create -> Import
Import via panel JSON
Paste the JSON code from this repo's dashboard.file

Host dashboard:

Click on Create -> Import
Import via grafana.com
Type in: 11952
https://grafana.com/grafana/dashboards/11952

Alerting

Create a notification channel:

"Alerting" -> "Notification channels" -> "Add channel"
Name: Telegram
Type: Telegram
BOT API TOKEN: You need to create A BOT API Token https://medium.com/shibinco/create-a-telegram-bot-using-botfather-and-get-the-api-token-900ba00e0f39
Chat ID: get the chat ID of your Telegram channel

Alerting

You can now set alerts on your dashboard. You can only set alerts on "graph-visualisations" as displayed metrics.

For a list of important alerts for a running Chainlink node you can have a look at our security research paper: https://linkriver.io/wp-content/uploads/2021/03/Chainlink_Node_Operations_Research_Paper.pdf

Files

README.md

Latest commit

History

README.md

File metadata and controls

Chainlink Prometheus/Grafana TLS monitoring

Indroduction

Comment

Create directories

TLS certificates

Prometheus

Node exporter

Grafana

Authentication

Install HTPASSWD

Prometheus auth

Node exporter auth

Node exporter

Create web.yml

Run node-exporter

Prometheus server

Create web.yml

Create prometheus.yml

Run prometheus-server

Loki

Create loki.yml

Run Loki

Promtail

Create promtail.yml

Run promtail

Grafana

Create default.ini

Run Grafana

Data source Integration

Prometheus

LOKI

Grafana dashboards

Chainlink dashboard:

Host dashboard:

Alerting

Create a notification channel:

Alerting