[DONT MERGE]Document bouncer metrics

crowdsec-docs/docs/bouncers/blocklist-mirror.mdx

@@ -81,7 +81,7 @@ cd crowdsec-blocklist-mirror-v*/
 sudo ./install.sh
 ```
 
-# Configuration
+## Configuration
 
 Before starting the `crowdsec-blocklist-mirror` service, please edit the configuration file to add your API URL and key.
 The default configuration file is located under : `/etc/crowdsec/bouncers/`
@@ -246,3 +246,48 @@ Example:
 1.2.3.4
 4.3.2.1
 ```
+
+## Observability
+
+The bouncer by default exposes the metrics at <listen_uri>/metrics . `listen_uri` is where the blocklist is served.
+
+The exposed metrics are:
+
+- `lapi_requests_total` : This is the count of total calls made to the CrowdSec LAPI by bouncer.
+- `lapi_requests_failures_total` : This is the total number of failed calls to CrowdSec LAPI.
+- `blocklist_requests_total`: This is counter vector for each route denoting total requests made.
+- `active_decision_count`: This is the current count of total number IPs served by the blocklist mirror.
+
+### Alerting when LAPI is unreachable
+
+The metrics can be used to receive alerts if the bouncer can't reach LAPI via prometheus. The following prometheus rule can be used.
+
+```yaml
+# rules.yml
+groups:
+- name: lapi
+  rules:
+  - alert: LAPIFailure
+    expr: delta(lapi_requests_failures_total[15s]) > 0
+    for: 5m
+```
+
+In prometheus alertmanager you can then handle as needed. A sample alertmanger config which posts a simple message on slack when LAPI request fails would look like:
+
+```yaml
+# alertmanager.yml
+global:
+  resolve_timeout: 1m
+  slack_api_url: '<your_slack_webhook_url>'
+
+route:
+  receiver: 'lapi'
+
+receivers:
+  - name: 'lapi'
+
+    slack_configs:
+    - channel: '#crowdsec-notifications' 
+      send_resolved: true
+      text: Bouncer can't reach LAPI
+```

crowdsec-docs/docs/bouncers/cloudflare.mdx

@@ -275,6 +275,51 @@ Example Usage:
 sudo crowdsec-cloudflare-bouncer -d 
 ```
 
+## Observability
+
+The bouncer by default exposes the metrics at 127.0.0.1:2112/metrics .
+
+The exposed metrics are:
+
+- `lapi_requests_total` : This is the count of total calls made to the CrowdSec LAPI by bouncer.
+- `lapi_requests_failures_total` : This is the total number of failed calls to CrowdSec LAPI.
+- `cloudflare_api_calls`: This is the count of total cloudflare API calls.
+- `response_time`: Histogram of cloudflare API call response time.
+
+### Alerting when LAPI is unreachable
+
+The metrics can be used to receive alerts if the bouncer can't reach LAPI via prometheus. The following prometheus rule can be used.
+
+```yaml
+# rules.yml
+groups:
+- name: lapi
+  rules:
+  - alert: LAPIFailure
+    expr: delta(lapi_requests_failures_total[15s]) > 0
+    for: 5m
+```
+
+In prometheus alertmanager you can then handle as needed. A sample alertmanger config which posts a simple message on slack when LAPI request fails would look like:
+
+```yaml
+# alertmanager.yml
+global:
+  resolve_timeout: 1m
+  slack_api_url: '<your_slack_webhook_url>'
+
+route:
+  receiver: 'lapi'
+
+receivers:
+  - name: 'lapi'
+
+    slack_configs:
+    - channel: '#crowdsec-notifications' 
+      send_resolved: true
+      text: Bouncer can't reach LAPI
+```
+
 ## How it works
 
 The service polls the CrowdSec Local API for new decisions. It then makes API calls to Cloudflare

crowdsec-docs/docs/bouncers/firewall.mdx

@@ -205,6 +205,10 @@ table ip6 crowdsec6 {
  - `deny_action` : firewall action to apply, defaults to `DROP`, but can be `REJECT`
  - `deny_log` : if set to `true`, enables logging of dropped packets (ie. `-j LOG`)
  - `deny_log_prefix` : if logging is true, this sets the log prefix, defaults to "crowdsec: "
+ - `prometheus.enabled`: set to true to serve metrics
+ - `prometheus.listen_addr`: address to serve metrics on
+ - `prometheus.listen_port`: port to serve metrics on
+
 
 ### Iptables/Ipset specific directives
 
@@ -240,6 +244,52 @@ nftables:
 
 if `set-only` is set to true, the bouncer will only manage the set contents.
 
+## Observability
+
+The bouncer by default exposes the metrics at 127.0.0.1:60601/metrics .
+
+The exposed metrics are:
+
+- `lapi_requests_total` : This is the count of total calls made to the CrowdSec LAPI by bouncer.
+- `lapi_requests_failures_total` : This is the total number of failed calls to CrowdSec LAPI.
+- `dropped_packets_via_cs_rule_total` : This is the total number of packets which were dropped due to rule and IP sets created by the bouncer.
+- `dropped_bytes_via_cs_rule_total` : This is the total number of bytes which were dropped due to rule and IP sets created by the bouncer. 
+
+
+### Alerting when LAPI is unreachable
+
+The metrics can be used to receive alerts if the bouncer can't reach LAPI via prometheus. The following prometheus rule can be used.
+
+```yaml
+# rules.yml
+groups:
+- name: lapi
+  rules:
+  - alert: LAPIFailure
+    expr: delta(lapi_requests_failures_total[15s]) > 0
+    for: 5m
+```
+
+In prometheus alertmanager you can then handle as needed. A sample alertmanger config which posts a simple message on slack when LAPI request fails would look like:
+
+```yaml
+# alertmanager.yml
+global:
+  resolve_timeout: 1m
+  slack_api_url: '<your_slack_webhook_url>'
+
+route:
+  receiver: 'lapi'
+
+receivers:
+  - name: 'lapi'
+
+    slack_configs:
+    - channel: '#crowdsec-notifications' 
+      send_resolved: true
+      text: Bouncer can't reach LAPI
+```
+
 ## Manual installation
 
 ### Assisted