From 3cf3229d87e86417b5a5ac24347809d1fc7d400e Mon Sep 17 00:00:00 2001 From: Shivam Sandbhor Date: Mon, 22 Aug 2022 12:01:33 +0530 Subject: [PATCH 1/2] Document bouncer metrics --- .../docs/bouncers/blocklist-mirror.mdx | 47 ++++++++++++++++++- crowdsec-docs/docs/bouncers/cloudflare.mdx | 45 ++++++++++++++++++ crowdsec-docs/docs/bouncers/firewall.mdx | 47 +++++++++++++++++++ 3 files changed, 138 insertions(+), 1 deletion(-) diff --git a/crowdsec-docs/docs/bouncers/blocklist-mirror.mdx b/crowdsec-docs/docs/bouncers/blocklist-mirror.mdx index 6832f669..ae0ce50d 100644 --- a/crowdsec-docs/docs/bouncers/blocklist-mirror.mdx +++ b/crowdsec-docs/docs/bouncers/blocklist-mirror.mdx @@ -81,7 +81,7 @@ cd crowdsec-blocklist-mirror-v*/ sudo ./install.sh ``` -# Configuration +## Configuration Before starting the `crowdsec-blocklist-mirror` service, please edit the configuration file to add your API URL and key. The default configuration file is located under : `/etc/crowdsec/bouncers/` @@ -246,3 +246,48 @@ Example: 1.2.3.4 4.3.2.1 ``` + +## Observability + +The bouncer by default exposes the metrics at /metrics . `listen_uri` is where the blocklist is served. + +The exposed metrics are: + +- `lapi_requests_total` : This is the count of total calls made to the CrowdSec LAPI by bouncer. +- `lapi_requests_failures_total` : This is the total number of failed calls to CrowdSec LAPI. +- `blocklist_requests_total`: This is counter vector for each route denoting total requests made. +- `active_decision_count`: This is the current count of total number IPs served by the blocklist mirror. + +### Alerting when LAPI is unreachable + +The metrics can be used to receive alerts if the bouncer can't reach LAPI via prometheus. The following prometheus rule can be used. + +```yaml +# rules.yml +groups: +- name: lapi + rules: + - alert: LAPIFailure + expr: delta(lapi_requests_failures_total[15s]) > 0 + for: 5m +``` + +In prometheus alertmanager you can then handle as needed. A sample alertmanger config which posts a simple message on slack when LAPI request fails would look like: + +```yaml +# alertmanager.yml +global: + resolve_timeout: 1m + slack_api_url: '' + +route: + receiver: 'lapi' + +receivers: + - name: 'lapi' + + slack_configs: + - channel: '#crowdsec-notifications' + send_resolved: true + text: Bouncer can't reach LAPI +``` \ No newline at end of file diff --git a/crowdsec-docs/docs/bouncers/cloudflare.mdx b/crowdsec-docs/docs/bouncers/cloudflare.mdx index 7c57064e..5c21ec05 100644 --- a/crowdsec-docs/docs/bouncers/cloudflare.mdx +++ b/crowdsec-docs/docs/bouncers/cloudflare.mdx @@ -275,6 +275,51 @@ Example Usage: sudo crowdsec-cloudflare-bouncer -d ``` +## Observability + +The bouncer by default exposes the metrics at 127.0.0.1:2112/metrics . + +The exposed metrics are: + +- `lapi_requests_total` : This is the count of total calls made to the CrowdSec LAPI by bouncer. +- `lapi_requests_failures_total` : This is the total number of failed calls to CrowdSec LAPI. +- `cloudflare_api_calls`: This is the count of total cloudflare API calls. +- `response_time`: Histogram of cloudflare API call response time. + +### Alerting when LAPI is unreachable + +The metrics can be used to receive alerts if the bouncer can't reach LAPI via prometheus. The following prometheus rule can be used. + +```yaml +# rules.yml +groups: +- name: lapi + rules: + - alert: LAPIFailure + expr: delta(lapi_requests_failures_total[15s]) > 0 + for: 5m +``` + +In prometheus alertmanager you can then handle as needed. A sample alertmanger config which posts a simple message on slack when LAPI request fails would look like: + +```yaml +# alertmanager.yml +global: + resolve_timeout: 1m + slack_api_url: '' + +route: + receiver: 'lapi' + +receivers: + - name: 'lapi' + + slack_configs: + - channel: '#crowdsec-notifications' + send_resolved: true + text: Bouncer can't reach LAPI +``` + ## How it works The service polls the CrowdSec Local API for new decisions. It then makes API calls to Cloudflare diff --git a/crowdsec-docs/docs/bouncers/firewall.mdx b/crowdsec-docs/docs/bouncers/firewall.mdx index 7cd9895d..ab5dd875 100644 --- a/crowdsec-docs/docs/bouncers/firewall.mdx +++ b/crowdsec-docs/docs/bouncers/firewall.mdx @@ -205,6 +205,10 @@ table ip6 crowdsec6 { - `deny_action` : firewall action to apply, defaults to `DROP`, but can be `REJECT` - `deny_log` : if set to `true`, enables logging of dropped packets (ie. `-j LOG`) - `deny_log_prefix` : if logging is true, this sets the log prefix, defaults to "crowdsec: " + - `prometheus.enabled`: set to true to serve metrics + - `prometheus.listen_addr`: address to serve metrics on + - `prometheus.listen_port`: port to serve metrics on + ### Iptables/Ipset specific directives @@ -240,6 +244,49 @@ nftables: if `set-only` is set to true, the bouncer will only manage the set contents. +## Observability + +The bouncer by default exposes the metrics at 127.0.0.1:60601/metrics . + +The exposed metrics are: + +- `lapi_requests_total` : This is the count of total calls made to the CrowdSec LAPI by bouncer. +- `lapi_requests_failures_total` : This is the total number of failed calls to CrowdSec LAPI. + +### Alerting when LAPI is unreachable + +The metrics can be used to receive alerts if the bouncer can't reach LAPI via prometheus. The following prometheus rule can be used. + +```yaml +# rules.yml +groups: +- name: lapi + rules: + - alert: LAPIFailure + expr: delta(lapi_requests_failures_total[15s]) > 0 + for: 5m +``` + +In prometheus alertmanager you can then handle as needed. A sample alertmanger config which posts a simple message on slack when LAPI request fails would look like: + +```yaml +# alertmanager.yml +global: + resolve_timeout: 1m + slack_api_url: '' + +route: + receiver: 'lapi' + +receivers: + - name: 'lapi' + + slack_configs: + - channel: '#crowdsec-notifications' + send_resolved: true + text: Bouncer can't reach LAPI +``` + ## Manual installation ### Assisted From a57fda13fcc2733fd2374519d9c21206b6488966 Mon Sep 17 00:00:00 2001 From: Shivam Sandbhor Date: Wed, 31 Aug 2022 14:59:46 +0530 Subject: [PATCH 2/2] Document dropped packet metric Signed-off-by: Shivam Sandbhor --- crowdsec-docs/docs/bouncers/firewall.mdx | 3 +++ 1 file changed, 3 insertions(+) diff --git a/crowdsec-docs/docs/bouncers/firewall.mdx b/crowdsec-docs/docs/bouncers/firewall.mdx index ab5dd875..ced2a8e6 100644 --- a/crowdsec-docs/docs/bouncers/firewall.mdx +++ b/crowdsec-docs/docs/bouncers/firewall.mdx @@ -252,6 +252,9 @@ The exposed metrics are: - `lapi_requests_total` : This is the count of total calls made to the CrowdSec LAPI by bouncer. - `lapi_requests_failures_total` : This is the total number of failed calls to CrowdSec LAPI. +- `dropped_packets_via_cs_rule_total` : This is the total number of packets which were dropped due to rule and IP sets created by the bouncer. +- `dropped_bytes_via_cs_rule_total` : This is the total number of bytes which were dropped due to rule and IP sets created by the bouncer. + ### Alerting when LAPI is unreachable