Move out alloy modules

Signed-off-by: Weifeng Wang <[email protected]> Move out alloy modules Signed-off-by: Weifeng Wang <[email protected]>
qclaogui · Apr 16, 2024 · e118700 · e118700
1 parent c994043
commit e118700
Show file tree

Hide file tree

Showing 39 changed files with 130 additions and 45 deletions.
diff --git a/README.md b/README.md
@@ -20,7 +20,7 @@ include: # include is available in Docker Compose version 2.20 and later, and Do
 
 services:
   github-exporter:
-    labels: # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    labels: # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
       metrics.grafana.com/scrape: true
     image: githubexporter/github-exporter:1.1.0
     environment:

diff --git a/alloy-modules/README.md b/alloy-modules/README.md
@@ -0,0 +1,9 @@
+# Alloy Modules
+
+[Modules](https://grafana.com/docs/alloy/latest/concepts/modules/) are a way to create Grafana [Alloy](https://grafana.com/docs/alloy/latest/) configurations which can be loaded as a component. Modules are a great way to parameterize a configuration to create reusable pipelines.
+
+## Modules
+
+- [Docker Compose](compose/)
+- [Kubernetes](kubernetes/)
+- [Grafana LGTMP Stack Provider](provider/)
diff --git a/...on/config/alloy/modules/compose/README.md → alloy-modules/compose/README.md b/...on/config/alloy/modules/compose/README.md → alloy-modules/compose/README.md
@@ -1,6 +1,17 @@
-# Docker Compose `labels_scrape` Components
+# Alloy Modules - Docker Compose
 
-## Logs
+## Modules
+
+- [Logs](logs/)
+- [Traces](traces/)
+- [Metrics](metrics/)
+- [Profiles](profiles/)
+- [Jobs](jobs/)
+- [Integrations](integrations/)
+
+## Compose `labels_scrape` Component for [`Logs`](logs/labels-scrape.alloy), [`Metrics`](metrics/labels-scrape.alloy) and [`Profiles`](profiles/labels-scrape.alloy)
+
+### Logs
 
 The following service labels are supported:
 
@@ -11,7 +22,7 @@ The following service labels are supported:
 
 ---
 
-## Metrics
+### Metrics
 
 The following service labels are supported for gathering of metrics for docker compose services:
 
@@ -28,7 +39,7 @@ The following service labels are supported for gathering of metrics for docker c
 
 ---
 
-## Profiles
+### Profiles
 
 The following service labels are supported for docker compose services:
 

diff --git a/alloy-modules/compose/integrations/README.md b/alloy-modules/compose/integrations/README.md
@@ -0,0 +1,9 @@
+# Integrations Components
+
+Docker Compose Integrations Components
+
+## Components
+
+- [cadvisor](cadvisor.alloy)
+- [memcached](memcached.alloy)
+- [node_exporter](node_exporter.alloy)
diff --git a/...dules/compose/integrations/cadvisor.alloy → ...dules/compose/integrations/cadvisor.alloy b/...dules/compose/integrations/cadvisor.alloy → ...dules/compose/integrations/cadvisor.alloy
diff --git a/...ules/compose/integrations/memcached.alloy → ...ules/compose/integrations/memcached.alloy b/...ules/compose/integrations/memcached.alloy → ...ules/compose/integrations/memcached.alloy
diff --git a/.../compose/integrations/node-exporter.alloy → .../compose/integrations/node-exporter.alloy b/.../compose/integrations/node-exporter.alloy → .../compose/integrations/node-exporter.alloy
diff --git a/...ig/alloy/modules/compose/jobs/minio.alloy → alloy-modules/compose/jobs/minio.alloy b/...ig/alloy/modules/compose/jobs/minio.alloy → alloy-modules/compose/jobs/minio.alloy
diff --git a/...oy/modules/compose/logs/keep-labels.alloy → alloy-modules/compose/logs/keep-labels.alloy b/...oy/modules/compose/logs/keep-labels.alloy → alloy-modules/compose/logs/keep-labels.alloy
diff --git a/.../modules/compose/logs/labels-scrape.alloy → ...-modules/compose/logs/labels-scrape.alloy b/.../modules/compose/logs/labels-scrape.alloy → ...-modules/compose/logs/labels-scrape.alloy
diff --git a/...compose/metrics/integrations-scrape.alloy → ...compose/metrics/integrations-scrape.alloy b/...compose/metrics/integrations-scrape.alloy → ...compose/metrics/integrations-scrape.alloy
diff --git a/...modules/compose/metrics/jobs-scrape.alloy → ...modules/compose/metrics/jobs-scrape.alloy b/...modules/compose/metrics/jobs-scrape.alloy → ...modules/compose/metrics/jobs-scrape.alloy
diff --git a/...odules/compose/metrics/labels-probe.alloy → ...odules/compose/metrics/labels-probe.alloy b/...odules/compose/metrics/labels-probe.alloy → ...odules/compose/metrics/labels-probe.alloy
diff --git a/...dules/compose/metrics/labels-scrape.alloy → ...dules/compose/metrics/labels-scrape.alloy b/...dules/compose/metrics/labels-scrape.alloy → ...dules/compose/metrics/labels-scrape.alloy
diff --git a/...ules/compose/profiles/labels-scrape.alloy → ...ules/compose/profiles/labels-scrape.alloy b/...ules/compose/profiles/labels-scrape.alloy → ...ules/compose/profiles/labels-scrape.alloy
diff --git a/...ompose/traces/process-and-transform.alloy → ...ompose/traces/process-and-transform.alloy b/...ompose/traces/process-and-transform.alloy → ...ompose/traces/process-and-transform.alloy
diff --git a/alloy-modules/kubernetes/README.md b/alloy-modules/kubernetes/README.md
@@ -0,0 +1,12 @@
+# Alloy Modules - Kubernetes(WIP)
+
+## Modules
+
+- [Logs](logs/)
+- [Traces](traces/)
+- [Metrics](metrics/)
+- [Profiles](profiles/)
+- [Jobs](jobs/)
+- [Integrations](integrations/)
+
+## Kubernetes `annotations_scrape` Component for [`Logs`](logs/annotations-scrape.alloy), [`Metrics`](metrics/annotations-scrape.alloy) and [`Profiles`](profiles/annotations-scrape.alloy)
diff --git a/...n/config/alloy/modules/provider/README.md → alloy-modules/provider/README.md b/...n/config/alloy/modules/provider/README.md → alloy-modules/provider/README.md
@@ -3,27 +3,26 @@
 Provide the receivers of the collected data(`logs` `metrics` `traces` `profiles`).
 
 ## Components
-
-- [self_hosted_stack](#self_hosted_stack)
-- [grafana_cloud](#grafana_cloud)
+  - [`self_hosted_stack`](#self_hosted_stack)
+  - [`grafana_cloud`](#grafana_cloud)
 
 ### `self_hosted_stack`
 
 Module to configure receivers for Self Hosted LGTMP Stack.
 
 ***Arguments***
 
-| Name                    | Required | Default                                 | Description                        |
-| :---------------------- | :------- | :-------------------------------------- | :--------------------------------- |
-| `metrics_endpoint_url`  | _no_     | `http://mimir:8080/api/v1/push`         | Where to send collected `metrics`. |
-| `logs_endpoint_url`     | _no_     | `http://loki:3100/loki/api/v1/push`     | Where to send collected `logs`.    |
-| `traces_endpoint_url`   | _no_     | `http://tempo:4318`                     | Where to send collected `traces`.  |
-| `profiles_endpoint_url` | _no_     | `http://pyroscope:4040`                 | Where to send collected `profiles`.|
+| Name                    | Required | Default                             | Description                         |
+|:------------------------|:---------|:------------------------------------|:------------------------------------|
+| `metrics_endpoint_url`  | _no_     | `http://mimir:8080/api/v1/push`     | Where to send collected `metrics`.  |
+| `logs_endpoint_url`     | _no_     | `http://loki:3100/loki/api/v1/push` | Where to send collected `logs`.     |
+| `traces_endpoint_url`   | _no_     | `http://tempo:4318`                 | Where to send collected `traces`.   |
+| `profiles_endpoint_url` | _no_     | `http://pyroscope:4040`             | Where to send collected `profiles`. |
 
 ***Exports***
 
-| Name                |        Type              | Description                                                      |
-| --------------------| ------------------------ | ---------------------------------------------------------------- |
+| Name                | Type                     | Description                                                      |
+|---------------------|--------------------------|------------------------------------------------------------------|
 | `metrics_receiver`  | `prometheus.Interceptor` | A value that other components can use to send metrics data to.   |
 | `logs_receiver`     | `loki.LogsReceiver`      | A value that other components can use to send logs data to.      |
 | `traces_receiver`   | `otelcol.Consumer`       | A value that other components can use to send trace data to.     |
@@ -35,8 +34,8 @@ Module to configure receivers for Self Hosted LGTMP Stack.
 import.git "provider" {
   repository     = "https://github.com/qclaogui/codelab-monitoring.git"
   revision       = "main"
-  path           = "docker-compose/common/config/alloy/modules/provider"
-  pull_frequency = "15m"
+  path           = "alloy-modules/provider"
+  pull_frequency = "24h"
 }
 
 // get the receivers from provider
@@ -80,18 +79,18 @@ The token must have permissions to read stack information. The setup of these pe
 ***Arguments***
 
 | Name         | Required | Default | Description                                        |
-| :----------- | :------- | :------ | :------------------------------------------------- |
+|:-------------|:---------|:--------|:---------------------------------------------------|
 | `stack_name` | _yes_    | `N/A`   | Name of your stack as shown in the account console |
 | `token`      | _yes_    | `N/A`   | Access policy token or API Key.                    |
 
 ***Exports***
 
-| Name                |        Type              | Description                                                                 |
-| --------------------| ------------------------ | --------------------------------------------------------------------------- |
-| `metrics_receiver`  | `prometheus.Interceptor` | A value that other components can use to send metrics data to.              |
-| `logs_receiver`     | `loki.LogsReceiver`      | A value that other components can use to send logs data to.                 |
-| `traces_receiver`   | `otelcol.Consumer`       | A value that other components can use to send trace data to.                |
-| `profiles_receiver` | `write.fanOutClient`     | A value that other components can use to send profiling data to.            |
+| Name                | Type                     | Description                                                                                                                  |
+|---------------------|--------------------------|------------------------------------------------------------------------------------------------------------------------------|
+| `metrics_receiver`  | `prometheus.Interceptor` | A value that other components can use to send metrics data to.                                                               |
+| `logs_receiver`     | `loki.LogsReceiver`      | A value that other components can use to send logs data to.                                                                  |
+| `traces_receiver`   | `otelcol.Consumer`       | A value that other components can use to send trace data to.                                                                 |
+| `profiles_receiver` | `write.fanOutClient`     | A value that other components can use to send profiling data to.                                                             |
 | `stack_information` | `object`                 | Decoded representation of the [Stack info endpoint](https://grafana.com/docs/grafana-cloud/api-reference/cloud-api/#stacks). |
 
 ***Example***
@@ -100,8 +99,8 @@ The token must have permissions to read stack information. The setup of these pe
 import.git "provider" {
   repository     = "https://github.com/qclaogui/codelab-monitoring.git"
   revision       = "main"
-  path           = "docker-compose/common/config/alloy/modules/provider"
-  pull_frequency = "15m"
+  path           = "alloy-modules/provider"
+  pull_frequency = "24h"
 }
 
 // get the receivers from provider

diff --git a/...lloy/modules/provider/grafana_cloud.alloy → alloy-modules/provider/grafana_cloud.alloy b/...lloy/modules/provider/grafana_cloud.alloy → alloy-modules/provider/grafana_cloud.alloy
diff --git a/.../alloy/modules/provider/self_hosted.alloy → alloy-modules/provider/self_hosted.alloy b/.../alloy/modules/provider/self_hosted.alloy → alloy-modules/provider/self_hosted.alloy
@@ -1,6 +1,6 @@
 /************************************************
 * Component: self_hosted_stack
-* Description: Self Hosteded Receiver Provider
+* Description: Self Hosted Receiver Provider
 *************************************************/
 
 declare "self_hosted_stack" {

diff --git a/compose.override.yaml b/compose.override.yaml
@@ -1,5 +1,5 @@
 
-# https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+# https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
 x-labels: &profiles-labels
   profiles.grafana.com/cpu.scrape: true
   profiles.grafana.com/memory.scrape: true

diff --git a/docker-compose/common/compose-include/alloy.yaml b/docker-compose/common/compose-include/alloy.yaml
@@ -6,7 +6,7 @@ services:
       - source: alloy_config_file
         target: /etc/alloy/config.alloy
     volumes:
-      - ../config/alloy/modules:/etc/alloy/modules
+      - ../../../alloy-modules:/etc/alloy/modules
       - /var/run/docker.sock:/var/run/docker.sock:ro
       - /var/lib/docker:/var/lib/docker:ro
       - /:/rootfs:ro

diff --git a/docker-compose/common/compose-include/minio.yaml b/docker-compose/common/compose-include/minio.yaml
@@ -1,7 +1,7 @@
 
 services:
   minio:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       logs.grafana.com/scrape: false
       metrics.grafana.com/scrape: true

diff --git a/docker-compose/common/config/alloy/all-in-one.alloy b/docker-compose/common/config/alloy/all-in-one.alloy
@@ -6,6 +6,15 @@ logging {
 /********************************************
  * Grafana LGTMP Stack Receiver Provider
  ********************************************/
+// // git remote
+// import.git "provider" {
+// 	repository     = "https://github.com/qclaogui/codelab-monitoring.git"
+// 	revision       = "main"
+// 	path           = "alloy-modules/provider"
+// 	pull_frequency = "24h"
+// }
+
+// local file
 import.file "provider" {
 	filename = coalesce(env("ALLOY_MODULES_FOLDER"), "/etc/alloy/modules") + "/provider"
 }

diff --git a/docker-compose/common/config/alloy/logs.alloy b/docker-compose/common/config/alloy/logs.alloy
@@ -6,6 +6,15 @@ logging {
 /********************************************
  * Grafana LGTMP Stack Receiver Provider
  ********************************************/
+// // git remote
+// import.git "provider" {
+// 	repository     = "https://github.com/qclaogui/codelab-monitoring.git"
+// 	revision       = "main"
+// 	path           = "alloy-modules/provider"
+// 	pull_frequency = "24h"
+// }
+
+// local file
 import.file "provider" {
 	filename = coalesce(env("ALLOY_MODULES_FOLDER"), "/etc/alloy/modules") + "/provider"
 }

diff --git a/docker-compose/common/config/alloy/metrics.alloy b/docker-compose/common/config/alloy/metrics.alloy
@@ -6,6 +6,15 @@ logging {
 /********************************************
  * Grafana LGTMP Stack Receiver Provider
  ********************************************/
+// // git remote
+// import.git "provider" {
+// 	repository     = "https://github.com/qclaogui/codelab-monitoring.git"
+// 	revision       = "main"
+// 	path           = "alloy-modules/provider"
+// 	pull_frequency = "24h"
+// }
+
+// local file
 import.file "provider" {
 	filename = coalesce(env("ALLOY_MODULES_FOLDER"), "/etc/alloy/modules") + "/provider"
 }

diff --git a/docker-compose/common/config/alloy/profiles.alloy b/docker-compose/common/config/alloy/profiles.alloy
@@ -6,6 +6,15 @@ logging {
 /********************************************
  * Grafana LGTMP Stack Receiver Provider
  ********************************************/
+// // git remote
+// import.git "provider" {
+// 	repository     = "https://github.com/qclaogui/codelab-monitoring.git"
+// 	revision       = "main"
+// 	path           = "alloy-modules/provider"
+// 	pull_frequency = "24h"
+// }
+
+// local file
 import.file "provider" {
 	filename = coalesce(env("ALLOY_MODULES_FOLDER"), "/etc/alloy/modules") + "/provider"
 }

diff --git a/docker-compose/common/config/alloy/traces.alloy b/docker-compose/common/config/alloy/traces.alloy
@@ -6,6 +6,15 @@ logging {
 /********************************************
  * Grafana LGTMP Stack Receiver Provider
  ********************************************/
+// // git remote
+// import.git "provider" {
+// 	repository     = "https://github.com/qclaogui/codelab-monitoring.git"
+// 	revision       = "main"
+// 	path           = "alloy-modules/provider"
+// 	pull_frequency = "24h"
+// }
+
+// local file
 import.file "provider" {
 	filename = coalesce(env("ALLOY_MODULES_FOLDER"), "/etc/alloy/modules") + "/provider"
 }

diff --git a/docker-compose/microservices-mode/logs/compose.yaml b/docker-compose/microservices-mode/logs/compose.yaml
@@ -14,7 +14,7 @@ include:
   - path: ../../common/compose-include/grafana.yaml
   - path: ../../common/compose-include/alloy.yaml
 
-# https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+# https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
 x-labels: &loki-labels
   logs.grafana.com/log-format: json
 
@@ -31,7 +31,7 @@ configs:
 
 services:
   gateway:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: false
       logs.grafana.com/log-format: nginx

diff --git a/docker-compose/microservices-mode/metrics/compose.yaml b/docker-compose/microservices-mode/metrics/compose.yaml
@@ -25,7 +25,7 @@ configs:
 
 services:
   gateway:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       - metrics.grafana.com/scrape=false
     depends_on: { distributor: { condition: service_healthy }, ruler: { condition: service_healthy } }

diff --git a/docker-compose/microservices-mode/profiles/compose.yaml b/docker-compose/microservices-mode/profiles/compose.yaml
@@ -14,7 +14,7 @@ include:
   - path: ../../common/compose-include/grafana.yaml
   - path: ../../common/compose-include/alloy.yaml
 
-# https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+# https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
 x-labels: &profiles-labels
   profiles.grafana.com/cpu.scrape: true
   profiles.grafana.com/memory.scrape: true
@@ -33,7 +33,7 @@ configs:
 
 services:
   gateway:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: false
     depends_on: { distributor: { condition: service_healthy } }

diff --git a/docker-compose/microservices-mode/traces/compose.yaml b/docker-compose/microservices-mode/traces/compose.yaml
@@ -34,7 +34,7 @@ configs:
 
 services:
   gateway:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: false
     depends_on: { distributor: { condition: service_healthy } }

diff --git a/docker-compose/monolithic-mode/all-in-one/compose.yaml b/docker-compose/monolithic-mode/all-in-one/compose.yaml
@@ -13,7 +13,7 @@ include:
   - path: ../../common/compose-include/grafana.yaml
   - path: ../../common/compose-include/alloy.yaml
 
-# https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+# https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
 x-labels: &profiles-labels
   profiles.grafana.com/cpu.scrape: true
   profiles.grafana.com/memory.scrape: true
@@ -136,7 +136,7 @@ services:
           - mimir-memberlist
 
   mimirtool:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: false
     image: ${MIMIRTOOL_IMAGE:-docker.io/grafana/mimirtool:2.12.0}

diff --git a/docker-compose/read-write-mode/logs/compose.yaml b/docker-compose/read-write-mode/logs/compose.yaml
@@ -27,7 +27,7 @@ configs:
 
 services:
   gateway:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: false
     depends_on: { loki-write: { condition: service_healthy } }

diff --git a/docker-compose/read-write-mode/metrics/compose.yaml b/docker-compose/read-write-mode/metrics/compose.yaml
@@ -25,7 +25,7 @@ configs:
 
 services:
   gateway:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: false
     depends_on: { mimir-write: { condition: service_healthy }, mimir-backend: { condition: service_healthy } }

diff --git a/examples/ghe-with-mixin/compose.yaml b/examples/ghe-with-mixin/compose.yaml
@@ -22,7 +22,7 @@ include:
 
 services:
   github-exporter:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: true
     image: githubexporter/github-exporter:1.1.0

diff --git a/examples/ghe-with-simple-dashboards/compose.yaml b/examples/ghe-with-simple-dashboards/compose.yaml
@@ -22,7 +22,7 @@ include:
 
 services:
   github-exporter:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: true
     image: githubexporter/github-exporter:1.1.0

diff --git a/examples/github-exporter/compose.yaml b/examples/github-exporter/compose.yaml
@@ -24,7 +24,7 @@ include:
 
 services:
   github-exporter:
-    # https://github.com/qclaogui/codelab-monitoring/blob/main/docker-compose/common/config/alloy/modules/compose/README.md
+    # https://github.com/qclaogui/codelab-monitoring/blob/main/alloy-modules/compose/README.md
     labels:
       metrics.grafana.com/scrape: true
     image: githubexporter/github-exporter:1.1.0