Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ssh exporter #2138

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ Main (unreleased)

- Add `otelcol.receiver.solace` component to receive traces from a Solace broker. (@wildum)

- Added `prometheus.exporter.ssh` custom metrics via ssh for remote hosts. (@EHSchmitt4395)

### Enhancements

- Add second metrics sample to the support bundle to provide delta information (@dehaansa)
Expand All @@ -33,7 +35,7 @@ Main (unreleased)

- Fixed an issue in the `otelcol.processor.attribute` component where the actions `delete` and `hash` could not be used with the `pattern` argument. (@wildum)

- Fixed a race condition that could lead to a deadlock when using `import` statements, which could lead to a memory leak on `/metrics` endpoint of an Alloy instance. (@thampiotr)
- Fixed a race condition that could lead to a deadlock when using `import` statements, which could lead to a memory leak on `/metrics` endpoint of an Alloy instance. (@thampiotr)

### Other changes

Expand Down Expand Up @@ -86,7 +88,7 @@ v1.5.0
- Add support for relative paths to `import.file`. This new functionality allows users to use `import.file` blocks in modules
imported via `import.git` and other `import.file`. (@wildum)

- `prometheus.exporter.cloudwatch`: The `discovery` block now has a `recently_active_only` configuration attribute
- `prometheus.exporter.cloudwatch`: The `discovery` block now has a `recently_active_only` configuration attribute
to return only metrics which have been active in the last 3 hours.

- Add Prometheus bearer authentication to a `prometheus.write.queue` component (@freak12techno)
Expand All @@ -99,9 +101,9 @@ v1.5.0

- Fixed a bug in `import.git` which caused a `"non-fast-forward update"` error message. (@ptodev)

- Do not log error on clean shutdown of `loki.source.journal`. (@thampiotr)
- Do not log error on clean shutdown of `loki.source.journal`. (@thampiotr)

- `prometheus.operator.*` components: Fixed a bug which would sometimes cause a
- `prometheus.operator.*` components: Fixed a bug which would sometimes cause a
"failed to create service discovery refresh metrics" error after a config reload. (@ptodev)

### Other changes
Expand Down Expand Up @@ -140,7 +142,7 @@ v1.4.3

- `pyroscope.scrape` no longer tries to scrape endpoints which are not active targets anymore. (@wildum @mattdurham @dehaansa @ptodev)

- Fixed a bug with `loki.source.podlogs` not starting in large clusters due to short informer sync timeout. (@elburnetto-intapp)
- Fixed a bug with `loki.source.podlogs` not starting in large clusters due to short informer sync timeout. (@elburnetto-intapp)

- `prometheus.exporter.windows`: Fixed bug with `exclude` regular expression config arguments which caused missing metrics. (@ptodev)

Expand All @@ -159,7 +161,7 @@ v1.4.2
- Fix parsing of the Level configuration attribute in debug_metrics config block
- Ensure "optional" debug_metrics config block really is optional

- Fixed an issue with `loki.process` where `stage.luhn` and `stage.timestamp` would not apply
- Fixed an issue with `loki.process` where `stage.luhn` and `stage.timestamp` would not apply
default configuration settings correctly (@thampiotr)

- Fixed an issue with `loki.process` where configuration could be reloaded even if there
Expand Down
1 change: 1 addition & 0 deletions docs/sources/reference/compatibility/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ The following components, grouped by namespace, _export_ Targets.
- [prometheus.exporter.snmp](../components/prometheus/prometheus.exporter.snmp)
- [prometheus.exporter.snowflake](../components/prometheus/prometheus.exporter.snowflake)
- [prometheus.exporter.squid](../components/prometheus/prometheus.exporter.squid)
- [prometheus.exporter.ssh](../components/prometheus/prometheus.exporter.ssh)
- [prometheus.exporter.statsd](../components/prometheus/prometheus.exporter.statsd)
- [prometheus.exporter.unix](../components/prometheus/prometheus.exporter.unix)
- [prometheus.exporter.windows](../components/prometheus/prometheus.exporter.windows)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
---
canonical: https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.exporter.ssh/
aliases:
- ../prometheus.exporter.ssh/ # /docs/alloy/latest/reference/components/prometheus.exporter.ssh/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aliases are not need for this component because it's brand new (this was used for mapping after the structure of the doc was changed)

description: Learn about prometheus.exporter.ssh
title: prometheus.exporter.ssh
---

# prometheus.exporter.ssh

The `prometheus.exporter.ssh` component embeds an SSH exporter for collecting metrics from remote servers over SSH and exporting them as Prometheus metrics.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add the experimental snippet to document the fact that the component is experimental

## Usage

```
prometheus.exporter.ssh "LABEL" {
// Configuration options
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Configuration options
targets {
...
}

All the mandatory arguments and blocks need to be shown in the Usage section.

}
```

## Arguments

The following arguments can be used to configure the exporter's behavior.
All arguments are optional unless specified. Omitted fields take their default values.
Comment on lines +23 to +24
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Observation: In reviewing this I've noticed that the wording of some sections in the Prometheus topics are not consistent with the rest of the components. For example, Arguments in other sections simply state: The following arguments are supported:

I'll create an Issue to follow up on this. No action or suggestions for this PR.


| Name | Type | Description | Default | Required |
| ----------------- | -------- | -------------------------------------------------- | ------- | -------- |
| `verbose_logging` | `bool` | Enable verbose logging for debugging purposes. | `false` | no |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need such a argument? Normally users would configure the logging level through the logging block.

| `targets` | `block` | One or more target configurations for SSH metrics. | | yes |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `targets` | `block` | One or more target configurations for SSH metrics. | | yes |

We don't list blocks in the table in the Arguments section. They are listed in the table in the Blocks section.


## Blocks

The following blocks are supported inside the definition of `prometheus.exporter.ssh`:

| Block | Description | Required |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add/include the Hierarchy column in this table?

| -------------- | ----------------------------------------------------------- | -------- |
| `targets` | Configures an SSH target to collect metrics from. | yes |
| `custom_metrics` | Defines custom metrics to collect from the target server. | yes |

### targets block

The `targets` block defines the remote servers to connect to and the metrics to collect. It supports the following arguments:

| Name | Type | Description | Default | Required |
| ----------------- | --------------------- | ---------------------------------------------------------------------- | ------- | -------- |
| `address` | `string` | The IP address or hostname of the target server. | | yes |
| `port` | `int` | SSH port number. | `22` | no |
| `username` | `string` | SSH username for authentication. | | yes |
| `password` | `secret` | Password for password-based SSH authentication. | | Required if `key_file` is not provided |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `password` | `secret` | Password for password-based SSH authentication. | | Required if `key_file` is not provided |
| `password` | `secret` | Password for password-based SSH authentication. | | no

If an argument is not always required, we'll usually just say "no" in the table and then we'll explain under the table that one of those two arguments must be supplied.

| `key_file` | `string` | Path to the private key file for key-based SSH authentication. | | Required if `password` is not provided |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to just have a key attribute which is a secret string. People can still get that string from a file if they use local.file. The benefit is that it's more flexible, because you can also get the secret from somewhere like remote.vault or remote.kubernetes.secret.

| `command_timeout` | `int` | Timeout in seconds for each command execution over SSH. | `30` | no |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be a duration? We rarely use dedicated seconds.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the expected behavior if command timeout > scrape timeout?

| `custom_metrics` | `block` | One or more custom metrics to collect from the target server. | | yes |

#### Authentication

You must provide either `password` or `key_file` for SSH authentication. If both are provided, `key_file` will be used.

### custom_metrics block

The `custom_metrics` block defines the metrics to collect from the target server. It supports the following arguments:

| Name | Type | Description | Default | Required |
| -------------- | --------------------- | ---------------------------------------------------------------------------- | ------- | -------- |
| `name` | `string` | The name of the metric. | | yes |
| `command` | `string` | The command to execute over SSH to collect the metric. | | yes |
| `type` | `string` | The type of the metric (`gauge` or `counter`). | | yes |
| `help` | `string` | Help text for the metric. | | no |
| `labels` | `map(string, string)` | Key-value pairs of labels to associate with the metric. | `{}` | no |
| `parse_regex` | `string` | Regular expression to parse the command output and extract the metric value. | | no |

#### Metric Types

- `gauge`: Represents a numerical value that can go up or down.
- `counter`: Represents a cumulative value that only increases.

#### parse_regex
Comment on lines +71 to +76
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#### Metric Types
- `gauge`: Represents a numerical value that can go up or down.
- `counter`: Represents a cumulative value that only increases.
#### parse_regex
The `type` argument can have either of the following values:
- `gauge`: Represents a numerical value that can go up or down.
- `counter`: Represents a cumulative value that only increases.

We usually don't use sub-headings under the tables, unless there's too much text and there's no other way to structure the information.


If the command output is not a simple numeric value, use `parse_regex` to extract the numeric value from the output.

---

## Secure Known Hosts Setup

### How It Works
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### How It Works
### How it works

I don't think we should uppercase each word. By the way, it's god to stick to the standard headings. USually the general "how it works" info is at the top section, and then notes related to specific functionality are in their respective attribute/block section.


The `prometheus.exporter.ssh` component uses the `known_hosts` file to validate host keys and protect against man-in-the-middle (MITM) attacks. Here's how it handles this:

1. **First-Time Setup**:
- If the `known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- If the `known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.
- If the `~/.ssh/known_hosts` file does not exist, the component creates it and fetches the host key using `ssh-keyscan`.

Would be good to know where the file is located.

- The fetched key is securely stored in the `known_hosts` file.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- The fetched key is securely stored in the `known_hosts` file.
- The fetched key is stored in the `known_hosts` file.

Is there anything more to this than just writing to a file?


2. **Subsequent Runs**:
- The component validates the server's host key against the stored key in `known_hosts`.
- If the keys match, the connection proceeds.
- If there is a mismatch, the component raises an error, requiring **manual intervention** to verify the legitimacy of the key change.

3. **Adding or Modifying Targets**:
- When a new target is added, or its address changes, the component automatically scans and stores the host key.
- If the target's key already exists but has changed, the connection is blocked until the discrepancy is resolved manually.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not use a new key from ssh-keyscan? Is it because we don't want to change pre-existing information under ~/.ssh? Would it be better to store the file in a location which Alloy owns, similar to how we store position files?


### Manual Resolution

If a host key mismatch occurs due to a legitimate key update:
- Manually update the `known_hosts` file with the new key using `ssh-keyscan` or other secure methods.

---

## Exported fields

{{< docs/shared lookup="reference/components/exporter-component-exports.md" source="alloy" version="<ALLOY_VERSION>" >}}

## Component health

`prometheus.exporter.ssh` is only reported as unhealthy if given an invalid configuration. In those cases, exported fields retain their last healthy values.

## Debug information

`prometheus.exporter.ssh` doesn't expose any component-specific debug information.

## Debug metrics

`prometheus.exporter.ssh` doesn't expose any component-specific debug metrics.

---

## Example

This example uses a [`prometheus.scrape` component][scrape] to collect metrics from `prometheus.exporter.ssh`:

```
prometheus.exporter.ssh "example" {
verbose_logging = true

targets {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using Alloy syntax arguments and blocks, I wonder if we should just have a targets argument which contains all the endpoint and auth information, similar to the targets argument for prometheus.exporter.snmp. It could work better with discovery components since they also output a targts array of type list(map(string)).

I'm not sure what is the best way to go yet... will need to think about it. Using Alloy syntax might be ok with some upcoming features like the foreach block
cc @wildum

address = "192.168.1.10"
port = 22
username = "admin"
password = "password"
command_timeout = 10

custom_metrics {
name = "load_average"
command = "cat /proc/loadavg | awk '{print $1}'"
type = "gauge"
help = "Load average over 1 minute"
}
}

targets {
address = "192.168.1.11"
port = 22
username = "monitor"
key_file = "/path/to/private.key"
command_timeout = 15

custom_metrics {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the metric come with labels too? E.g. the address? If it does, it would be good to document it.

name = "disk_usage"
command = "df / | tail -1 | awk '{print $5}'"
type = "gauge"
help = "Disk usage percentage"
parse_regex = "(\\d+)%"
}
}
}

// Configure a prometheus.scrape component to collect SSH metrics.
prometheus.scrape "demo" {
targets = prometheus.exporter.ssh.example.targets
forward_to = [prometheus.remote_write.demo.receiver]
}

prometheus.remote_write "demo" {
endpoint {
url = PROMETHEUS_REMOTE_WRITE_URL

basic_auth {
username = USERNAME
password = PASSWORD
}
}
}
```

Replace the following:

- `PROMETHEUS_REMOTE_WRITE_URL`: The URL of the Prometheus remote_write-compatible server to send metrics to.
- `USERNAME`: The username to use for authentication to the `remote_write` API.
- `PASSWORD`: The password to use for authentication to the `remote_write` API.

[scrape]: ../prometheus.scrape/

<!-- START GENERATED COMPATIBLE COMPONENTS -->

## Compatible components

`prometheus.exporter.ssh` has exports that can be consumed by the following components:

- Components that consume [Targets](../../../compatibility/#targets-consumers)

{{< admonition type="note" >}}
Connecting some components may not be sensible or components may require further configuration to make the connection work correctly.
Refer to the linked documentation for more details.
{{< /admonition >}}

<!-- END GENERATED COMPATIBLE COMPONENTS -->
1 change: 1 addition & 0 deletions internal/component/all/all.go
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ import (
_ "github.com/grafana/alloy/internal/component/prometheus/exporter/snmp" // Import prometheus.exporter.snmp
_ "github.com/grafana/alloy/internal/component/prometheus/exporter/snowflake" // Import prometheus.exporter.snowflake
_ "github.com/grafana/alloy/internal/component/prometheus/exporter/squid" // Import prometheus.exporter.squid
_ "github.com/grafana/alloy/internal/component/prometheus/exporter/ssh" // Import prometheus.exporter.ssh
_ "github.com/grafana/alloy/internal/component/prometheus/exporter/statsd" // Import prometheus.exporter.statsd
_ "github.com/grafana/alloy/internal/component/prometheus/exporter/unix" // Import prometheus.exporter.unix
_ "github.com/grafana/alloy/internal/component/prometheus/exporter/windows" // Import prometheus.exporter.windows
Expand Down
Loading