Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Elasticsearch] Fix the missing configuration environment variables propagation #1013

Merged
merged 2 commits into from
Oct 4, 2024
Merged

[Elasticsearch] Fix the missing configuration environment variables propagation #1013

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1c0aba0
[Elasticsearch] Fix the missing configuration environment variables p…
aleksmaus Oct 3, 2024
58ea997
Fixed typo in the doc
aleksmaus Oct 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 21 additions & 11 deletions config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"strconv"
"strings"
"text/template"
"time"

kingpin "github.com/alecthomas/kingpin/v2"
"github.com/spf13/viper"
Expand Down Expand Up @@ -94,17 +95,23 @@ var httpOutputDefaults = map[string]map[string]any{
"DropEventThresholds": "10000:critical: 1000:critical: 100:critical: 10:warning: 1:warning",
},
"Elasticsearch": {
"HostPort": "",
"Index": "falco",
"Type": "_doc",
"MinimumPriority": "",
"Suffix": "daily",
"Username": "",
"Password": "",
"FlattenFields": false,
"CreateIndexTemplate": false,
"NumberOfShards": 3,
"NumberOfReplicas": 3,
"ApiKey": "",
"HostPort": "",
"Index": "falco",
"Type": "_doc",
"MinimumPriority": "",
"Suffix": "daily",
"Username": "",
"Password": "",
"Pipeline": "",
"FlattenFields": false,
"CreateIndexTemplate": false,
"EnableCompression": false,
"NumberOfShards": 3,
"NumberOfReplicas": 3,
"Batching.Enabled": false,
"Batching.BatchSize": 5242880,
"Batching.FlushInterval": time.Second,
},
"Quickwit": {
"HostPort": "",
Expand Down Expand Up @@ -589,6 +596,9 @@ func getConfig() *types.Configuration {
v.GetStringMapString("AlertManager.CustomSeverityMap")
v.GetStringMapString("GCP.PubSub.CustomAttributes")
v.GetStringMapString("OTLP.Traces.ExtraEnvVars")

c.Elasticsearch.CustomHeaders = v.GetStringMapString("Elasticsearch.CustomHeaders")

if err := v.Unmarshal(c); err != nil {
log.Printf("[ERROR] : Error unmarshalling config : %s", err)
}
Expand Down
46 changes: 23 additions & 23 deletions docs/outputs/elasticsearch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,29 +13,29 @@

## Configuration

| Setting | Env var | Default value | Description |
| ------------------------------------- | ------------------------------------ | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `elasticsearch.hostport` | `ELASTICSEARCH_HOSTPORT` | | http://{domain or ip}:{port}, if not empty, Elasticsearch output is **enabled** |
| `elasticsearch.index` | `ELASTICSEARCH_INDEX` | `falco` | Index |
| `elasticsearch.type` | `ELASTICSEARCH_TYPE` | `_doc` | Index |
| `elasticsearch.pipeline` | `ELASTICSEARCH_PIPELINE` | | Optional ingest pipeline name. Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html |
| `elasticsearch.suffix` | `ELASTICSEARCH_SUFFIX` | `daily` | Date suffix for index rotation : `daily`, `monthly`, `annually`, `none` |
| `elasticsearch.apikey` | `ELASTICSEARCH_APIKEY` | | Use this APIKey to authenticate to Elasticsearch |
| `elasticsearch.username` | `ELASTICSEARCH_USERNAME` | | Use this username to authenticate to Elasticsearch |
| `elasticsearch.password` | `ELASTICSEARCH_PASSWORD` | | Use this password to authenticate to Elasticsearch |
| `elasticsearch.flattenfields` | `ELASTICSEARCH_FLATTENFIELDS` | `false` | Replace . by _ to avoid mapping conflicts, force to true if `createindextemplate=true` |
| `elasticsearch.createindextemplate` | `ELASTICSEARCH_CREATEINDEXTEMPLATE` | `false` | Create an index template |
| `elasticsearch.numberofshards` | `ELASTICSEARCH_NUMBEROFSHARDS` | `3` | Number of shards set by the index template |
| `elasticsearch.numberofreplicas` | `ELASTICSEARCH_REPLICAS` | `3` | Number of replicas set by the index template |
| `elasticsearch.customheaders` | `ELASTICSEARCH_CUSTOMHEADERS` | | Custom headers to add in POST, useful for Authentication |
| `elasticsearch.mutualtls` | `ELASTICSEARCH_MUTUALTLS` | `false` | Authenticate to the output with TLS, if true, checkcert flag will be ignored (server cert will always be checked) |
| `elasticsearch.checkcert` | `ELASTICSEARCH_CHECKCERT` | `true` | Check if ssl certificate of the output is valid |
| `elasticsearch.minimumpriority` | `ELASTICSEARCH_MINIMUMPRIORITY` | `""` (= `debug`) | Minimum priority of event for using this output, order is `emergency,alert,critical,error,warning,notice,informational,debug or ""` |
| `elasticsearch.maxconcurrentrequests` | `ELASTICSEARCH_MAXCONCURRENTREQUESTS`| `1` | Max number of concurrent requests |
| `elasticsearch.enablecompression` | `ELASTICSEARCH_ENABLECOMPRESSION` | `false` | Enables gzip compression |
| `elasticsearch.batching.enabled` | | `false` | Enables batching (utilizing Elasticsearch bulk API) |
| `elasticsearch.batching.batchsize` | | `5242880` | Batch size in bytes, default 5MB |
| `elasticsearch.batching.flushinterval`| | `1s` | Batch flush interval, use valid Go duration string |
| Setting | Env var | Default value | Description |
| ------------------------------------- | -------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `elasticsearch.hostport` | `ELASTICSEARCH_HOSTPORT` | | http://{domain or ip}:{port}, if not empty, Elasticsearch output is **enabled** |
| `elasticsearch.index` | `ELASTICSEARCH_INDEX` | `falco` | Index |
| `elasticsearch.type` | `ELASTICSEARCH_TYPE` | `_doc` | Index |
| `elasticsearch.pipeline` | `ELASTICSEARCH_PIPELINE` | | Optional ingest pipeline name. Documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html |
| `elasticsearch.suffix` | `ELASTICSEARCH_SUFFIX` | `daily` | Date suffix for index rotation : `daily`, `monthly`, `annually`, `none` |
| `elasticsearch.apikey` | `ELASTICSEARCH_APIKEY` | | Use this APIKey to authenticate to Elasticsearch |
| `elasticsearch.username` | `ELASTICSEARCH_USERNAME` | | Use this username to authenticate to Elasticsearch |
| `elasticsearch.password` | `ELASTICSEARCH_PASSWORD` | | Use this password to authenticate to Elasticsearch |
| `elasticsearch.flattenfields` | `ELASTICSEARCH_FLATTENFIELDS` | `false` | Replace . by _ to avoid mapping conflicts, force to true if `createindextemplate=true` |
| `elasticsearch.createindextemplate` | `ELASTICSEARCH_CREATEINDEXTEMPLATE` | `false` | Create an index template |
| `elasticsearch.numberofshards` | `ELASTICSEARCH_NUMBEROFSHARDS` | `3` | Number of shards set by the index template |
| `elasticsearch.numberofreplicas` | `ELASTICSEARCH_REPLICAS` | `3` | Number of replicas set by the index template |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `elasticsearch.numberofreplicas` | `ELASTICSEARCH_REPLICAS` | `3` | Number of replicas set by the index template |
| `elasticsearch.numberofreplicas` | `ELASTICSEARCH_NUMEROFREPLICAS` | `3` | Number of replicas set by the index template |

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still a typo :-) corrected.

| `elasticsearch.customheaders` | `ELASTICSEARCH_CUSTOMHEADERS` | | Custom headers to add in POST, useful for Authentication |
| `elasticsearch.mutualtls` | `ELASTICSEARCH_MUTUALTLS` | `false` | Authenticate to the output with TLS, if true, checkcert flag will be ignored (server cert will always be checked) |
| `elasticsearch.checkcert` | `ELASTICSEARCH_CHECKCERT` | `true` | Check if ssl certificate of the output is valid |
| `elasticsearch.minimumpriority` | `ELASTICSEARCH_MINIMUMPRIORITY` | `""` (= `debug`) | Minimum priority of event for using this output, order is `emergency,alert,critical,error,warning,notice,informational,debug or ""` |
| `elasticsearch.maxconcurrentrequests` | `ELASTICSEARCH_MAXCONCURRENTREQUESTS` | `1` | Max number of concurrent requests |
| `elasticsearch.enablecompression` | `ELASTICSEARCH_ENABLECOMPRESSION` | `false` | Enables gzip compression |
| `elasticsearch.batching.enabled` | `ELASTICSEARCH_BATCHING_ENABLED` | `false` | Enables batching (utilizing Elasticsearch bulk API) |
| `elasticsearch.batching.batchsize` | `ELASTICSEARCH_BATCHING_BATCHSIZE` | `5242880` | Batch size in bytes, default 5MB |
| `elasticsearch.batching.flushinterval`| `ELASTICSEARCH_BATCHING_FLUSHINTERVAL` | `1s` | Batch flush interval, use valid Go duration string |

> [!NOTE]
The Env var values override the settings from yaml file.
Expand Down