diff --git a/NEWS.md b/NEWS.md index ed5df12a58b..b5dae254ada 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,220 +1,409 @@ -4.5.0 +4.6.0 ===== -Read Axoflow's [blog post](https://axoflow.com/axosyslog-release-4-5/) for more details. +Read Axoflow's [blog post](https://axoflow.com/axosyslog-release-4-6/) for more details. You can read more about the new features in the AxoSyslog [documentation](https://axoflow.com/docs/axosyslog-core/). ## Highlights -#### Sending log messages to OpenObserve +### Forwarding logs to Google BigQuery -The `openobserve-log()` destination feeds OpenObserve via the [JSON API](https://openobserve.ai/docs/api/ingestion/logs/json/). +The `bigquery()` destination inserts logs to a Google BigQuery table via the +high-performance gRPC API. + +Authentication is done via [Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc). + +You can locate your BigQuery table with the `project()` `dataset()` and `table()` +options. + +There are two ways to configure your table's schema. + - You can set the columns and their respective type and template with the + `schema()` option. The available types are: `STRING`, `BYTES`, `INTEGER`, + `FLOAT`, `BOOLEAN`, `TIMESTAMP`, `DATE`, `TIME`, `DATETIME`, `JSON`, + `NUMERIC`, `BIGNUMERIC`, `GEOGRAPHY`, `RECORD`, `INTERVAL`. + - Alternatively you can import a `.proto` file with the `protobuf-schema()` option, + and map the templates for each column. + +The performance can be further improved with the `workers()`, `batch-lines()`, +`batch-bytes()`, `batch-timeout()` and `compression()` options. By default the +messages are sent with one worker, one message per batch and without compression. + +Keepalive can be configured with the `keep-alive()` block and its `time()`, +`timeout()` and `max-pings-without-data()` options. Example config: ``` -openobserve-log( - url("http://openobserve-endpoint") - port(5080) - stream("default") - user("root@example.com") - password("V2tsn88GhdNTKxaS") +bigquery( + project("test-project") + dataset("test-dataset") + table("test-table") + workers(8) + + schema( + "message" => "$MESSAGE" + "app" STRING => "$PROGRAM" + "host" STRING => "$HOST" + "pid" INTEGER => int("$PID") + ) + + on-error("drop-property") + + # or alternatively instead of schema(): + # protobuf-schema("/tmp/test.proto" + # => "$MESSAGE", "$PROGRAM", "$HOST", "$PID") + + # keep-alive(time(20000) timeout(10000) max-pings-without-data(0)) ); ``` -([#4698](https://github.com/syslog-ng/syslog-ng/pull/4698)) -#### Sending messages to Google Pub/Sub +Example `.proto` schema: +``` +syntax = "proto2"; +​ +message CustomRecord { + optional string message = 1; + optional string app = 2; + optional string host = 3; + optional int64 pid = 4; +} +``` -The `google-pubsub()` destination feeds Google Pub/Sub via the [HTTP REST API](https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.topics/publish). +([#4733](https://github.com/syslog-ng/syslog-ng/pull/4733)) +([#4770](https://github.com/syslog-ng/syslog-ng/pull/4770)) +([#4756](https://github.com/syslog-ng/syslog-ng/pull/4756)) + + +### Collecting native macOS system logs + +Two new sources have been added on macOS: `darwin-oslog()`, `darwin-oslog-stream()`. +`darwin-oslog()` replaced the earlier file source based solution with a native OSLog +framework based one, and is automatically used in the `system()` source on darwin +platform if the **darwinosl** plugin is presented. + +This plugin is available only on macOS 10.15 Catalina and above, the first version +that has the OSLog API. + +#### `darwin-oslog()` + +This is a native OSLog Framework based source to read logs from the local store of +the unified logging system on darwin OSes. +For more info, see https://developer.apple.com/documentation/oslog?language=objc + +The following parameters can be used for customization: + - `filter-predicate()` + - string value, which can be used to filter the log messages natively + - default value: `(eventType == 'logEvent' || eventType == 'lossEvent' || eventType == 'stateEvent' || eventType == 'userActionEvent') && (logType != 'debug')` + - for more details, see + - `man log` + - https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Predicates/Articles/pSyntax.html + - `go-reverse()` + - boolean value, setting to `yes` will provide a reverse-ordered log list + (from latest to oldest) + - default value: `no` + - `do-not-use-bookmark()` + - boolean value, setting to `yes` will prevent syslog-ng from continuing to + feed the logs from the last remembered position after a (re-)start, which means, + depending on the other settings, the feed will always start from the end/beginning + of the available log list + - default value: `no`, which means syslog-ng will attempt to continue feeding from + the last remembered log position after a (re-)start + - `max-bookmark-distance()` + - integer value, maximum distance in seconds that far an earlier bookmark can point + backward, e.g. if syslog-ng was stopped for 10 minutes and max-bookmark-distance + is set to 60 then syslog-ng will start feeding the logs only from the last 60 + seconds at startup, 9 minutes of logs 'will be lost' + - default value: `0`, which means no limit + - `read-old-records()` + - boolean value, controls if syslog-ng should start reading logs from the oldest + available at first start (or if no bookmark can be found) + - default value: `no` + - `fetch-delay()` + - integer value, controls how much time syslog-ng should wait between reading/sending + log messages, this is a fraction of a second, where wait_time = 1 second / n, so, + e.g. n=1 means that only about 1 log will be read and sent in each second, + and n=1 000 000 means only 1 microsecond (the allowed minimum value now!) + will be the delay between read/write attempts + - Use with care, though lower delay time can increase log feed performance, at the + same time could lead to a heavy system load! + - default value: `10 000` + - `fetch-retry-delay()` + - integer value, controls how many seconds syslog-ng will wait before a repeated + attempt to read/send once it's out of available logs + - default value: `1` + - `log-fetch-limit()` + - **Warning**: _This option is now disabled due to an OSLog API bug_ + _(https://openradar.appspot.com/radar?id=5597032077066240), once it's fixed it_ + _will be enabled again_ + - integer value, that limits the number of logs syslog-ng will send in one run + - default value: `0`, which means no limit + +NOTE: the persistent OSLog store is not infinite, depending on your system setting usually, +it keeps about 7 days of logs on disk, so it could happen that the above options cannot +operate the way you expect, e.g. if syslog-ng was stopped for about more then a week it +could happen that will not be able to restart from the last saved bookmark position +(as that might not be presented in the persistent log anymore) + +#### `darwin-oslog-stream()` + +This is a wrapper around the OS command line "log stream" command that can provide a live +log stream feed. Unlike in the case of `darwin-oslog()` the live stream can contain +non-persistent log events too, so take care, there might be a huge number of log events +every second that could put an unusual load on the device running syslog-ng with this source. +Unfortunately, there's no public API to get the same programmatically, so this one is +implemented using a program() source. + +Possible parameters: + - `params()` + - a string that can contain all the possible params the macOS `log` tool can accept + - see `log --help stream` for full reference, and `man log` for more details + - IMPORTANT: the parameter `--style` is used internally (defaults to `ndjson`), so it + cannot be overridden, please use other sysylog-ng features (templates, rewrite rules, etc.) + for final output formatting + - default value: `--type log --type trace --level info --level debug`, + you can use \``def-osl-stream-params`\` for referencing it if you wish to keep the + defaults when you add your own + +([#4423](https://github.com/syslog-ng/syslog-ng/pull/4423)) + +### Collecting qBittorrent logs + +The new `qbittorrent()` source, reads qBittorrent logs from its log file output. + +Example minimal config: +``` +source s_qbittorrent { + qbittorrent( + dir("/path/to/my/qbittorrent/root/log/dir") + ); +}; +``` -Example config: +The root dir of the qBittorrent logs can be found in the +"Tools" / "Preferences" / "Behavior" / "Log file" / "Save path" field. + +As the `qbittorrent()` source is based on a `file()` source, all of the `file()` +source options are applicable, too. + +([#4760](https://github.com/syslog-ng/syslog-ng/pull/4760)) + +### Collecting pihole FTL logs + +The new `pihole-ftl()` source reads pihole FTL (Faster Than Light) logs, which +are usually accessible in the "Tools" / "Pi-hole diagnosis" menu. + +Example minimal config: ``` -google-pubsub( - project("syslog-ng-project") - topic("syslog-ng-topic") - auth( - service-account( - key("/path/to/service-account-key.json") - ) - ) -); +source s_pihole_ftl { + pihole-ftl(); +}; ``` -See the [Google Pub/Sub documentation](https://cloud.google.com/pubsub/docs/building-pubsub-messaging-system) to learn more about configuring a service account. -([#4651](https://github.com/syslog-ng/syslog-ng/pull/4651)) +By default it reads the `/var/log/pihole/FTL.log` file. +You can change the root dir of Pi-hole's logs with the `dir()` option, +where the `FTL.log` file can be found. + +As the `pihole-ftl()` source is based on a `file()` source, all of the +`file()` source options are applicable, too. + +([#4760](https://github.com/syslog-ng/syslog-ng/pull/4760)) + +### Parsing Windows Eventlog XMLs + +The new `windows-eventlog-xml-parser()` introduces parsing support for Windows Eventlog XMLs. + +Its parameters are the same as the `xml()` parser. + +Example config: +``` +parser p_win { + windows-eventlog-xml-parser(prefix(".winlog.")); +}; +``` -#### Parsing PostgreSQL logs +([#4793](https://github.com/syslog-ng/syslog-ng/pull/4793)) -The `postgresql-csvlog-parser()`: add a new parser to process CSV log formatted by -PostgreSQL (https://www.postgresql.org/docs/current/runtime-config-logging.html). -The CSV format is extracted into a set of name-value pairs. -([#4586](https://github.com/syslog-ng/syslog-ng/pull/4586)) ## Features - * `http()`: Added support for using templates in the `url()` option. - - In syslog-ng a template can only be resolved on a single message, as the same - template might have different resolutions on different messages. A http batch - consists of multiple messages, so it is not trivial to decide which message should - be used for the resolution. - - When batching is enabled and multiple workers are configured it is important to - only batch messages which generate identical URLs. In this scenario one must set - the `worker-partition-key()` option with a template that contains all the templates - used in the `url()` option, otherwise messages will be mixed. - - For security reasons, all the templated contents in the `url()` option are getting - URL encoded automatically. Also the following parts of the url cannot be templated: - * scheme - * host - * port - * user - * password - ([#4663](https://github.com/syslog-ng/syslog-ng/pull/4663)) - - * `$TRANSPORT`: this is a new name-value pair that syslog-ng populates - automatically. It indicates the "transport" mechanism used to - retrieve/receive the message. It is up to the source driver to determine - the value. Currently the following values were implemented: - - BSD syslog drivers: `tcp()`, `udp()` & `network()` - * `rfc3164+tls` - * `rfc3164+tcp` - * `rfc3164+udp` - * `rfc3164+proxied-tls` - * `rfc3164+` - - UNIX domain drivers: `unix-dgram()`, `unix-stream()` - * `unix-stream` - * `unix-dgram` - - RFC5424 style syslog: `syslog()`: - * `rfc5426`: syslog over udp - * `rfc5425`: syslog over tls - * `rfc6587`: syslog over tcp - * `rfc5424+`: syslog over a logproto plugin - - Other drivers: - * otlp: `otel()` driver - * mqtt: `mqtt()` driver - * hypr-api: `hypr-audit-source()` driver - - `$IP_PROTO`: indicate the IP protocol version used to retrieve/receive the - message. Contains either "4" to indicate IPv4 and "6" to indicate IPv6. - ([#4673](https://github.com/syslog-ng/syslog-ng/pull/4673)) - - * `network()` and `syslog()` drivers: Added `ignore-validity-period` as a new flag to `ssl-options()`. - - By specifying `ignore-validity-period`, you can ignore the validity periods - of certificates during the certificate validation process. - ([#4642](https://github.com/syslog-ng/syslog-ng/pull/4642)) - - * `tls()` in `udp()`/`tcp()`/`network()` and `syslog()` drivers: add support - for a new `http()` compatible ssl-version() option. This makes the TLS - related options for http() and other syslog-like drivers more similar. This - requires OpenSSL 1.1.0. - ([#4682](https://github.com/syslog-ng/syslog-ng/pull/4682)) - - * `cloud-auth()`: Added a new plugin for drivers, which implements different cloud related authentications. - - Currently the only supported authentication is [GCP's Service Account](https://cloud.google.com/iam/docs/service-account-overview) for the `http()` destination. + * `cloud-auth()`: Added support for `user-managed-service-account()` `gcp()` auth method. - Example config: + This authentication method can be used on VMs in GCP to use the linked service. + + Example minimal config, which tries to use the "default" service account: + ``` + cloud-auth( + gcp( + user-managed-service-account() + ) + ) + ``` + + Full config: ``` - http( - cloud-auth( - gcp( - service-account( - key("/path/to/service-account-key.json") - audience("https://pubsub.googleapis.com/google.pubsub.v1.Publisher") - ) + cloud-auth( + gcp( + user-managed-service-account( + name("alltilla@syslog-ng-test-project.iam.gserviceaccount.com") + metadata-url("my-custom-metadata-server:8080") ) ) - ); + ) ``` - ([#4651](https://github.com/syslog-ng/syslog-ng/pull/4651)) - - * `csv-parser()`: allow parsing the extracted values into matches ($1, $2, $3 ...) - by omitting the columns() parameter, which normally specifies the column - names. - ([#4678](https://github.com/syslog-ng/syslog-ng/pull/4678)) - - * `--check-startup`: a new command line option for syslog-ng along with the - existing `--syntax-only`. This new option will do a complete configuration - initialization and then exit with exit code indicating the result. Since - this also initializes things like network listeners, it will probably _not_ - work when there is another syslog-ng instance running in the background. The - recommended use of this option is a dedicated config check container, as - explained in #4592. - ([#4646](https://github.com/syslog-ng/syslog-ng/pull/4646)) -## Bugfixes + This authentication method is extremely useful with syslog-ng's `google-pubsub()` destination, + when it is running on VMs in GCP, for example: + ``` + destination { + google-pubsub( + project("syslog-ng-test-project") + topic("syslog-ng-test-topic") + auth(user-managed-service-account()) + ); + }; + ``` + + For more info about this GCP authentication method, see: + * https://cloud.google.com/compute/docs/access/authenticate-workloads#curl + * https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances + ([#4755](https://github.com/syslog-ng/syslog-ng/pull/4755)) - * `s3`: Fixed an ImportError. + * `opentelemetry()`, `syslog-ng-otlp()` sources: Added `workers()` option. - `ImportError: cannot import name 'SharedBool' from 'syslogng.modules.s3.s3_object'` - ([#4700](https://github.com/syslog-ng/syslog-ng/pull/4700)) + This feature enables processing the OTLP messages on multiple threads, + which can greatly improve the performance. + By default it is set to `workers(1)`. + ([#4774](https://github.com/syslog-ng/syslog-ng/pull/4774)) - * `loki()`: fixed mixing non-related label values - ([#4713](https://github.com/syslog-ng/syslog-ng/pull/4713)) + * `opentelemetry()`, `syslog-ng-otlp()` destinations: Added `compression()` option. - * type hinting: Parsing and casting fractions are now done locale independently. - ([#4702](https://github.com/syslog-ng/syslog-ng/pull/4702)) + This boolean option can be used to enable gzip compression in gRPC requests. + By default it is set to `compression(no)`. + ([#4765](https://github.com/syslog-ng/syslog-ng/pull/4765)) - * `metrics-probe()`: Fixed a crash. + * `opentelemetry()`, `syslog-ng-otlp()` destinations: Added `batch-bytes()` option. - This crash occurred when a `metrics-probe()` instance was used in multiple source threads, - like a `network()` source with multiple connections. - ([#4685](https://github.com/syslog-ng/syslog-ng/pull/4685)) + This option lets the user limit the bytes size of a batch. As there is a + default 4 MiB batch limit by OTLP, it is necessary to keep the batch size + smaller, but it would be hard to configure without this option. - * `flags()` argument to various drivers: fix a potential crash in case a flag with at least 32 characters is used. - No such flag is defined by syslog-ng, so the only way to trigger the crash is to use an invalid configuration file. - ([#4689](https://github.com/syslog-ng/syslog-ng/pull/4689)) + Please note that the batch can be at most 1 message larger than the set + limit, so consider this when setting this value. - * Fix `$PROTO` value for `transport(tls)` connections, previously it was set - to "0" while in reality these are tcp connections (e.g. "6"). + The default value is 4 MB, which is a bit below 4 MiB. - Fix how syslog-ng sets $HOST for V4-mapped addresses in case of IPv6 source - drivers (e.g. `udp6()`/`tcp6()` or when using `ip-protocol(6)` for `tcp()`/`udp()`). - Previously V4-mapped addresses would be represented as - `"::ffff:"`. This is not wrong per-se, but would potentially - cause the same host to be represented in multiple ways. With the fix, - syslog-ng would just use `""` in these cases. - ([#4673](https://github.com/syslog-ng/syslog-ng/pull/4673)) + The calculation of the batch size is done before compression, which is + the same as the limit is calculated on the server. - * `db-parser()`: support nested match characters in `@QSTRING@` pattern parser - ([#4717](https://github.com/syslog-ng/syslog-ng/pull/4717)) + Example config: + ``` + syslog-ng-otlp( + url("localhost:12345") + workers(16) + log-fifo-size(1000000) + batch-timeout(5000) # ms + batch-lines(1000000) # Huge limit, batch-bytes() will limit us sooner -## Other changes + batch-bytes(1MB) # closes and flushes the batch after the last message pushed it above the 1 MB limit + # not setting batch-bytes() defaults to 4 MB, which is a bit below the default 4 MiB limit + ); + ``` + ([#4772](https://github.com/syslog-ng/syslog-ng/pull/4772)) + + * `opentelemetry()`, `syslog-ng-otlp()`: Added syslog-ng style list support. + ([#4794](https://github.com/syslog-ng/syslog-ng/pull/4794)) - * `LogSource` and `LogFetcher`: additional documentation was added to these - Python classes to cover explicit source-side batching functionalities (e.g. - the `auto_close_batch` attribute and the `close_batch()` method). - ([#4673](https://github.com/syslog-ng/syslog-ng/pull/4673)) + * `$(tag)` template function: expose bit-like tags that are set on messages. - * `rate-limit()`: Renamed the `template()` option to `key()`, which better communicates the intention. - ([#4679](https://github.com/syslog-ng/syslog-ng/pull/4679)) + Syntax: + `$(tag )` - * templates: The `template-escape()` option now only escapes the top-level template function. + Unless the value-if-set/unset arguments are specified `$(tag)` results in a + boolean type, expanding to "0" or "1" depending on whether the message has + the specified tag set. - Before syslog-ng 4.5.0 if you had embedded template functions, the `template-escape(yes)` setting - escaped the output of each template function, so the parent template function received an - already escaped string. This was never the intention of the `template-escape()` option. + If value-if-set/unset are present, `$(tag)` would return a string, picking the + second argument `` if the message has `` and picking the + third argument `` if the message does not have `` + ([#4766](https://github.com/syslog-ng/syslog-ng/pull/4766)) - Although this is a breaking change, we do not except anyone having a config that is affected. - If you have such a config, make sure to follow-up this change. If you need help with it, feel - free to open an issue or discussion on GitHub, or contact us on the Axoflow Discord server. - ([#4666](https://github.com/syslog-ng/syslog-ng/pull/4666)) + * `set-severity()` support for aliases: widespread aliases to severity values + produced by various applications are added to set-severity(). + ([#4763](https://github.com/syslog-ng/syslog-ng/pull/4763)) - * `loki()`: The `timestamp()` option now supports quoted strings. + * `flags(seqnum-all)`: available in all destination drivers, this new flag + changes `$SEQNUM` behaviour, so that all messages get a sequence number, not + just local ones. Previously syslog-ng followed the logic of the RFC5424 + meta.sequenceId structured data element, e.g. only local messages were to + get a sequence number, forwarded messages retained their original sequenceId + that we could potentially receive ourselves. - The valid values are the following, with or without quotes, case insensitive: - * "current" - * "received" - * "msg" - ([#4688](https://github.com/syslog-ng/syslog-ng/pull/4688)) + For example, this destination would include the meta.sequenceId SDATA + element even for non-local logs and increment that value by every message + transmitted: + + `destination { syslog("127.0.0.1" port(2001) flags(seqnum-all)); };` + + This generates a message like this on the output, even if the message is + not locally generated (e.g. forwarded from another syslog sender): + + ``` + <13>1 2023-12-09T21:51:30+00:00 localhost sdff - - [meta sequenceId="1"] f sdf fsd + <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="2"] f sdf fsd + <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="3"] f sdf fsd + <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="4"] f sdf fsd + <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="5"] f sdf fsd + ``` + ([#4745](https://github.com/syslog-ng/syslog-ng/pull/4745)) + + * `loggen`: improve loggen performance for synthetic workloads, so we can test + for example up to 650k msg/sec on a AMD Ryzen 7 Pro 6850U CPU. + ([#4476](https://github.com/syslog-ng/syslog-ng/pull/4476)) + + +## Bugfixes + + * `metrics-probe()`: Fixed not cleaning up dynamic labels for each message if no static labels are set. + ([#4750](https://github.com/syslog-ng/syslog-ng/pull/4750)) + + * `regexp-parser()`: Fixed a bug, which stored some values incorrectly if `${MESSAGE}` was changed with a capture group. + ([#4759](https://github.com/syslog-ng/syslog-ng/pull/4759)) + + * `network()` source: fix marking originally valid utf-8 messages when `sanitize-utf8` is enabled + ([#4744](https://github.com/syslog-ng/syslog-ng/pull/4744)) + + * `python()`: Fixed a memory leak in `list` typed `LogMessage` values. + ([#4790](https://github.com/syslog-ng/syslog-ng/pull/4790)) + +## Packaging + + * `VERSION` renamed to `VERSION.txt`: due to a name collision with C++ based + builds on MacOS, the file containing our version number was renamed to + VERSION.txt. + ([#4775](https://github.com/syslog-ng/syslog-ng/pull/4775)) + + * Added `gperf` as a build dependency. + ([#4763](https://github.com/syslog-ng/syslog-ng/pull/4763)) + + +## Notes to developers + + * `LogThreadedSourceDriver`: Added multi-worker API, which is a breaking change. + + Check the Pull Request for inspiration on how to follow up these changes. + ([#4774](https://github.com/syslog-ng/syslog-ng/pull/4774)) + + +## Other changes + + * `network()`/`syslog()` sources: support UTF-8 sanitization/validation of RFC 5424 and `no-parse` messages + + The `sanitize-utf8`, `validate-utf8` flags are now supported when parsing RFC 5424 messages or when parsing is disabled. + ([#4744](https://github.com/syslog-ng/syslog-ng/pull/4744)) + + * APT packages: Added Ubuntu Mantic Minotaur. + ([#4737](https://github.com/syslog-ng/syslog-ng/pull/4737)) ## syslog-ng Discord @@ -222,7 +411,6 @@ For a bit more interactive discussion, join our Discord server: [![Axoflow Discord Server](https://discordapp.com/api/guilds/1082023686028148877/widget.png?style=banner2)](https://discord.gg/E65kP9aZGm) - ## Credits syslog-ng is developed as a community project, and as such it relies @@ -234,5 +422,4 @@ of syslog-ng, contribute. We would like to thank the following people for their contribution: -Attila Szakacs, Balazs Scheidler, Cedric Arickx, Fabrice Fontaine, -Hofi, László Várady, Romain Tartière, Szilard Parrag, yashmathne +Attila Szakacs, Balazs Scheidler, Hofi, László Várady, Romain Tartière diff --git a/news/bugfix-4744.md b/news/bugfix-4744.md deleted file mode 100644 index b6505ac3c2f..00000000000 --- a/news/bugfix-4744.md +++ /dev/null @@ -1 +0,0 @@ -`network()` source: fix marking originally valid utf-8 messages when `sanitize-utf8` is enabled diff --git a/news/bugfix-4750.md b/news/bugfix-4750.md deleted file mode 100644 index 68075753cd5..00000000000 --- a/news/bugfix-4750.md +++ /dev/null @@ -1 +0,0 @@ -`metrics-probe()`: Fixed not cleaning up dynamic labels for each message if no static labels are set. diff --git a/news/bugfix-4759.md b/news/bugfix-4759.md deleted file mode 100644 index d1acb54c467..00000000000 --- a/news/bugfix-4759.md +++ /dev/null @@ -1 +0,0 @@ -`regexp-parser()`: Fixed a bug, which stored some values incorrectly if `${MESSAGE}` was changed with a capture group. diff --git a/news/bugfix-4790.md b/news/bugfix-4790.md deleted file mode 100644 index 14eb8cb47f8..00000000000 --- a/news/bugfix-4790.md +++ /dev/null @@ -1 +0,0 @@ -`python()`: Fixed a memory leak in `list` typed `LogMessage` values. diff --git a/news/developer-note-4774.md b/news/developer-note-4774.md deleted file mode 100644 index e9b1f3274cc..00000000000 --- a/news/developer-note-4774.md +++ /dev/null @@ -1,3 +0,0 @@ -`LogThreadedSourceDriver`: Added multi-worker API, which is a breaking change. - -Check the Pull Request for inspiration on how to follow up these changes. diff --git a/news/feature-4423.md b/news/feature-4423.md deleted file mode 100644 index 23fed55fc2f..00000000000 --- a/news/feature-4423.md +++ /dev/null @@ -1,59 +0,0 @@ -`system()`, `darwin-oslog()`, `darwin-oslog-stream()` sources: Added 2 darwin OSLog based sources `darwin-oslog()` and `darwin-oslog-stream()`. - -'darwin-oslog()' replaced the earlier file source based solution with a native OSLog framework based one, and is automatically used in the `system()` source on darwin platform if the **darwinosl** plugin is presented. - -This plugin available only on macOS 10.15 Catalina and above, the first version that has the OSLog API. - -### darwin-oslog - -This is a native OSLog Framework based source to read logs from the local store of the unified logging system on darwin OSes. -For more info, see https://developer.apple.com/documentation/oslog?language=objc - - -The following parameters can be used for customization: - -- filter-predicate - - string value, which can be used to filter the log messages natively - - default value: `(eventType == 'logEvent' || eventType == 'lossEvent' || eventType == 'stateEvent' || eventType == 'userActionEvent') && (logType != 'debug')` - - for more details, see - - `man log` - - https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Predicates/Articles/pSyntax.html -- go-reverse - - boolean value, setting to **yes** will provide a reverse-ordered log list (from latest to oldest) - - default value: **no** -- do-not-use-bookmark - - boolean value, setting to **yes** will prevent syslog-ng from continuing to feed the logs from the last remembered position after a (re-)start, which means, depending on the other settings, the feed will always start from the end/beginning of the available log list - - default value: **no**, which means syslog-ng will attempt to continue feeding from the last remembered log position after a (re-)start -- max-bookmark-distance - - integer value, maximum distance in seconds that far an earlier bookmark can point backward, e.g. if syslog-ng was stopped for 10 minutes and max-bookmark-distance is set to 60 then syslog-ng will start feeding the logs only from the last 60 seconds at startup, 9 minutes of logs 'will be lost' - - default value: 0, which means no limit -- read-old-records - - boolean value, controls if syslog-ng should start reading logs from the oldest available at first start (or if no bookmark can be found) - - default value: no -- fetch-delay - - integer value, controls how much time syslog-ng should wait between reading/sending log messages, this is a fraction of a second, where wait_time = 1 second / n, so, e.g. n=1 means that only about 1 log will be read and sent in each second, and n=1 000 000 means only 1 microsecond (the allowed minimum value now!) will be the delay between read/write attempts - (Use with care, though lower delay time can increase log feed performance, at the same time could lead to a heavy system load!) - - default value: 10 000 -- fetch-retry-delay - - integer value, controls how many seconds syslog-ng will wait before a repeated attempt to read/send once it's out of available logs - - default value: 1 -- log-fetch-limit - - **Warning**: _This option is now disabled due to an OSLog API bug (https://openradar.appspot.com/radar?id=5597032077066240), once it's fixed it will be enabled again_ - - integer value, that limits the number of logs syslog-ng will send in one run - - default value: 0, which means no limit - -NOTE: the persistent OSLog store is not infinite, depending on your system setting usually, it keeps about 7 days of logs on disk, so it could happen that the above options cannot operate the way you expect, e.g. if syslog-ng was stopped for about more then a week it could happen that will not be able to restart from the last saved bookmark position (as that might not be presented in the persistent log anymore) - -### darwin-oslog-stream - -This is a wrapper around the OS command line "log stream" command that can provide a live log stream feed. -Unlike in the case of `darwin-oslog()` the live stream can contain non-persistent log events too, so take care, there might be a huge number of log events every second that could put an unusual load on the device running syslog-ng with this source. -Unfortunately, there's no public API to get the same programmatically, so this one is implemented using a program() source. - -Possible parameters: - -- params - - a string that can contain all the possible params the macOS `log` tool can accept, please see, `log --help stream` for full reference, and `man log` for more details. - - IMPORTANT: the parameter --style is used internally (defaults to `ndjson`), so it cannot be overridden, please use other sysylog-ng features (templates, rewrite rules, etc.) for final output formatting - - default value: `--type log --type trace --level info --level debug`, you can use \``def-osl-stream-params`\` for referencing it if you wish to keep the defaults when you add your own diff --git a/news/feature-4476.md b/news/feature-4476.md deleted file mode 100644 index 8edef3c3e5d..00000000000 --- a/news/feature-4476.md +++ /dev/null @@ -1,2 +0,0 @@ -`loggen`: improve loggen performance for synthetic workloads, so we can test -up to 650k msg/sec on my AMD Ryzen 7 Pro 6850U CPU. diff --git a/news/feature-4745.md b/news/feature-4745.md deleted file mode 100644 index 10c68905a60..00000000000 --- a/news/feature-4745.md +++ /dev/null @@ -1,21 +0,0 @@ -`flags(seqnum-all)`: available in all destination drivers, this new flag -changes $SEQNUM behaviour, so that all messages get a sequence number, not -just local ones. Previously syslog-ng followed the logic of the RFC5424 -meta.sequenceId structured data element, e.g. only local messages were to -get a sequence number, forwarded messages retained their original sequenceId -that we could potentially receive ourselves. - -For example, this destination would include the meta.sequenceId SDATA -element even for non-local logs and increment that value by every message -transmitted: - - destination { syslog("127.0.0.1" port(2001) flags(seqnum-all)); }; - -This generates a message like this on the output, even if the message is -not locally generated (e.g. forwarded from another syslog sender): - - <13>1 2023-12-09T21:51:30+00:00 localhost sdff - - [meta sequenceId="1"] f sdf fsd - <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="2"] f sdf fsd - <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="3"] f sdf fsd - <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="4"] f sdf fsd - <13>1 2023-12-09T21:51:32+00:00 localhost sdff - - [meta sequenceId="5"] f sdf fsd diff --git a/news/feature-4755.md b/news/feature-4755.md deleted file mode 100644 index 7da01056ea9..00000000000 --- a/news/feature-4755.md +++ /dev/null @@ -1,40 +0,0 @@ -`cloud-auth()`: Added support for `user-managed-service-account()` `gcp()` auth method. - -This authentication method can be used on VMs in GCP to use the linked service. - -Example minimal config, which tries to use the "default" service account: -``` -cloud-auth( - gcp( - user-managed-service-account() - ) -) -``` - -Full config: -``` -cloud-auth( - gcp( - user-managed-service-account( - name("alltilla@syslog-ng-test-project.iam.gserviceaccount.com") - metadata-url("my-custom-metadata-server:8080") - ) - ) -) -``` - -This authentication method is extremely useful with syslog-ng's `google-pubsub()` destination, -when it is running on VMs in GCP, for example: -``` -destination { - google-pubsub( - project("syslog-ng-test-project") - topic("syslog-ng-test-topic") - auth(user-managed-service-account()) - ); -}; -``` - -For more info about this GCP authentication method, see: - * https://cloud.google.com/compute/docs/access/authenticate-workloads#curl - * https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances diff --git a/news/feature-4760-1.md b/news/feature-4760-1.md deleted file mode 100644 index 57d60aad5bb..00000000000 --- a/news/feature-4760-1.md +++ /dev/null @@ -1,18 +0,0 @@ -`pihole-ftl()`: Added a new source, which reads Pi-hole FTL logs. - -The FTL (Faster Than Light) logs are the logs which can be found -in the "Tools" / "Pi-hole diagnosis" menu. - -Example minimal config: -``` -source s_pihole_ftl { - pihole-ftl(); -}; -``` - -By default it reads the `/var/log/pihole/FTL.log` file. -You can change the root dir of Pi-hole's logs with the `dir()` option, -where the `FTL.log` file can be found. - -As the `pihole-ftl()` source is based on a `file()` source, all of the -`file()` source options are applicable, too. diff --git a/news/feature-4760-2.md b/news/feature-4760-2.md deleted file mode 100644 index b650d18a888..00000000000 --- a/news/feature-4760-2.md +++ /dev/null @@ -1,16 +0,0 @@ -`qbittorrent()`: Added a new source, which reads qBittorrent logs. - -Example minimal config: -``` -source s_qbittorrent { - qbittorrent( - dir("/path/to/my/qbittorrent/root/log/dir") - ); -}; -``` - -The root dir of the qBittorrent logs can be found in the -"Tools" / "Preferences" / "Behavior" / "Log file" / "Save path" field. - -As the `qbittorrent()` source is based on a `file()` source, all of the `file()` -source options are applicable, too. diff --git a/news/feature-4763.md b/news/feature-4763.md deleted file mode 100644 index a2c533dde4a..00000000000 --- a/news/feature-4763.md +++ /dev/null @@ -1,2 +0,0 @@ -`set-severity()` support for aliases: widespread aliases to severity values -produced by various applications are added to set-severity(). diff --git a/news/feature-4765.md b/news/feature-4765.md deleted file mode 100644 index 5dff9a666df..00000000000 --- a/news/feature-4765.md +++ /dev/null @@ -1,3 +0,0 @@ -Add compression to `syslog-ng-otlp()` and `opentelemetry()`: the new -compression() option can be used to enable deflate compression in gRPC -requests. diff --git a/news/feature-4766.md b/news/feature-4766.md deleted file mode 100644 index 93768ce4302..00000000000 --- a/news/feature-4766.md +++ /dev/null @@ -1,13 +0,0 @@ -`$(tag)` template function: expose bit-like tags that are set on -messages. - -Syntax: - $(tag ) - -Unless the value-if-set/unset arguments are specified $(tag) results in a -boolean type, expanding to "0" or "1" depending on whether the message has -the specified tag set. - -If value-if-set/unset are present, $(tag) would return a string, picking the -second argument if the message has and picking the -third argument if the message does not have diff --git a/news/feature-4772.md b/news/feature-4772.md deleted file mode 100644 index 6a04dd1e3a1..00000000000 --- a/news/feature-4772.md +++ /dev/null @@ -1,28 +0,0 @@ -`opentelemetry()`, `syslog-ng-otlp()` destinations: Added a new `batch-bytes()` option. - -This option lets the user limit the bytes size of a batch. As there is a -default 4 MiB batch limit by OTLP, it is necessary to keep the batch size -smaller, but it would be hard to configure without this option. - -Please note that the batch can be at most 1 message larger than the set -limit, so consider this when setting this value. - -The default value is 4 MB, which is a bit below 4 MiB. - -The calculation of the batch size is done before compression, which is -the same as the limit is calculated on the server. - -Example config: -``` - syslog-ng-otlp( - url("localhost:12345") - workers(16) - log-fifo-size(1000000) - - batch-timeout(5000) # ms - batch-lines(1000000) # Huge limit, batch-bytes() will limit us sooner - - batch-bytes(1MB) # closes and flushes the batch after the last message pushed it above the 1 MB limit - # not setting batch-bytes() defaults to 4 MB, which is a bit below the default 4 MiB limit - ); -``` diff --git a/news/feature-4774.md b/news/feature-4774.md deleted file mode 100644 index 591ab1d9090..00000000000 --- a/news/feature-4774.md +++ /dev/null @@ -1,4 +0,0 @@ -`opentelemetry()`, `syslog-ng-otlp()`: Added `workers()` option on source side. - -This feature enables processing the OTLP messages on multiple threads, -which can greatly improve the performance. diff --git a/news/feature-4793.md b/news/feature-4793.md deleted file mode 100644 index 28be4ff49d0..00000000000 --- a/news/feature-4793.md +++ /dev/null @@ -1,10 +0,0 @@ -`windows-eventlog-xml-parser()`: Added a new parser to parse Windows Eventlog XMLs. - -Its parameters are the same as the `xml()` parser. - -Example config: -``` -parser p_win { - windows-eventlog-xml-parser(prefix(".winlog.")); -}; -``` diff --git a/news/other-4737.md b/news/other-4737.md deleted file mode 100644 index bd750f7d32c..00000000000 --- a/news/other-4737.md +++ /dev/null @@ -1 +0,0 @@ -APT packages: Added Ubuntu Mantic Minotaur. diff --git a/news/other-4744.md b/news/other-4744.md deleted file mode 100644 index 9e025fb0004..00000000000 --- a/news/other-4744.md +++ /dev/null @@ -1,3 +0,0 @@ -`network()`/`syslog()` sources: support UTF-8 sanitization/validation of RFC 5424 and `no-parse` messages - -The `sanitize-utf8`, `validate-utf8` flags are now supported when parsing RFC 5424 messages or when parsing is disabled. diff --git a/news/packaging-4763.md b/news/packaging-4763.md deleted file mode 100644 index 569dd2de9e8..00000000000 --- a/news/packaging-4763.md +++ /dev/null @@ -1 +0,0 @@ -Added `gperf` as a build dependency. diff --git a/news/packaging-4775.md b/news/packaging-4775.md deleted file mode 100644 index c09dc957af9..00000000000 --- a/news/packaging-4775.md +++ /dev/null @@ -1,3 +0,0 @@ -`VERSION` renamed to `VERSION.txt`: due to a name collision with C++ based -builds on MacOS, the file containing our version number was renamed to -VERSION.txt.