Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

loki.source.syslog: do not drop syslog messages if they can not be parsed. #2275

Open
Nachtfalkeaw opened this issue Dec 12, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@Nachtfalkeaw
Copy link

Nachtfalkeaw commented Dec 12, 2024

Request

Losing logs only because they do not fit into any syslog standard / rfc is a bad idea. It may contain sensitive information which should not belost.
This is always a disadvantage of a produkct like loki or alloy's loki.source.syslog.

There should be an option like:
"allow unoparsed messages" = false (default

There should be an option

"allow unoparsed messages" = true
"unparsed_messages_destination" = [default] or [loki.process.xyz.receiver]

There was a discussion on slack asking for what happens with messages which count against loki_source_syslog_parsing_errors_total
https://grafana.slack.com/archives/C01050C3D8F/p1733952946344909?thread_ts=1733928788.934089&cid=C01050C3D8F

Code:
https://github.com/grafana/alloy/blob/main/internal/component/loki/source/syslog/internal/syslogtarget/transport.go#L326

Use case

The user should be able to decide if he wants to allow unparsed syslog messages.
If it can not be parsed then the user should have two additional possibilities. He should decide if the message should follow the default path or if the message should be routed to another component which then maybe starts to process the message (loki.process) as the user wants.

An additional advantage would be, if someone is sending rfc5424 messages to a loki rfc3164 listener the message can not be parsed.
No problem, no need for an additional listener, but "allow unparsed messages" and define a different forward_to endpoint where the user can try to parse the syslog message with another listener or another parser.

Another recent limitation of loki.source.syslog is that there can be only one listener per port.
e.g. rfc5424 is on port 514 UDP/TCP. But if some applications send rfc3164 syslogs and can not change the destination port then this can not be solved with the same grafana alloy loki.source.syslog component.

For these reasons:

  1. unparsed messages should be allows to be processed
  2. user should be able which path these messages should go. send to another listener which is parsing it as another rfc or another component like loki.process
  3. The feature (2.) allows to consume syslog messsages of all rfcs on the same listener and decide later how to process these messsages if default configured rfc can not be identified.
    loki.source.syslog: drops multiline syslog messages and only accepts first message #2286
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant