loki.source.syslog: drops multiline syslog messages and only accepts first message

[Nachtfalkeaw]

What's wrong?

Im am running alloy 1.5.1 with loki.source.syslog:

loki.source.syslog "syslog_receiver_udp_5514" {
  // RFC3164 listeners on port 5514
  listener {
    address = "0.0.0.0:5514"
    protocol = "udp"
    use_incoming_timestamp = false
    max_message_length = 0
    syslog_format = "rfc3164"
    labels = {
      protocol = "udp",
      format = "rfc3164",
      port = "5514",
      service_name = "syslog_rfc3164_5514",
    }
  }
  relabel_rules = loki.relabel.syslog_01.rules
  forward_to = [loki.process.syslog_processor.receiver]
}

From my Unifi application I receive syslog messages which seem to be multiline or truncated UDP message. Grafana alloy's loki.source.syslog is complaining it cannot parse the lines.

ts=2024-12-15T00:48:08.5810616Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog_receiver_udp_5514 err="invalid or unsupported framing. first byte: '{'"

error parsing syslog stream

In the following screenshots you can see a tcpdump showing all syslog messages captured with füll size. the "red" and "green" show the syslogs which could be parsed because they start with <xx> and the logs in the blue box are missing and were dropped but they should have been added to the log in the red box as multiline or a Long single logline.

I could imagine a few possible ideas:

1. Add an option to allow syslog messages within a configurable time range like e.g. 3s without correct framing to be appended to the previous valid parsed syslog message of the Same sources IP and only drop messages older than configured 3s.
2. Add an option to allow up to X (e.g. 10) syslog messages without correct framing to be appended to the previous valid syslog message and only drop messages number 11+
3. Never drop messages, try to append it like configured before and if not possible to append then forward it, maybe add an additional internal label like __syslog_message_unparseable and add as many other internal labels as possible to make it possible for users in further steps like loki.process to map these messages. (loki.source.syslog: do not drop syslog messages if they can not be parsed. #2275)
4. Add the unparseable Log Line to the debug logs.
5. Add the RFC complaining to the logs.

Steps to reproduce

Send syslogs with multiline to loki.source.syslog

System information

Linux ubiquiti 5.10.0-33-amd64 #1 SMP Debian 5.10.226-1 (2024-10-03) x86_64 GNU/Linux

Software version

alloy, version v1.5.1 (branch: HEAD, revision: dc8a365)

Configuration

as above:

loki.source.syslog "syslog_receiver_udp_5514" {
  // RFC3164 listeners on port 5514
  listener {
    address = "0.0.0.0:5514"
    protocol = "udp"
    use_incoming_timestamp = false
    max_message_length = 0
    syslog_format = "rfc3164"
    labels = {
      protocol = "udp",
      format = "rfc3164",
      port = "5514",
      service_name = "syslog_rfc3164_5514",
    }
  }
  relabel_rules = loki.relabel.syslog_01.rules
  forward_to = [loki.process.syslog_processor.receiver]
}

Logs

as above

`ts=2024-12-15T00:48:08.5810616Z level=warn msg="error parsing syslog stream" component_path=/ component_id=loki.source.syslog.syslog_receiver_udp_5514 err="invalid or unsupported framing. first byte: '{'"`

`error parsing syslog stream`

[Nachtfalkeaw]

Hello again,

it looks like it was my fault. My Unifi Device was sending "syslog" and "netconsole" messages and for both I expected I can use the same loki.source.syslog. However "netconsole" is differnet than syslog and for that reason it was - I tjhink - dropped correctly.

Probably this would have been easier to identify if the log error message had more details - e.g. the raw message which arrived.

So my points 3, 4 and 5 remain:

1. (working, my fault)
2. (working, my fault)
3. Never drop messages, try to append it like configured before and if not possible to append then forward it, maybe add an additional internal label like __syslog_message_unparseable and add as many other internal labels as possible to make it possible for users in further steps like loki.process to map these messages. (loki.source.syslog: do not drop syslog messages if they can not be parsed. #2275)
4. Add the unparseable Log Line to the debug logs.
5. Add the RFC complaining to the logs.

[Nachtfalkeaw]

I think after the investigation it is a duplicate of this (#2275) and I will close it.