Obey Syslog size limits #9
Conversation
jscheid
commented
Dec 19, 2014
jscheid
commented
- Ensure default tag is max 32 characters long
- Split long lines into chunks so they aren't truncated by Syslog max message size
RFC 3164 section 4.1.3 specifies that the tag MUST NOT exceed 32 characters. This is enforced by https://github.com/eric/syslog_protocol/blob/001000ffe27a4557c3ec312b9c3c50385e6a923b/lib/syslog_protocol/packet.rb#L48-L50 Ensure that the default tag generated from the program name and PID is within these bounds.
RFC 3164 section 4.1 (also 4.3.2 and 6.1) specifies that the packet length MUST NOT exceed 1024 bytes. This is enforced by https://github.com/eric/syslog_protocol/blob/001000ffe27a4557c3ec312b9c3c50385e6a923b/lib/syslog_protocol/packet.rb#L16-L21 by truncating the packet to this length (by default). Ensure that we don't lose parts of long log messages by splitting them into chunks accordingly and sending each chunk in an individual packet. Prefix continuation chunks with a configurable prefix (default: ellipsis followed by space) so they can be identified as such more easily. Also make the maximum packet size configurable; while RFC 3164 specifies a maximum packet size of 1024, some implementations (e.g. rsyslog) support larger sizes.
|
Hi, I don't know if you've seen it but there is some more information in the commit messages. In retrospect, this should probably keep the default behaviour of no chunking for backwards compatibility. I'm happy to make that change if you agree. Do you have any other concerns? |
|
@jscheid for context, can you add the back story which led you to want this? That is, what led you to say "Hey this should be added and is worth my time to write"? |
|
@troy, Sure! So we're running remote_syslog_logger in a PCI-DSS v3 compliant environment. PCI-DSS requires a full audit trail and we're taking it seriously, therefore truncated or dropped log records are simply not good enough. (If you're curious, yes we're logging via TCP not UDP.) Of course, most of our log records are, like everybody else's, much shorter than our server limits, but every now and then we're logging something biggish (say, an email body or a public key or something) and it's good to have it in its entirety. Of course we can always increase server-side limits, but we can't make them infinite and so we could never be quite sure that all data will make it. Chopping records into chunks is a nice and easy way to work around this problem. That story being told, I can't imagine many people outside of the requirements of PCI compliance saying "oh, it's ok that this long log record didn't make it, it probably wasn't that important anyway". I'd wager that when people ask their systems to log something, they expect it to be logged in its entirety. (We could wrap your code in some custom pre-processor of course, but I felt like your project -- being a convenience wrapper around syslog_protocol, so to speak -- was a good home for it.) Regarding the 32-character limit for the tag, it's pretty straightforward. If I remember correctly, just make a script called Hopefully this explains our motivation better, and I hope this change can benefit your other users as well. (PS we've been using this patch in production for eight months or so now.) |
Tested to work with activesupport 5.2.2.
