Downstairs logs are not getting rotated/archive correctly #972
Assignees
Milestone
Comments
|
I believe the current log archival interval is 10 minutes. I think this process, and the use of logadm to rotate files, doesn’t care at all about the size of the files. It’ll happy rotate and then archive an empty file. Is it possible that these programs emit no long messages at all in some 10 minute periods? If so, I think this might be working as “intended”, though that’s not to say we want all these empty files hanging around.
|
|
Yeah, the downstairs might not emit messages for weeks :) The problem I'm seeing is that the original first 10 lines that the downstairs prints at startup will be logged, but I |
|
Possibly the same issue reported here: oxidecomputer/omicron#4161 |
|
I think I must be missing something. Is the issue that the log files don't exist, or that they do not contain the log messages you expect? The issue you linked only affects zone-bundling, and not the actual rotation or archival of the log files. The point I was trying to make above is that I expect most of these files to be empty. The logs are rotated, and then archived, every 10 minutes. If there are no log messages in a 10-minute period then (1) an empty log file will be archived, and (2) it's possible no new messages will show up in the current log file. That would also mean there's no *.1, *.2 file etc, because they're archived before the rotation process would need to make that many files. |
|
Ah, sorry I'm not explaining this issue very well. And, I don't know exactly which subsystem this problem is yet either. The problem is that, on initial startup, every downstairs will put out 10 lines or so of log messages, then go quiet. I don't always see that for my downstairs logs, sometimes I can find a log file with those initial 10 |
|
This may be a case of oxidecomputer/omicron#3860. If the downstairs was created shortly before an update, those log lines will be gone forever. Here is an example of a Nexus zone on Oct 25, when an upgrade happened around 21:00-23:00: The bottom log lines of the Oct 25 20:46 log: The head of the log in the next one: |
|
Checking on the current build on dogfood, and I see that the original downstairs log is now part of every archive. You can see in this sample of the archive directory. Every "new" downstairs log archive has contents which is |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Something is preventing downstairs log rotation from working correctly.
When a downstairs starts, there are always about 10 lines of code that are printed.
After that, it's pretty quiet and in general will only log if the upstairs disconnects, or
there is a repair that happens.
However, if I look on the crucible zones on dogfood, there are no *.0 *.1 files, and many
of the current smf log files are empty:
Some of the downstairs are no longer running, but even the ones that are, some have zero length files:
The text was updated successfully, but these errors were encountered: