Create zone bundles from ZFS snapshots #4225
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Here is a snippet of the sled-agent log during this operation: I've tested this a bunch on my local machine, and the bundles look accurate. I would still like to test this against the zone which is most likely to create problems, ClickHouse. That one spews an absolute mountain of data into its log files, so it'll be a good check that this does actually prevent corruption. Here's a snippet of a succesful bundle: So we're still getting the rotated log files and all the command output. I'd also like to check that we get the archived bundles as well before merging this. |
- Fixes #4010 - Previously, we copied log files directly out of their original locations, which meant we contended with several other components: `logadm` rotating the log file; the log archiver moving the to longer-term storage; and the program writing to the file itself. This commit changes the operation of the bundler, to first create a ZFS snapshot of the filesystem(s) containing the log files, clone them, and then copy files out of the clones. We destroy those clones / snapshots after completing, and when the sled-agent starts to help with crash-safety.
bnaecker
force-pushed
the
bundle-logs-via-zfs-snapshot
branch
from
ada2189 to
fe1fc69
Compare
|
This is a really neat way to fix the consistency problem. |
|
Thanks Andy! Hope this strategy is more robust. I couldn’t read from the snapshot when I was experimenting manually, but I must have been missing something else. This will certainly make the code simpler, thank you!
|
jmpesp
approved these changes
Oct 10, 2023
- Use snapshots directly, no need for clones - Set properties on ZFS snapshots at creation time, avoiding possible crash inconsistency
|
Thanks for the suggestion @jmpesp, I've added the properties directly at snapshot creation time. @citrus-it could you please take a look too, if you get a chance? I took your suggestion, reading the from the snapshots directly without an intervening clone. |
|
I've confirmed that ClickHouse bundles can be made without any problems: I'd still like to test this on a real Gimlet before merging. The log archival process ignores the file-backed vdevs we use on my developer machine, so I want to make sure we get those correctly before closing this. I'll do that tonight, if I can snag a machine like |
citrus-it
reviewed
Oct 10, 2023
|
Ok, I've had the chance to test this a bunch on BRM44220001 # /opt/oxide/sled-agent/zone-bundle get --create oxz_switch
Created zone bundle: oxz_switch/79f7cf0a-fa6e-4ac2-b9e5-bf483cb4fbd5
BRM44220001 # tar -tf 79f7cf0a-fa6e-4ac2-b9e5-bf483cb4fbd5.tar.gz
metadata.toml
ptree
uptime
last
who
svcs
netstat
pfiles.24187
pstack.24187
pargs.24187
oxide-mgs:default.log
oxide-mgs:default.log.1697165100
oxide-mgs:default.log.1697134496
oxide-mgs:default.log.1697148899
oxide-mgs:default.log.1697166900
pfiles.24203
pstack.24203
pargs.24203
oxide-uplink:default.log
oxide-uplink:default.log.1697165075
oxide-uplink:default.log.1697166426
oxide-uplink:default.log.1697134361
oxide-uplink:default.log.1697134500
pfiles.24236
pstack.24236
pargs.24236
system-illumos-mg-ddm:default.log
system-illumos-mg-ddm:default.log.1697165100
system-illumos-mg-ddm:default.log.1697166518
system-illumos-mg-ddm:default.log.1697134386
system-illumos-mg-ddm:default.log.1697134501
pfiles.24293
pstack.24293
pargs.24293
oxide-dendrite:default.log
oxide-dendrite:default.log.1697165100
oxide-dendrite:default.log.1697134501
oxide-dendrite:default.log.1697148901
oxide-dendrite:default.log.1697166900
pfiles.24198
pstack.24198
pargs.24198
oxide-tfport:default.log
oxide-tfport:default.log.1697165091
oxide-tfport:default.log.1697134471
oxide-tfport:default.log.1697148872
oxide-tfport:default.log.1697166879And ClickHouse, which was the worst offender in terms of zone-bundle corruption caused by the process itself writing into the log file concurrently with the bundling: BRM44220001 # tar -tf 3f30b9ac-908c-4caa-93dd-136d57131d0f.tar.gz
metadata.toml
ptree
uptime
last
who
svcs
netstat
pfiles.6563
pstack.6563
pargs.6563
oxide-clickhouse:default.log
oxide-clickhouse:default.log.1697113799
oxide-clickhouse:default.log.1697084997
oxide-clickhouse:default.log.1697099398
oxide-clickhouse:default.log.1697128199
oxide-clickhouse:default.log.1697070601
oxide-clickhouse:default.log.1697051699
oxide-clickhouse:default.log.1697068801
oxide-clickhouse:default.log.1697165995
oxide-clickhouse:default.log.1697134500
oxide-clickhouse:default.log.1697148901
oxide-clickhouse:default.log.1697166896
pfiles.6588
pstack.6588
pargs.6588
oxide-clickhouse:default.log
oxide-clickhouse:default.log.1697113799
oxide-clickhouse:default.log.1697084997
oxide-clickhouse:default.log.1697099398
oxide-clickhouse:default.log.1697128199
oxide-clickhouse:default.log.1697070601
oxide-clickhouse:default.log.1697051699
oxide-clickhouse:default.log.1697068801
oxide-clickhouse:default.log.1697165995
oxide-clickhouse:default.log.1697134500
oxide-clickhouse:default.log.1697148901
oxide-clickhouse:default.log.1697166896Note that neither hits the tar directory checksum errors. Bundling all zones on the host also no longer hits that failure mode: |
|
This also fixes #4269. |
- Fixes #4269, so the rpool/zone dataset can be mounted in the GZ - Only fetch archived logs once - Fix logic for matching archived log file names
citrus-it
approved these changes
Oct 16, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
logadmrotating the log file; the log archiver moving the to longer-term storage; and the program writing to the file itself. This commit changes the operation of the bundler, to first create a ZFS snapshot of the filesystem(s) containing the log files, clone them, and then copy files out of the clones. We destroy those clones / snapshots after completing, and when the sled-agent starts to help with crash-safety.