Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add raw file backend #991

Merged
merged 62 commits into from
Nov 9, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
2362e5e
Add raw file backend
mkeeter Oct 3, 2023
b4a6f68
Update comments
mkeeter Oct 6, 2023
a3bc2da
Bump version
mkeeter Oct 6, 2023
a7461af
Implement fast migration
mkeeter Oct 9, 2023
3ae42f4
Further optimization
mkeeter Oct 9, 2023
8a71b11
Fix ordering
mkeeter Oct 9, 2023
1e02809
Preallocate during migration
mkeeter Oct 9, 2023
e247dfb
Better definition of EXTENT_META_SQLITE
mkeeter Oct 9, 2023
0f5ddca
Updates per review
mkeeter Oct 12, 2023
837ec44
Rephrase
mkeeter Oct 12, 2023
01dc06b
Remove outdated comment
mkeeter Oct 12, 2023
5400ae8
Clarify read-only snapshot situation
mkeeter Oct 12, 2023
b52ca74
extent-read-done probe is elsewhere
mkeeter Oct 12, 2023
cb9a772
Make active_context always valid (instead of optional)
mkeeter Oct 17, 2023
3ffbc0a
Shuffle array order
mkeeter Oct 17, 2023
e14c5cd
Almost fixed reading file from disk
mkeeter Oct 17, 2023
10ce166
Fix migration
mkeeter Oct 17, 2023
a71b807
Preparing to coalesce context writes
mkeeter Oct 17, 2023
4e5c823
Write metadata in (semi) bulk
mkeeter Oct 17, 2023
4412b84
Add DTrace probe for write count
mkeeter Oct 17, 2023
600be6e
Add unit test for serialized sizes
mkeeter Oct 18, 2023
867fd56
Shrink metadata sizes
mkeeter Oct 18, 2023
02ab716
Coalesce block context reads
mkeeter Oct 18, 2023
ba02080
Remove seeking in favor of pread
mkeeter Oct 18, 2023
badd2f9
Implement context defragmentation
mkeeter Oct 18, 2023
5563c2a
Only defragment if we're doing extra work
mkeeter Oct 19, 2023
0ce8cab
Only hash the file when slot is ambiguous
mkeeter Oct 19, 2023
e7bae58
Fix indentation
mkeeter Oct 19, 2023
de5f1a0
Encode active context in metadata region
mkeeter Oct 20, 2023
7bad3dd
Improvements and bug fixes
mkeeter Oct 20, 2023
e1e28b1
Move metadata to the very end of the file
mkeeter Oct 23, 2023
e5f1d22
Only get block contexts once (hopefully)
mkeeter Oct 23, 2023
caa5b50
Only rehash if this isn't read-only
mkeeter Oct 24, 2023
3667b3a
Move file size to a function
mkeeter Oct 26, 2023
863a1dc
Add defragmentation test
mkeeter Oct 26, 2023
b84cb78
Move migration file knowledge to RawInner
mkeeter Oct 26, 2023
ea777ca
Add RawLayout class to manage layout
mkeeter Oct 26, 2023
c3ed14d
Add docstring
mkeeter Oct 27, 2023
9642e66
Add support for alternate backends to integration tests
mkeeter Oct 27, 2023
786d498
Add raw-on-SQLite integration test
mkeeter Oct 27, 2023
bd91778
Add migration test
mkeeter Oct 27, 2023
26b6b6d
serialize_into should be infallible
mkeeter Oct 27, 2023
88cb382
Make open() and create() consistent (taking directory)
mkeeter Oct 27, 2023
7b883ba
Fix last failing test
mkeeter Oct 27, 2023
8583774
More specific errors
mkeeter Oct 27, 2023
ade7392
Remove debug stuff
mkeeter Oct 30, 2023
7c6c217
Fix comment
mkeeter Oct 30, 2023
ec5cb52
Typo
mkeeter Oct 30, 2023
856c18b
More comments
mkeeter Oct 30, 2023
b6ef63a
Add some assertions
mkeeter Oct 30, 2023
dc81096
Add assertion
mkeeter Oct 30, 2023
e94ea4e
More defragment tests
mkeeter Oct 30, 2023
fb59d22
Check that defragment is only called on a clean extent
mkeeter Nov 3, 2023
b66519d
Add multiple-write-to-same-block test
mkeeter Nov 7, 2023
57a52b6
Post-rebase fixes
mkeeter Nov 7, 2023
a8b3705
Restore buf after `get_active_contexts`
mkeeter Nov 7, 2023
416094e
Update comment
mkeeter Nov 7, 2023
2a5ed27
Add some handling of partial pread/pwrite(v)
mkeeter Nov 7, 2023
971ed1e
Explain dirty / context slot mismatch
mkeeter Nov 7, 2023
868bca4
Remove buf (it doesn't make things faster and uses up RAM)
mkeeter Nov 7, 2023
0d4f573
Simplify dirty context slot tracking to reduce RAM
mkeeter Nov 7, 2023
8fe775b
Remove more premature optimization
mkeeter Nov 7, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 13 additions & 1 deletion common/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,15 @@ pub enum CrucibleError {

#[error("Invalid downstairs replace {0}")]
ReplaceRequestInvalid(String),

#[error("missing context slot for block {0}")]
MissingContextSlot(u64),

#[error("metadata deserialization failed: {0}")]
BadMetadata(String),

#[error("context slot deserialization failed: {0}")]
BadContextSlot(String),
}

impl From<std::io::Error> for CrucibleError {
Expand Down Expand Up @@ -386,7 +395,10 @@ impl From<CrucibleError> for dropshot::HttpError {
| CrucibleError::SubvolumeSizeMismatch
| CrucibleError::UpstairsAlreadyActive
| CrucibleError::UpstairsDeactivating
| CrucibleError::UuidMismatch => {
| CrucibleError::UuidMismatch
| CrucibleError::MissingContextSlot(..)
| CrucibleError::BadMetadata(..)
| CrucibleError::BadContextSlot(..) => {
dropshot::HttpError::for_internal_error(e.to_string())
}
}
Expand Down
2 changes: 1 addition & 1 deletion common/src/region.rs
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ impl RegionDefinition {
}
}

#[derive(Deserialize, Serialize, Clone, Debug, PartialEq)]
#[derive(Serialize, Clone, Debug, PartialEq)]
pub struct RegionOptions {
/**
* The size of each block in bytes. Must be a power of 2, minimum 512.
Expand Down
2 changes: 2 additions & 0 deletions downstairs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ hex.workspace = true
http.workspace = true
hyper-staticfile.workspace = true
hyper.workspace = true
itertools.workspace = true
libc.workspace = true
mime_guess.workspace = true
nix.workspace = true
Expand Down Expand Up @@ -69,3 +70,4 @@ version_check = "0.9.4"
asm = ["usdt/asm"]
default = []
zfs_snapshot = []
integration-tests = [] # Enables creating SQLite volumes
Loading
Loading