Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDF5 filters (compression) #1644

Open
wants to merge 10 commits into
base: dev
Choose a base branch
from
+398 −83
Open

HDF5 filters (compression) #1644

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ff74a62
Preparation work: Refactoring
franzpoeschel Jul 2, 2024
73cad12
Basic compression/filtering in HDF5
franzpoeschel Jul 2, 2024
fcf69e4
Configure generic filters via JSON object
franzpoeschel Jul 3, 2024
3ccd238
Full support for the set_filter API
franzpoeschel Jul 3, 2024
718b3ab
Fix: captured structured bindings are a C++20 extension
franzpoeschel Jul 3, 2024
fc40d70
Refactoring to satisfy the Github bot
franzpoeschel Jul 3, 2024
8171153
Fix includes
franzpoeschel Jul 3, 2024
28974fd
Switch to JSON config for NVidia compiler's benefit
franzpoeschel Jul 3, 2024
1554707
Verbose CI debugging lets goo
franzpoeschel Dec 9, 2024
bd479f4
Revert "Verbose CI debugging lets goo"
franzpoeschel Dec 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion examples/7_extended_write_serial.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,15 @@ int main()
}
]
}
},
"hdf5": {
"dataset": {
"chunks": "auto",
"permanent_filters": {
"id": "fletcher32",
"flags": "optional"
}
}
}
})END";
d.options = datasetConfig;
Expand All @@ -144,7 +153,27 @@ int main()
d = io::Dataset(dtype, mpiDims);
electrons["positionOffset"]["x"].resetDataset(d);

auto dset = io::Dataset(io::determineDatatype<uint64_t>(), {2});
auto dset = io::Dataset(
io::determineDatatype<uint64_t>(),
{2},
R"(
{
"hdf5": {
"dataset": {
"chunks": "auto",
"permanent_filters": [
{
"aggression": 5,
"type": "zlib"
},
{
"flags": "MANDATORY",
"id": "shuffle"
}
]
}
}
})");
electrons.particlePatches["numParticles"].resetDataset(dset);
electrons.particlePatches["numParticlesOffset"].resetDataset(dset);

Expand Down
Loading
Loading