Reading multi-source compressed JSONL files #17161
Conversation
shrshi
added
cuIO
|
Status update: Implementing benchmark for compressed JSONL files in #17219 |
|
Plot of throughput of Benchmark raw output: <style> </style>
Without the custom data source for compressed inputs, the parser errors out for all of the above data sizes. I suspect that the heuristic for compressed inputs in the previous implementation falls short without the retry policy, but some more investigation is required to understand the failure, especially for the smaller data sizes. |
I would expect a performance improvement since we need not run the tokenizer + tree algorithms repeatedly for each compressed file (up to a ~2GB total uncompressed source size limit per batch). I can modify the benchmark in #17219 to measure the exact impact. |
There was a problem hiding this comment.
wouldn't mind merging this file and comp.hpp. Definitely not required in this PR :)
There was a problem hiding this comment.
Will open a follow-on PR for this
| @@ -980,27 +980,27 @@ __device__ int parse_gzip_header(uint8_t const* src, size_t src_size) | |||
| { | |||
| uint8_t flags = src[3]; | |||
| hdr_len = 10; | |||
| if (flags & GZIPHeaderFlag::fextra) // Extra fields present | |||
| if (flags & detail::GZIPHeaderFlag::fextra) // Extra fields present | |||
There was a problem hiding this comment.
looks like another file missing namespace detail ? Not for this PR :D
There was a problem hiding this comment.
Ah yep, it looks like all the header files in src/io/comp need to be moved to detail namespace -
brotli_dict.hpp
unbz2.hpp
nvcomp_adapter.cuh (?)
nvcomp_adapter.hpp (?)
gpuinflate.hpp
brotli_tables.hpp
Haha yes, will defer this to a follow-on PR :D
|
/merge |
Depends on #17161 for implementations of compression and decompression functions (`io/comp/comp.cu`, `io/comp/comp.hpp`, `io/comp/io_uncomp.hpp` and `io/comp/uncomp.cpp`) Adds support for writing GZIP- and SNAPPY-compressed JSON to the JSON writer. Verifies correctness using a parameterized test in `tests/io/json/json_writer.cpp` Authors: - Shruti Shivakumar (https://github.com/shrshi) - Vukasin Milovanovic (https://github.com/vuule) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Karthikeyan (https://github.com/karthikeyann) - Vukasin Milovanovic (https://github.com/vuule) URL: #17323
Depends on #17161 for implementations of compression and decompression functions (`io/comp/comp.cu`, `io/comp/comp.hpp`, `io/comp/io_uncomp.hpp` and `io/comp/uncomp.cpp`)\ Depends on #17323 for compressed JSON writer implementation. Adds benchmark to measure performance of the JSON reader for compressed inputs. Authors: - Shruti Shivakumar (https://github.com/shrshi) - Muhammad Haseeb (https://github.com/mhaseeb123) Approvers: - MithunR (https://github.com/mythrocks) - Vukasin Milovanovic (https://github.com/vuule) - Karthikeyan (https://github.com/karthikeyann) - Muhammad Haseeb (https://github.com/mhaseeb123) URL: #17219
Description
Fixes #17068
Fixes #12299
This PR introduces a new datasource for compressed inputs which enables batching and byte range reading of multi-source JSONL files using the reallocate-and-retry policy. Moreover. instead of using a 4:1 compression ratio heuristic, the device buffer size is estimated accurately for GZIP, ZIP, and SNAPPY compression types. For remaining types, the files are first decompressed then batched.
TODO: Reuse existing JSON tests but with an additional compression parameter to verify correctness.Handled by #17219, which implements compressed JSON writer required for the above test.Multi-source compressed input tests added!
Checklist