Perf of reading a zarr

[mdespriee]

Hey there,

My question is a bit out of the scope of zarr, but I wonder if any of you had this issue, and managed to improve the situation:

I have an app reading from zarr files, and I profiled a bit the perf.
To my surprise a vast amount of time is spent in parsing timestamps in boto !

I guess all this is buried deep in fsspec/s3fs implem, but I wonder what kind of timestamps it's about (file timestamps ?), and if there are some option to alleviate this ? Caches ? Disable some feature related to these timestamps ?

Thanks for your insights

[JackKelly]

This is very interesting, thank you for sharing! We are actively exploring ways to improve the performance of Zarr implementations.

It looks like the culprit is dateutil.parser._parse.parser.parse(). If you haven't done so already, it might be worth submitting an issue to the dateutil GitHub issue queue?

(Although, thank you for flagging this up! This is definitely relevant for Zarr. And relevant to my own personal agenda of trying to find bits of the Python scientific stack that are ripe for re-implementing in Rust 🙂. Parsing dates sounds like something that could benefit a lot from being done in a compiled language)

[rabernat]

There are likely way simpler and easier ways to optimize this rather than re-implementing core Python libraries like dateutil in Rust. (Not part of the scientific stack.)

We can start by asking, "why is boto spending so much time parsing timestamps"? Note that this has nothing to do wiht the Zarr data itself. It's part of the overhead of talking to S3.

@mdespriee - would you be able to share the code you used to generate this profile?

[mdespriee]

@rabernat I'll try to make a reproducer.

[mdespriee]

@JackKelly @rabernat I forgot to mention something in the context. This zarr is built by continuous appending, so I have dozens of slices on time dimension. I think this amplifies the number of round-trips to s3, and makes the timestamp parsing more prominent.

[mdespriee]

The root cause is pydata/xarray#8488.
The fact that groups are entirely scanned, when they grow the number of round trips to s3 explodes, which make the slower part of this code to surface (timestamps parsing).

I rewrote my app without using zarr groups, and the behavior is in line with what I expect.