How to measure size (in bytes) of a column read from a ROOT file?

[kondratyevd]

Hi, I would like to read a single column from a ROOT file using uproot and measure its size in bytes. I am interested in both the disk size taken by the column in the ROOT file, and the RAM usage when it is loaded into memory; as I understand, those will not be the same due to compression in ROOT files.

What is the best way to achieve this?

I am currently using uproot==4.3.7, but either uproot4 or uproot5 solution would work.

[jpivarski]

After opening the file and selecting the TBranch like this:

file = uproot.open("PATH/TO/FILE.root")
tree = file["TREE_NAME"]
branch = tree["BRANCH_NAME"]

(replacing everything in UPPERCASE), you can get the compressed size on disk with branch.compressed_bytes, the uncompressed size with branch.uncompressed_bytes, and you can read the data into an array with array = branch.array() and then look at the array's number of bytes with array.nbytes.

If you pass library="np" to the array function, it will be a NumPy array, for which nbytes counts the size of the single buffer in memory only (so if it's not numerical data and it gets materialized as Python objects in an array with dtype=object, the nbytes will only count the pointers to those Python objects, not the data itself).

If you don't pass anything or pass library="ak" to the array function, it will be an Awkward Array, for which nbytes adds up the sizes of all of its constituent buffers. If the data type in the TBranch is simple, the NumPy nbytes would be equal to the Awkward nbytes. You can see the internal structure of an Awkward Array (to know how many and which buffers it's made of) by asking for array.layout.

The size is affected by more than just compression. The TBranch data is stored in chunks called TBaskets, and each TBasket has a header, which is usually less than 100 bytes and some of it sits outside of the compressed part. The values returned by the compressed_bytes and uncompressed_bytes properties are numbers in the ROOT file, ROOT's own reporting, and the array returned by the array function is made by decompressing and interpreting each TBasket, followed by concatenating the parts. If you need to dig deeper, you can look at branch.num_baskets and similar functions to examine each TBasket.

[kondratyevd]

@jpivarski thanks a lot for such a detailed answer! I have a couple of follow-up questions:

1. Are there any caveats here to keep in mind in case of nested branches (leaves?), e.g. tree["Muon"]["pt"], or is this equivalent to the case where branches are not nested?
2. In your opinion, what could be the minimal example of accessing all of the actual data in a given column? Naively, I would use some simple operation such as np.mean().

[jpivarski]

1. No caveats for nested TBranches. The TBranch nesting is just how the metadata are organized in the TTree object, and the data they point to in TBaskets is a separate thing. In fact, most TBranches that contain TBranches don't have any associated data. (TBranches representing TClonesArray is one exception, but this is usually true.) The TLeaf class is purely metadata and has no associated data; despite these names, a more useful way to think about them is:

• TBranch: one array/column of data that can be selectively read in chunks (TBaskets). A TBranch can contain arbitrarily many TBranches, and if it contains zero TBranches, then in normal CS tree terminology, it's a "leaf."
• TLeaf: metadata that describes the data type, either exclusively (e.g. TLeafI means that the data type is int32_t) or non-exclusively, if part of the data type is described by TStreamerInfo, or in the TBranch's title, etc. Multiple TLeaves corresponds to NumPy's concept of a structured array, which in C is an array of structs.

2. As soon as you call uproot.TBranch.array or uproot.TTree.arrays (or similar), the array is loaded into memory—all values have been accessed. You don't have to compute anything on them.

Since this is Python, nothing gets optimized away: even if you request the data and then do nothing with it, Uproot will still go through the process of reading the data from disk, decompressing, interpreting, and forming an array. In compiled languages, I often sum over the values and print them out to ensure that the compiler doesn't remove the code that I want to performance-test, but that's not necessary here. Still, if you want to do it anyway, summation is one of the least expensive things you can do.¹

Footnotes

1. A compiled + operation is approximately 1 clock tick, approximately 1 nanosecond, and if you do a reduction, np.sum instead of np.add, then no new arrays need to be created to store the output. Division is maybe 20 times more expensive, but in np.mean it's only done once, so it hardly matters. What's going to matter more than anything else is paging the data from RAM into the CPU. ↩