Getting information from TTime fMilliSec member

[aleberti]

Hi all,

I have a question which is related to the one I posted in the ROOT forum, see https://root-forum.cern.ch/t/ttime-saved-to-a-ttree-in-root5-and-root6/54031 . So, in short, between ROOT5 and ROOT6, what is written out in ROOT files can be a bit different, in the specific case the fMilliSec -private- member of the TTime class is not written out in the same way. Therefore, one cannot access easily that member anymore with uproot if ROOT6 is used.

To give the possibility to test, I attach two ROOT files, one created with ROOT5.34.36 and one with ROOT6.24.08. Here I use a class called MTime, which has a TTime object as private member of the class.

root5_6_examples.zip

So when I try reading the fMilliSec values (I should get -37390935 and -28750935 respectively for the two events in the files) with uproot I get:

In [1]: import uproot

In [2]: uproot.__version__
Out[2]: '4.3.7'

In [3]: rootf5 = uproot.open("example_root5.root")

In [4]: rootf5["Events"]["MTime/fTime/fTime.fMilliSec"].array(library="np")
Out[4]: array([-37390935, -28750935])

In [5]: rootf6 = uproot.open("example_root6.root")

In [6]: rootf6["Events"].keys()
Out[6]:
['MTime',
 'MTime/MParContainer',
 'MTime/MParContainer/TObject',
 'MTime/MParContainer/TObject/fUniqueID',
 'MTime/MParContainer/TObject/fBits',
 'MTime/fMjd',
 'MTime/fTime',
 'MTime/fNanoSec']

In [7]: rootf5["Events"]["MTime/fTime"].array(library="np")
Out[7]: {'fTime.fMilliSec': array([-37390935, -28750935])}

In [8]: rootf6["Events"]["MTime/fTime"].array(library="np")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~/Software/miniconda3/envs/magic-cta-pipe-analysis-school-2023/lib/python3.8/site-packages/uproot/interpretation/numerical.py in basket_array(self, data, byte_offsets, basket, branch, context, cursor_offset, library)
    341         try:
--> 342             output = data.view(dtype).reshape((-1,) + shape)
    343         except ValueError as err:

ValueError: When changing to a larger dtype, its size must be a divisor of the total size in bytes of the last axis of the array.

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
<ipython-input-9-6aec0c18a45a> in <cell line: 1>()
----> 1 rootf6["Events"]["MTime/fTime"].array(library="np")

~/Software/miniconda3/envs/magic-cta-pipe-analysis-school-2023/lib/python3.8/site-packages/uproot/behaviors/TBranch.py in array(self, interpretation, entry_start, entry_stop, decompression_executor, interpretation_executor, array_cache, library)
   2206                         ranges_or_baskets.append((branch, basket_num, range_or_basket))
   2207
-> 2208         _ranges_or_baskets_to_arrays(
   2209             self,
   2210             ranges_or_baskets,

~/Software/miniconda3/envs/magic-cta-pipe-analysis-school-2023/lib/python3.8/site-packages/uproot/behaviors/TBranch.py in _ranges_or_baskets_to_arrays(hasbranches, ranges_or_baskets, branchid_interpretation, entry_start, entry_stop, decompression_executor, interpretation_executor, library, arrays, update_ranges_or_baskets)
   3491
   3492         elif isinstance(obj, tuple) and len(obj) == 3:
-> 3493             uproot.source.futures.delayed_raise(*obj)
   3494
   3495         else:

~/Software/miniconda3/envs/magic-cta-pipe-analysis-school-2023/lib/python3.8/site-packages/uproot/source/futures.py in delayed_raise(exception_class, exception_value, traceback)
     34     Raise an exception from a background thread on the main thread.
     35     """
---> 36     raise exception_value.with_traceback(traceback)
     37
     38

~/Software/miniconda3/envs/magic-cta-pipe-analysis-school-2023/lib/python3.8/site-packages/uproot/behaviors/TBranch.py in basket_to_array(basket)
   3435             basket_arrays = branchid_arrays[branch.cache_key]
   3436
-> 3437             basket_arrays[basket.basket_num] = interpretation.basket_array(
   3438                 basket.data,
   3439                 basket.byte_offsets,

~/Software/miniconda3/envs/magic-cta-pipe-analysis-school-2023/lib/python3.8/site-packages/uproot/interpretation/numerical.py in basket_array(self, data, byte_offsets, basket, branch, context, cursor_offset, library)
    342             output = data.view(dtype).reshape((-1,) + shape)
    343         except ValueError as err:
--> 344             raise ValueError(
    345                 """basket {} in tree/branch {} has the wrong number of bytes ({}) """
    346                 """for interpretation {}

ValueError: basket 0 in tree/branch /Events;1:MTime/fTime has the wrong number of bytes (28) for interpretation AsStridedObjects(Model_TTime_v2)
in file example_root6.root

In the TTime class docs (https://root.cern/doc/master/TTime_8h_source.html#l00076), I see that in principle one can get the fMilliSec value by using the long long operator. Is there a way to do something similar using uproot?

Thanks in advance!

[jpivarski]

Yes, the structures in the two files are different, but it might be just that the ROOT 5 one was written with full splitting (splitLevel=99) and the ROOT 6 one was written without splitting (splitLevel=0).

>>> import uproot
>>> f5 = uproot.open("example_root5.root")
>>> f6 = uproot.open("example_root6.root")

>>> f5["Events/MTime/fTime"].show()
name                 | typename                 | interpretation                
---------------------+--------------------------+-------------------------------
fTime                | TTime                    | AsGroup(<TBranchElement 'fTime
fTime.fMilliSec      | int64_t                  | AsDtype('>i8')

>>> f6["Events/MTime/fTime"].show()
name                 | typename                 | interpretation                
---------------------+--------------------------+-------------------------------
fTime                | TTime                    | AsStridedObjects(Model_TTime_v

The f5 one has an fTime TBranch with a subbranch named fTime.fMilliSec (64-bit integers) and the f6 one only has an fTime TBranch whose type is the whole struct.

The struct doesn't contain anything other than that 64-bit integer (long long).

>>> f6.file.streamer_named("TTime").show()
TTime (v2)
    fMilliSec: long long (TStreamerBasicType)

When we look at the raw bytes of the first entry in f6,

>>> f6["Events/MTime/fTime"].debug_array(0)
array([ 64,   0,   0,  10,   0,   2, 255, 255, 255, 255, 253, 197, 117,
       169], dtype=uint8)

we see the correct value

>>> import numpy as np
>>> np.array([255, 255, 255, 255, 253, 197, 117, 169], "u1").view(">i8")
array([-37390935])

preceded by 6 extra bytes. This is a header for the TTime struct. In the first 4 bytes, 64 is a high bit saying that 10 is the size of the object (which is true) and in the next 2 bytes, 2 is the class version of this TTime.

When Uproot tried to come up with an Interpretation for this data, it should have included ("@num_bytes", ">u4"), ("@instance_version", ">u2") in the NumPy dtype. It's missing these headers:

>>> f6["Events/MTime/fTime"].interpretation.from_dtype
dtype([('fMilliSec', '>i8')])

The interpretation ought to be

>>> interp = uproot.AsDtype([("@num_bytes", ">u4"), ("@instance_version", ">u2"), ('fMilliSec', '>i8')])

>>> f6["Events/MTime/fTime"].array(interp, library="np")
array([(1073741834, 2, -37390935), (1073741834, 2, -28750935)],
      dtype=[('@num_bytes', '<u4'), ('@instance_version', '<u2'), ('fMilliSec', '<i8')])

from which you can pull out the fMilliSec.

>>> f6["Events/MTime/fTime"].array(interp, library="np")["fMilliSec"]
array([-37390935, -28750935])

It's always been difficult to figure out the right Interpretation for a TBranch, particularly with including or not including these headers. Some TBranches have headers at all levels, some have them at some levels, and some don't have them at all, and it's unclear (to me) where that information can be found.

We could convert this into a bug report (the uproot.interpretation.identify.interpretation_of function decided that this one has no headers when it actually does have headers) in the hope that someone will be able to think of a way to determine this information once and for all.

In the meantime, though, you can pass the interp above into the TBranch.array method manually to get what you want.

[aleberti]

Thank you @jpivarski ! Regarding the split level, these files were created by me, so the same split level was used for both.

Regarding the (temporary) solution i.e. using the interpretation you provided, it should be fine for my purposes.

For the bug report, should I do it or do you take care? I guess mine is just a specific case of a more general problem.

Also, maybe you can also take another look at the conversation in the ROOT forum (https://root-forum.cern.ch/t/ttime-saved-to-a-ttree-in-root5-and-root6/54031/4). It seems that I stumbled upon a bug which may affect other classes in ROOT.

[jpivarski]

I just read the discussion in the ROOT Forum, and I understand now that this class diverges from the normal pattern by being an old class dictionary. So interpretation_of couldn't be expected to discover that the header is needed. In that case, we can handle it as a special case; the same is true of other old classes. Issue #861 is a reminder to do that.

There is a more general problem: Interpretations with missing or extra headers are common reasons for not being able to read a TBranch (without a manual Interpretation). It seems that this is not one of those examples... or maybe the reason this is a recurring problem is because there are a lot of "old" classes? If so, then maybe issue #861 is not the best solution, as it would lead to a lot of duplicated code; maybe we need a list of classes that have headers, despite the general rule that new-style classes do not...