Add LGDO format conversion utilities

[gipert]

We should implement a method for each LGDO to convert underlying data to third-party formats like NumPy, Pandas, AwkwardArray. I'm thinking about something like:

lgdo_obj.convert(fmt="pandas.DataFrame", copy=False)

Where fmt could take pandas.DataFrame, numpy.ndarray, awkward.Array.

This way, we would store the conversion code along with the LGDO implementation and make it easier to jump between data representations (like in load_nda(), load_pd(), build_tcm(), the DataLoader, etc).

We need of course to make a distinction between copy and zero-copy conversions.

[gipert]

I propose then to deprecate load_nda() (and load_pd()) in favor of:

store.read_object("obj", "file.lh5").convert(fmt="numpy.ndarray")

Which would return the same.

This new convert() functions should also handle units at some point. With numpy.ndarray, we could just use Pint's NumPy support and that should work. With pandas.DataFrame, we could use pint-pandas – but I'm not sure whether the package is fully functional.

[MoritzNeuberger]

I am confused about how the return type annotation would work in this case. Can you have a single function with multiple types of output depending on the input parameters?

[gipert]

Yes, it would look like this:

def convert(...) -> pandas.DataFrame | numpy.NDArray | ...:
    pass

[MoritzNeuberger]

Over the last few days, I have been playing around with implementing this feature. For the most part, it is straightforward, although a few questions arose:

VectorOfVectors:

• To convert it to a numpy.ndarray, I now first convert it to an aoesa using to_aoesa and use its convert function. to_aoesa also uses np.empty to implement the nda, and when preserve_dtype is set to True we also have the problem that the previously empty entries are filled with random values. I assume it is not preferable to have preserve_dtype set to False in which case these values would be set to nan.

Struct/Table:

• The implementation in numpy.ndarray is not easy either. For now, I solved it by returning a dict containing the key and value entries of Struct/Table in two separate numpy arrays. What would be a better way to implement this?

WaveformTable/encoded data:

• Does it need convert?

copy:

• I have implemented this option wherever possible. That is, always for pandas.DataFrame and when necessary for numpy.ndarrays. AFAIK awkward arrays usually do not copy?

ToDos:

• Figure out how to implement units with pint. Is this possible with awkward arrays?
• Write tests

[MoritzNeuberger]

I think it would be easier to see in code. I will prepare a PR with the status as it is at the moment.