Update vov description for flattened_data other than array<1>

[jasondet]

No description provided.

[jasondet]

@oschulz @gipert I've been wanting for a long time to update the vov spec to allow for flattened data other than array<1>. Please review this suggested change where we simply specify that it can be any array-like datatype.

[oschulz]

The flattened data should always be a one-dimensional vector, I think. If we want to support vectors of multi-dimensional arrays of non-equal size, we need an additional dataset that provides the size of dim 2 to n of each member arrays (in Julia, in ArraysOfArrays.VectorOfArrays we use an additional vector kernel_size for this).

[gipert]

@jasondet what is again the use case you have in mind?

I'm not sure your proposed changes include this, but I would like to have an additional dimension in VectorOfVectors, to store, in the event tier, data for each SiPM pulse in a waveform:

[ # event dimension
  [ # channel dimension
    [1, 2, 3, 4], # pulse in channel dimension
    [5, 6, 7, 8, 9]
  ],
  [
    [1, 2, 3, 4],
    [5, 6, 7, 8, 9],
    [0]
  ],
]

I also believe, like @oschulz says, that this should not be represented by a structure of nested LGDOs, but should rather be a single LGDO. Would also make it easier to read in as an Awkward Array in legend-pydataobj (we could then rewrite VectorOfVectors to wrap it, as I wanted to do).

@oschulz do you have a proposal in mind?

[gipert]

For reference, this is how Awkward folks recommend writing arrays to disk: https://awkward-array.org/doc/main/reference/generated/ak.to_buffers.html#ak-to-buffers

[oschulz]

I'm not sure your proposed changes include this, but I would like to have an additional dimension in VectorOfVectors, to store, in the event tier, data for each SiPM pulse in a waveform

So do you want vectors-of-vectors-of-vectors (this we can do already), or a vector of two-dimensional arrays of varying size (this we don't cover yet)?

[gipert]

vectors-of-vectors-of-vectors. Do we really support this already? How? I'm confused

[oschulz]

vectors-of-vectors-of-vectors. Do we really support this already? How? I'm confused

Yes, we do - the Julia event-tier files use it (multiple hits in multiple LAr-channels in multiple events). It worked almost out of the box (had to do one small bugfix).

The LH5 datatype is simply "array<1>{array<1>{array<1>{real}}}" and flattened_data is a vector-of-vectors with datatype "array<1>{array<1>{real}}".

So our vectors-of-vectors are just "naturally" nestable.

[oschulz]

To clarify this a bit more: A vector-of-vectors-of-vectors, in this scheme (and I think this is natural) is a vector-of-(vectors-of-vectors). And so is can simply be constructed using a vector-of-vectors as it's flattened content. In Julia, we use ArraysOfArrays.VectorOfArrays as the in-memory representation, which supports this kind of nesting out of the box, without having a single special line of code for it (this is why think it's the natural approach, with a generically written vector-of-vectors it "just works").

[gipert]

Yes, we do

Well, it's actually not documented: https://legend-exp.github.io/legend-data-format-specs/dev/hdf5/#Vector-of-vectors.

Anyways, i will put this legend-pydataobj development in my ToDo list. We need to switch to Awkward arrays for this.

[oschulz]

Well, it's actually not documented

True, the example there was only for vector-of-vector-of-reals. I've added how nesting works to the documentation now.

[gipert]

Nice thanks!

[oschulz]

@jasondet should we close this for now, then?

[jasondet]

yeah should be okay thx oli