Eager/early serialize of components to arrow in Rust and C++

[Wumpf]

As we'll soon introduce tagged components and simple multi-datatype components, it gets harder and harder to represent Archetypes (and concrete ComponentBatches) as collection of concrete types.
Let's take the example of a generalized rotation component/archetype field which may be represented by various datatypes: we no longer can store concrete types on an archetype and have to type-earse them right away instead.
Note that this way C++ and Rust get much closer to the Python SDK in this regard.

This fits very well into our desire to get rid of concrete component types in the SDK languages which today almost always take the form of struct ComponentType(pub datatypes::TheDataType) together with myriad of constructors, trait impls and utilities. I.e. a lot of forwarding code.
Eager serialization allows us to implement component semantics on archetypes instead with concrete construction methods. E.g. with_quaternion and with_axis_angle would both populate the multi-datatype rotation component which gets tagged appropriately.
When logging raw component batches/columns this would become more explicit as you're expected to supply a datatype array/collection together with the appropriate component tag (which will still be provided by the SDK, but more in registry fashion rather a class/struct per component). This follows the exact same mechanism of how an archetype construct its internal ComponentBatches.

A drawback of this approach is that most accesses of archetypes requires deserialization back into the source datatypes which can be cumbersome in some cases. However, this is what we expect to do when a user reads back data from the store, so this is something that may soon become common-place anyways.

Another nice side effect is that the "ephemeral rerun::Collection hazard" goes away as we'd no longer store pointers to user data, making the API a lot safer to use. (rerun::Collection becomes a pure pass-through type as it should be)

• rerun::Collection borrows data too eagerly, making it very easy to cause segfaults & read of invalid data #7081

---

This ticket is a meetup discussion outcome of @jleibs and @Wumpf with some additional input by @emilk

Advantages

• We can support "component generics" (i.e. radii that is either f32 or f64(see https://github.com/rerun-io/rerun/blob/main/design/component_datatypes.md)
• C++ and Python can easily call into Rust to process the contents of a component or archetype (arrow is FFI-friendly)

Related

• Send dataframe API #7204
• Allow archetypes to clear and/or update existing components #3381
• Minimal component-transform interface #1882
• https://github.com/rerun-io/rerun/blob/main/design/component_datatypes.md