Skip to content

Commit

Permalink
Dataframe v2: support for clear semantics (#7652)
Browse files Browse the repository at this point in the history
Support clear semantics in the dataframe API.
Tombstones are never visible to end-users, only their effect.

Like every other Dataframe v2 feature PR, and following recommendations
from @jleibs, this prioritizes convenience of implementation over
everything else, for now.
All clear chunks are fetched, post-processed, and re-injected into the
view contents during init(), and then the streaming join runs as usual
after that.

Static clear semantics can get pretty unhinged, but that's A) not
specific to the dataframe API and B) so extremely niche that our time is
better spent on real-world problems right now:
- #7650 
- #7631 

---

- Fixes #7495
- Fixes #7414
- Fixes #7468
- Fixes #7493
- DNM: requires #7649
  • Loading branch information
teh-cmc authored Oct 10, 2024
1 parent ab69022 commit c6d842b
Show file tree
Hide file tree
Showing 2 changed files with 551 additions and 165 deletions.
16 changes: 13 additions & 3 deletions crates/store/re_chunk_store/src/dataframe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ use arrow2::{
array::ListArray as ArrowListArray,
datatypes::{DataType as ArrowDatatype, Field as ArrowField},
};
use nohash_hasher::IntSet;

use re_chunk::TimelineName;
use re_log_types::ResolvedTimeRange;
use re_log_types::{EntityPath, TimeInt, Timeline};
use re_log_types::{EntityPath, ResolvedTimeRange, TimeInt, Timeline};
use re_types_core::{ArchetypeName, ComponentName};

use crate::ChunkStore;
Expand Down Expand Up @@ -765,12 +765,22 @@ impl ChunkStore {
})
});

use re_types_core::Archetype as _;
let clear_related_components: IntSet<ComponentName> =
re_types_core::archetypes::Clear::all_components()
.iter()
.copied()
.collect();

let components = static_components
.chain(temporal_components)
.filter(|col| match col {
ColumnDescriptor::Time(_) => true,
ColumnDescriptor::Component(descr) => {
!descr.component_name.is_indicator_component()
let is_indicator = descr.component_name.is_indicator_component();
// Tombstones are not exposed to end users -- only their _effect_.
let is_tombstone = clear_related_components.contains(&descr.component_name);
!is_indicator && !is_tombstone
}
})
.collect::<BTreeSet<_>>();
Expand Down
Loading

0 comments on commit c6d842b

Please sign in to comment.