New data APIs 15: one query crate to rule them all #6036
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teh-cmc
added
🔍 re_query
This was referenced Apr 18, 2024
teh-cmc
force-pushed
the
cmc/data_apis_14_uncached_views_for_real
branch
from
95b2530 to
1acbf05
Compare
teh-cmc
force-pushed
the
cmc/data_apis_15_one_crate_to_rule_them_all
branch
from
df4f79c to
89b2102
Compare
teh-cmc
force-pushed
the
cmc/data_apis_14_uncached_views_for_real
branch
4 times, most recently
from
286fac1 to
4c15862
Compare
teh-cmc
force-pushed
the
cmc/data_apis_15_one_crate_to_rule_them_all
branch
from
59a03aa to
8ba1ca4
Compare
emilk
approved these changes
Apr 25, 2024
teh-cmc
force-pushed
the
cmc/data_apis_14_uncached_views_for_real
branch
from
4c15862 to
1354bd2
Compare
teh-cmc
force-pushed
the
cmc/data_apis_15_one_crate_to_rule_them_all
branch
from
5939bf4 to
d90df58
Compare
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Static-aware, key-less, component-based, cached range APIs.
```rust
let caches = re_query_cache2::Caches::new(&store);
// First, get the raw results for this query.
//
// They might or might not already be cached. We won't know for sure until we try to access
// each individual component's data below.
let results: CachedRangeResults = caches.range(
&store,
&query,
&entity_path.into(),
MyPoints::all_components().iter().copied(), // no generics!
);
// Then, grab the results for each individual components.
// * `get_required` returns an error if the component batch is missing
// * `get_or_empty` returns an empty set of results if the component if missing
// * `get` returns an option
//
// At this point we still don't know whether they are cached or not. That's the next step.
let all_points: &CachedRangeComponentResults = results.get_required(MyPoint::name())?;
let all_colors: &CachedRangeComponentResults = results.get_or_empty(MyColor::name());
let all_labels: &CachedRangeComponentResults = results.get_or_empty(MyLabel::name());
// Then comes the time to resolve/convert and deserialize the data.
// These steps have to be done together for efficiency reasons.
//
// That's when caching comes into play.
// If the data has already been accessed in the past, then this will just grab the
// pre-deserialized, pre-resolved/pre-converted result from the cache.
// Otherwise, this will trigger a deserialization and cache the result for next time.
let all_points = all_points.to_dense::<MyPoint>(&resolver);
let all_colors = all_colors.to_dense::<MyColor>(&resolver);
let all_labels = all_labels.to_dense::<MyLabel>(&resolver);
// The cache might not have been able to resolve and deserialize the entire dataset across all
// available timestamps.
//
// We can use the following APIs to check the status of the front and back sides of the data range.
//
// E.g. it is possible that the front-side of the range is still waiting for pending data while
// the back-side has been fully loaded.
assert!(matches!(
all_points.status(),
(PromiseResult::Ready(()), PromiseResult::Ready(()))
));
// Zip the results together using a stateful time-based join.
let all_frames = range_zip_1x2(
all_points.range_indexed(),
all_colors.range_indexed(),
all_labels.range_indexed(),
);
// Then comes the time to resolve/convert and deserialize the data, _for each timestamp_.
// These steps have to be done together for efficiency reasons.
//
// Both the resolution and deserialization steps might fail, which is why this returns a `Result<Result<T>>`.
// Use `PromiseResult::flatten` to simplify it down to a single result.
eprintln!("results:");
for ((data_time, row_id), points, colors, labels) in all_frames {
let colors = colors.unwrap_or(&[]);
let color_default_fn = || {
static DEFAULT: MyColor = MyColor(0xFF00FFFF);
&DEFAULT
};
let labels = labels.unwrap_or(&[]).iter().cloned().map(Some);
let label_default_fn = || None;
// With the data now fully resolved/converted and deserialized, the joining logic can be
// applied.
//
// In most cases this will be either a clamped zip, or no joining at all.
let results = clamped_zip_1x2(points, colors, color_default_fn, labels, label_default_fn)
.collect_vec();
eprintln!("{data_time:?} @ {row_id}:\n {results:?}");
}
```
---
Part of a PR series to completely revamp the data APIs in preparation
for the removal of instance keys and the introduction of promises:
- #5573
- #5574
- #5581
- #5605
- #5606
- #5633
- #5673
- #5679
- #5687
- #5755
- #5990
- #5992
- #5993
- #5994
- #6035
- #6036
- #6037
Builds on top of the static data PR series:
- #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Title. The new cache being natively component-based makes things much smoothier than before. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Text logs, line plots and scatter plots. A bit faster than `main`, with a bit less memory overhead. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Migrate all spatial views that were using the old cache APIs to the new ones. Instance keys are not queried at all anymore. All views are now range-aware by default. Also took the opportunity to somewhat streamline everything. The 10min air-traffic example with full visible range is about 2-2.5x faster than before. I'm sure I broke a few things here and there, I'll run a full check suite once everything's said and done. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
`re_query_cache` is gone, `re_query_cache2` takes its place -- simple as that. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
force-pushed
the
cmc/data_apis_14_uncached_views_for_real
branch
from
3755de5 to
3e2a18d
Compare
Base automatically changed from
cmc/data_apis_14_uncached_views_for_real
to
main
April 26, 2024 10:44
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Migrate every little thing that didn't use to go through the cached APIs. `Image` and `Mesh3D` are temporarily cached even though they shouldn't be, that's taken care of in a follow-up PR. Once again, I probably broke a million edge cases -- I want to get as fast as possible to removing instance keys before doing an in-depth quality pass. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
force-pushed
the
cmc/data_apis_15_one_crate_to_rule_them_all
branch
from
d90df58 to
bbabfbf
Compare
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Make it possible to not cache some components, all while pretending really hard that they've been cached. - Related: #5974 --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is now only one way to query data:
re_query(well you can still query the datastore directly if you're a monster, but that's for another PR).All queries go through both the query cache and the deserialization cache.
There will be a follow-up PR to disable the deserialization cache for specific components.
Most of this is just (re)moving stuff around except for the last two commits which take care of porting the cached test suites since they cannot depend on uncached APIs to do comparisons anymore.
Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises:
ClampedZipiterator machinery #5573Component, once and for all #5605RangeZipiterator machinery #5679Builds on top of the static data PR series:
TimeInt#5534Checklist
mainbuild: app.rerun.ionightlybuild: app.rerun.io