Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

craft blueprint for Catalog view from raw chunks #8449

Merged
merged 6 commits into from
Dec 16, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 130 additions & 7 deletions crates/store/re_grpc_client/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,12 @@ mod address;
pub use address::{InvalidRedapAddress, RedapAddress};
use re_chunk::external::arrow2;
use re_log_types::external::re_types_core::ComponentDescriptor;
use re_types::blueprint::archetypes::{ContainerBlueprint, ViewportBlueprint};
use re_types::blueprint::archetypes::{ViewBlueprint, ViewContents};
use re_types::blueprint::components::{ContainerKind, RootContainer};
use re_types::components::RecordingUri;
use re_types::Component;
use re_types::external::uuid;
use re_types::{Archetype, Component};
use url::Url;

// ----------------------------------------------------------------------------
Expand All @@ -15,10 +19,13 @@ use std::error::Error;

use arrow2::array::Utf8Array as Arrow2Utf8Array;
use arrow2::datatypes::Field as Arrow2Field;
use re_chunk::{Arrow2Array, Chunk, ChunkId, TransportChunk};
use re_chunk::{
Arrow2Array, Chunk, ChunkBuilder, ChunkId, EntityPath, RowId, Timeline, TransportChunk,
};
use re_log_encoding::codec::{wire::decode, CodecError};
use re_log_types::{
ApplicationId, LogMsg, SetStoreInfo, StoreId, StoreInfo, StoreKind, StoreSource, Time,
ApplicationId, BlueprintActivationCommand, EntityPathFilter, LogMsg, SetStoreInfo, StoreId,
StoreInfo, StoreKind, StoreSource, Time,
};
use re_protos::common::v0::RecordingId;
use re_protos::remote_store::v0::{
Expand Down Expand Up @@ -77,6 +84,10 @@ enum StreamError {

// ----------------------------------------------------------------------------

const CATALOG_BP_STORE_ID: &str = "catalog_blueprint";
const CATALOG_REC_STORE_ID: &str = "catalog";
const CATALOG_APPLICATION_ID: &str = "redap_catalog";

/// Stream an rrd file or metadsasta catalog over gRPC from a Rerun Data Platform server.
///
/// `on_msg` can be used to wake up the UI thread on Wasm.
Expand Down Expand Up @@ -276,11 +287,16 @@ async fn stream_catalog_async(

drop(client);

// We need a whole StoreInfo here.
let store_id = StoreId::from_string(StoreKind::Recording, "catalog".to_owned());
if activate_catalog_blueprint(&tx).is_err() {
re_log::debug!("Failed to activate catalog blueprint");
return Ok(());
}

// Craft the StoreInfo for the actual catalog data
let store_id = StoreId::from_string(StoreKind::Recording, CATALOG_REC_STORE_ID.to_owned());

let store_info = StoreInfo {
application_id: ApplicationId::from("redap_catalog"),
application_id: ApplicationId::from(CATALOG_APPLICATION_ID),
store_id: store_id.clone(),
cloned_from: None,
is_official_example: false,
Expand Down Expand Up @@ -309,7 +325,6 @@ async fn stream_catalog_async(
TransportChunk::CHUNK_METADATA_KEY_ID.to_owned(),
ChunkId::new().to_string(),
);

let mut chunk = Chunk::from_transport(&tc)?;

// enrich catalog data with RecordingUri that's based on the ReDap endpoint (that we know)
Expand Down Expand Up @@ -376,3 +391,111 @@ async fn stream_catalog_async(

Ok(())
}

// Craft a blueprint from relevant chunks and activate it
// TODO(zehiko) - manual crafting of the blueprint as we have below will go away and be replaced
// by either a blueprint crafted using rust Blueprint API or a blueprint fetched from ReDap (#8470)
fn activate_catalog_blueprint(
tx: &re_smart_channel::Sender<LogMsg>,
) -> Result<(), Box<dyn std::error::Error>> {
let blueprint_store_id =
StoreId::from_string(StoreKind::Blueprint, CATALOG_BP_STORE_ID.to_owned());
let blueprint_store_info = StoreInfo {
application_id: ApplicationId::from(CATALOG_APPLICATION_ID),
store_id: blueprint_store_id.clone(),
cloned_from: None,
is_official_example: false,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Side note: IIRC this is related to how we deal with analytics. We probably should do something special for these meta-recordings at some point as well.

started: Time::now(),
store_source: StoreSource::Unknown,
store_version: None,
};

if tx
.send(LogMsg::SetStoreInfo(SetStoreInfo {
row_id: *re_chunk::RowId::new(),
info: blueprint_store_info,
}))
.is_err()
{
re_log::debug!("Receiver disconnected");
return Ok(());
}

let timepoint = [(Timeline::new_sequence("blueprint"), 1)];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re_viewer_context::blueprint_timeline()

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not sure it's worth bringing in re_viewer_context to grpc client just for this one.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, right, I see suggestions below are also from this crate...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and adding it would create a dependency cycle :/

Copy link
Member

@abey79 abey79 Dec 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was expecting that dependency cycle. I dont think we should do anything about it in this PR. My suggestion is to simply add a bunch of todos, and maybe open an issue to document this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a brief huddle with @abey79 and he explained a bit the dependencies and what I'm hitting here makes sense and we don't want grpc client to depend on re_viewer_context. When rust blueprint API is properly introduced, we might depend on that crate (something like re_blueprint), but arguably by then catalog view blueprint might simply move to Data Platform and live along other customer recording's blueprints.

For now we'll go with what we have here and I'll add some TODOs in the code.


let vb = ViewBlueprint::new("Dataframe")
.with_visible(true)
.with_space_origin("/");

// TODO(zehiko) we shouldn't really be creating all these ids and entity paths manually... (#8470)
let view_uuid = uuid::Uuid::new_v4();
let view_entity_path = format!("/view/{view_uuid}");
let view_chunk = ChunkBuilder::new(ChunkId::new(), view_entity_path.clone().into())
.with_archetype(RowId::new(), timepoint, &vb)
.build()?;

let epf = EntityPathFilter::parse_forgiving("/**", &Default::default());
let vc = ViewContents::new(epf.iter_expressions());
let view_contents_chunk = ChunkBuilder::new(
ChunkId::new(),
format!(
"{}/{}",
view_entity_path.clone(),
ViewContents::name().short_name()
)
.into(),
)
.with_archetype(RowId::new(), timepoint, &vc)
.build()?;

let rc = ContainerBlueprint::new(ContainerKind::Grid)
.with_contents(&[EntityPath::from(view_entity_path)])
.with_visible(true);

let container_uuid = uuid::Uuid::new_v4();
let container_chunk = ChunkBuilder::new(
ChunkId::new(),
format!("/container/{container_uuid}").into(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, these are derived from ContainerId::registry_path()

)
.with_archetype(RowId::new(), timepoint, &rc)
.build()?;

let vp = ViewportBlueprint::new().with_root_container(RootContainer(container_uuid.into()));
let viewport_chunk = ChunkBuilder::new(ChunkId::new(), "/viewport".into())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re_viewport_blueprint::VIEWPORT_PATH

.with_archetype(RowId::new(), timepoint, &vp)
.build()?;

for chunk in &[
view_chunk,
view_contents_chunk,
container_chunk,
viewport_chunk,
] {
if tx
.send(LogMsg::ArrowMsg(
blueprint_store_id.clone(),
chunk.to_arrow_msg()?,
))
.is_err()
{
re_log::debug!("Receiver disconnected");
return Ok(());
}
}

let blueprint_activation = BlueprintActivationCommand {
blueprint_id: blueprint_store_id.clone(),
make_active: true,
make_default: true,
};

if tx
.send(LogMsg::BlueprintActivationCommand(blueprint_activation))
.is_err()
{
re_log::debug!("Receiver disconnected");
return Ok(());
}

Ok(())
}
Loading