Introduce `VideoFrameReference` archetype and base video visualization on it #7396

Wumpf · 2024-09-10T15:11:37Z

What

Large chunk of Video frame references #7368

Introduce VideoFrameReference archetype
- comes with a bunch of subtypes
Make the video visualizer use this
- in the process I did some general iteration on how this visualizer ticks
  - note that I departed from the usual quite encrusted patterns of how to implement visualizers that naturally found its way into here as well. Imho the patterns don't hold up all that well in general (we had to grind over them many times in quick succession) and less so in particular for "special" archetypes like video or mesh (etc.)
video data loader emits VideoFrameReference instead of the adhoc `VideoTick

Still missing for completion of the video frame reference task:

examples to use this from SDKs (by directly logging video frames)
- this surely will show that we need more extensions
pretty/edit ui for video timestamps
- @Wumpf are you saying that you can then scrub videos from a blueprint slider?! Yes. That's exactly what will end up happening ;D
pretty/edit ui for entity paths
exposed utilities to SDKs for generation of video frame references

@ reviewers: Special attention please on the fbs definition, thank you!

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested the web demo (if applicable):
- Using examples from latest main build: rerun.io/viewer
- Using full set of examples from nightly build: rerun.io/viewer
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
If applicable, add a new check to the release checklist!
If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

To run all checks from main, comment on the PR with @rerun-bot full-check.

github-actions · 2024-09-10T15:12:05Z

Deployed docs

Commit	Link
`5cfd549`	https://landing-phsglo6gv-rerun.vercel.app/docs

emilk · 2024-09-10T15:16:17Z

crates/store/re_types/definitions/rerun/archetypes/asset_video.fbs

 /// Follow <https://github.com/rerun-io/rerun/issues/7298> for updates on the native support.
+///
+/// In order to display a video, you need to log a [archetypes.VideoFrameReference] for each frame.
+// TODO(andreas): More docs and examples on how to use this.


Let's point these TODOs to the correct issue

.vscode/settings.json

Wumpf · 2024-09-10T15:18:08Z

~~(drafted until I remove accidentially pushed stuff)~~
fixed

rerun_py/Cargo.toml

pixi.toml

jleibs · 2024-09-10T15:40:23Z

crates/store/re_types/definitions/rerun/datatypes/video_timestamp.fbs

+/// Timestamp inside a [archetypes.AssetVideo].
+struct VideoTimestamp (


I would still love to eventually have time_mode = FrameNumber.

So calling this a Timestamp will create a bit of a strange naming pattern.

coincidentally I brought this very thing up on Slack expressing a similar (but not very deep) concern. @emilk pointed out that frame numbers can be seen as a form timestamp unit in this context.
very open to other suggestions not sure on this one 🤷

https://rerunio.slack.com/archives/C041NHU952S/p1725981749912279?thread_ts=1725981199.569259&cid=C041NHU952S

Some ideas from ChatGPT

VideoPosition: Emphasizes that the struct represents a position in the video, which could be either in frames or seconds. VideoMarker: Suggests a point in the video, adaptable to frames or time. TimeOrFrame: Makes it clear that the struct can hold either a time value or a frame number. FrameOrTimeStamp: Explicitly names the two modes. VideoCoordinate: Represents a general position in time (seconds or frames) as coordinates often imply different dimensions. MediaTimeUnit: Indicates the struct holds a unit of time in different formats, either frames or seconds. PlaybackPosition: Focuses on the position in playback, where both time and frame counts are relevant. TimelineIndex: Suggests an index along a timeline that can be expressed in either frames or time.

jprochazk

Works great!

Naming is a non-issue imo, we can change it in a follow-up PR if we think VideoTimestamp really doesn't work

@Wumpf

…n on it (#7396) ### What * Large chunk of #7368 --- * Introduce `VideoFrameReference` archetype * comes with a bunch of subtypes * Make the video visualizer use this * in the process I did some general iteration on how this visualizer ticks * note that I departed from the usual quite encrusted patterns of how to implement visualizers that naturally found its way into here as well. Imho the patterns don't hold up all that well in general (we had to grind over them many times in quick succession) and less so in particular for "special" archetypes like video or mesh (etc.) * video data loader emits `VideoFrameReference` instead of the adhoc `VideoTick Still missing for completion of the video frame reference task: * examples to use this from SDKs (by directly logging video frames) * this surely will show that we need more extensions * pretty/edit ui for video timestamps * @Wumpf are you saying that you can then scrub videos from a blueprint slider?! Yes. That's exactly what will end up happening ;D * pretty/edit ui for entity paths * exposed utilities to SDKs for generation of video frame references @ reviewers: Special attention please on the fbs definition, thank you! <img width="1813" alt="image" src="https://github.com/user-attachments/assets/d221ff1e-7ec0-4067-8916-7a8dd2ed8002"> ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Using examples from latest `main` build: [rerun.io/viewer](https://rerun.io/viewer/pr/7396?manifest_url=https://app.rerun.io/version/main/examples_manifest.json) * Using full set of examples from `nightly` build: [rerun.io/viewer](https://rerun.io/viewer/pr/7396?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG * [x] If applicable, add a new check to the [release checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)! * [x] If have noted any breaking changes to the log API in `CHANGELOG.md` and the migration guide - [PR Build Summary](https://build.rerun.io/pr/7396) - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html) To run all checks from `main`, comment on the PR with `@rerun-bot full-check`.

Wumpf added 7 commits September 10, 2024 11:50

rename asset_video fbs file

e59c1c6

add video frame reference archetype

c9bdfbd

codegen

219d203

Extensions for video timestamp

4bcfc11

Replace VideoTick with new video timestamps

a5cd6c9

video visualizer now operates on video frame references

904303c

use acquire release semantics on video cache

b040f4c

Wumpf added exclude from changelog PRs with this won't show up in CHANGELOG.md 🎞️ video labels Sep 10, 2024

emilk reviewed Sep 10, 2024

View reviewed changes

.vscode/settings.json Outdated Show resolved Hide resolved

jprochazk reviewed Sep 10, 2024

View reviewed changes

.vscode/settings.json Outdated Show resolved Hide resolved

Wumpf marked this pull request as draft September 10, 2024 15:17

emilk reviewed Sep 10, 2024

View reviewed changes

rerun_py/Cargo.toml Outdated Show resolved Hide resolved

pixi.toml Outdated Show resolved Hide resolved

improved commentry

5df57f3

Wumpf force-pushed the andreas/video-frame-references branch from a7f0a2b to 5df57f3 Compare September 10, 2024 15:18

typo fixes

c0efad5

Wumpf marked this pull request as ready for review September 10, 2024 15:20

new video section in python docs

e7e1ab9

jleibs reviewed Sep 10, 2024

View reviewed changes

Wumpf requested a review from jprochazk September 10, 2024 18:52

jprochazk approved these changes Sep 11, 2024

View reviewed changes

Wumpf added 2 commits September 11, 2024 11:07

todo note improvements

e1f408e

add missing components to check_all_components_ui.py

5cfd549

Wumpf merged commit e77459b into main Sep 11, 2024
39 of 40 checks passed

Wumpf deleted the andreas/video-frame-references branch September 11, 2024 09:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce `VideoFrameReference` archetype and base video visualization on it #7396

Introduce `VideoFrameReference` archetype and base video visualization on it #7396

Wumpf commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

emilk Sep 10, 2024

Wumpf commented Sep 10, 2024 •

edited

Loading

jleibs Sep 10, 2024

Wumpf Sep 10, 2024

jprochazk Sep 11, 2024

jprochazk left a comment

		/// Timestamp inside a [archetypes.AssetVideo].
		struct VideoTimestamp (

Introduce VideoFrameReference archetype and base video visualization on it #7396

Introduce VideoFrameReference archetype and base video visualization on it #7396

Conversation

Wumpf commented Sep 10, 2024 • edited Loading

What

Checklist

github-actions bot commented Sep 10, 2024 • edited Loading

Deployed docs

emilk Sep 10, 2024

Choose a reason for hiding this comment

Wumpf commented Sep 10, 2024 • edited Loading

jleibs Sep 10, 2024

Choose a reason for hiding this comment

Wumpf Sep 10, 2024

Choose a reason for hiding this comment

jprochazk Sep 11, 2024

Choose a reason for hiding this comment

jprochazk left a comment

Choose a reason for hiding this comment

Introduce `VideoFrameReference` archetype and base video visualization on it #7396

Introduce `VideoFrameReference` archetype and base video visualization on it #7396

Wumpf commented Sep 10, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024 •

edited

Loading

Wumpf commented Sep 10, 2024 •

edited

Loading