Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve docs about video support #7687

Merged
merged 10 commits into from
Oct 11, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ namespace rerun.archetypes;

/// A video binary.
///
/// NOTE: Videos can only be viewed in the Rerun web viewer.
/// Only MP4 containers with a limited number of codecs are currently supported, and not in all browsers.
/// Only MP4 containers with AV1 is generally supported,
/// though the web viewer sometimes supports some more video codex, depending on browser.
emilk marked this conversation as resolved.
Show resolved Hide resolved
///
/// See <https://rerun.io/docs/concepts/video?speculative-link> for details of what is and isn't supported.
/// See <https://rerun.io/docs/reference/video?speculative-link> for details of what is and isn't supported.
///
/// In order to display a video, you also need to log a [archetypes.VideoFrameReference] for each frame.
///
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ namespace rerun.archetypes;
/// Used to display individual video frames from a [archetypes.AssetVideo].
/// To show an entire video, a video frame reference for each frame of the video should be logged.
///
/// See <https://rerun.io/docs/concepts/video?speculative-link> for details of what is and isn't supported.
/// See <https://rerun.io/docs/reference/video?speculative-link> for details of what is and isn't supported.
///
/// \example archetypes/video_auto_frames title="Video with automatically determined frames" image="https://static.rerun.io/video_manual_frames/320a44e1e06b8b3a3161ecbbeae3e04d1ccb9589/1200w.png"
/// \example archetypes/video_manual_frames title="Demonstrates manual use of video frame references" image="https://static.rerun.io/video_manual_frames/9f41c00f84a98cc3f26875fba7c1d2fa2bad7151/1200w.png"
Expand Down
6 changes: 3 additions & 3 deletions crates/store/re_types/src/archetypes/asset_video.rs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions docs/content/reference/types/archetypes/asset_video.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

28 changes: 17 additions & 11 deletions docs/content/reference/video.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,26 +23,33 @@ This gives the best compression ratio, reducing file sizes and bandwidth require

snippet: archetypes/video_auto_frames

## `AssetVideo` limitations
Video support is new in Rerun, and has several limitations:
## Video playback limitations
Video support is new in Rerun, and has a few limitations:

* [#7298](https://github.com/rerun-io/rerun/issues/7298): Video playback only works in the web viewer
* [#7354](https://github.com/rerun-io/rerun/issues/7354): Only the MP4 container format is supported
* [#7298](https://github.com/rerun-io/rerun/issues/7298): Only the AV1 codec is supported in the native viewer
* [#5181](https://github.com/rerun-io/rerun/issues/5181): There is no audio support
* [#7594](https://github.com/rerun-io/rerun/issues/7594): HDR video is not supported
* There is no video encoder in the Rerun SDK, so you need to create the video file yourself
* Only a limited sets of codecs are supported (see below)
* A limited sets of codecs are supported on web (see below)

## Web viewer support
As of writing, playback of `AssetVideo` is only supported on the web viewer.
Native video playback is coming, and can be tracked [in this GitHub issue](https://github.com/rerun-io/rerun/issues/7298).
## Streaming video
Rerun does not yet support streaming video support. For scenarios where you don't need live video, you can work around this limitation by logging many small `AssetVideo`s to the same Entity Path. See [#7484](https://github.com/rerun-io/rerun/issues/7484) for more.

Video playback is done using the browser's own video decoder, so the supported codecs depend on your browser.
## Codec support

### Native viewer
In the native viewer, AV1 is the only supported codec. H.264 is coming soon ([#7298](https://github.com/rerun-io/rerun/issues/7298)).


### Web viewer
Video playback in the Rerun Web Viewer is done using the browser's own video decoder, so the supported codecs depend on your browser.

Overall, we recommend using Chrome or another Chromium-based browser, as it seems to have the best video support as of writing.

When choosing a codec, we recommend [AV1](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#av1),
as it seems to have the best overall playback support while also having very high compression quality.
Since AV1 is patent-free, it is also likely the first codec we will support in the native viewer.
AV1 is also patent-free, and is the only codec we currently support in the native viewer (see [#7298](https://github.com/rerun-io/rerun/issues/7298)). H.264/avc is another popular choice, and native support for that is coming soon.
emilk marked this conversation as resolved.
Show resolved Hide resolved

For decoding video in the Web Viewer, we use the [WebCodecs API](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API).
This API enables us to take advantage of the browser's hardware accelerated video decoding capabilities.
Expand All @@ -58,7 +65,7 @@ With that in mind, here are the browsers which we have tested and verified to ge

[^1]: Firefox on Linux has been observed to [stutter when playing back H.264 video](https://github.com/rerun-io/rerun/issues/7532).
[^2]: Any Chromium-based browser should work, but we don't test all of them.
[^3]: Chrome on Windows has been observed to stutter on playback. It can be mitigated by [using software decoding](https://rerun.io/docs/getting-started/troubleshooting#video-stuttering), but this may lead to high memory usage. See also https://github.com/rerun-io/rerun/issues/7595.
[^3]: Chrome on Windows has been observed to stutter on playback. It can be mitigated by [using software decoding](https://rerun.io/docs/getting-started/troubleshooting#video-stuttering), but this may lead to high memory usage. See [#7595](https://github.com/rerun-io/rerun/issues/7595).

When it comes to codecs, we aim to support any codec which the browser supports, but
we currently cannot guarantee that all of them will work. For more information about
Expand All @@ -77,7 +84,6 @@ At the moment, we test the following codecs:
[^6]: Safari/WebKit has been observed suttering when playing `hvc1` but working fine with `hevc1`. Despite support being advertised Safari 16.5 has been observed not support H.265 decoding.
[^7]: Only supported if hardware encoding is available. Therefore always affected by Windows stuttering issues, see [^3].

(TODO(#7594))[https://github.com/rerun-io/rerun/issues/7594]: HDR video is generally not supported at this point.

## Links
* [Web video codec guide, by Mozilla](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs)
6 changes: 3 additions & 3 deletions rerun_cpp/src/rerun/archetypes/asset_video.hpp

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion rerun_cpp/src/rerun/archetypes/video_frame_reference.hpp

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions rerun_py/rerun_sdk/rerun/archetypes/asset_video.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading