Skip to content

Commit

Permalink
Improve docs about video support (#7687)
Browse files Browse the repository at this point in the history
We do support AV1 on native, and now the docs say so too.

* Closes #7662
 
### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7687?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7687?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!
* [x] If have noted any breaking changes to the log API in
`CHANGELOG.md` and the migration guide
* [x] I've read the docs preview at
https://landing-e6yens1hp-rerun.vercel.app/docs/reference/video

- [PR Build Summary](https://build.rerun.io/pr/7687)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.

---------

Co-authored-by: Andreas Reich <[email protected]>
  • Loading branch information
emilk and Wumpf authored Oct 11, 2024
1 parent ad631cd commit 9f34cb7
Show file tree
Hide file tree
Showing 11 changed files with 39 additions and 34 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@ namespace rerun.archetypes;

/// A video binary.
///
/// NOTE: Videos can only be viewed in the Rerun web viewer.
/// Only MP4 containers with a limited number of codecs are currently supported, and not in all browsers.
/// Only MP4 containers with AV1 are generally supported,
/// though the web viewer supports more video codecs, depending on browser.
///
/// See <https://rerun.io/docs/concepts/video?speculative-link> for details of what is and isn't supported.
/// See <https://rerun.io/docs/reference/video?speculative-link> for details of what is and isn't supported.
///
/// In order to display a video, you also need to log a [archetypes.VideoFrameReference] for each frame.
///
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ namespace rerun.archetypes;
/// Used to display individual video frames from a [archetypes.AssetVideo].
/// To show an entire video, a video frame reference for each frame of the video should be logged.
///
/// See <https://rerun.io/docs/concepts/video?speculative-link> for details of what is and isn't supported.
/// See <https://rerun.io/docs/reference/video?speculative-link> for details of what is and isn't supported.
///
/// \example archetypes/video_auto_frames title="Video with automatically determined frames" image="https://static.rerun.io/video_manual_frames/320a44e1e06b8b3a3161ecbbeae3e04d1ccb9589/1200w.png"
/// \example archetypes/video_manual_frames title="Demonstrates manual use of video frame references" image="https://static.rerun.io/video_manual_frames/9f41c00f84a98cc3f26875fba7c1d2fa2bad7151/1200w.png"
Expand Down
6 changes: 3 additions & 3 deletions crates/store/re_types/src/archetypes/asset_video.rs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions docs/content/reference/types/archetypes/asset_video.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

33 changes: 19 additions & 14 deletions docs/content/reference/video.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,26 +23,32 @@ This gives the best compression ratio, reducing file sizes and bandwidth require

snippet: archetypes/video_auto_frames

## `AssetVideo` limitations
Video support is new in Rerun, and has several limitations:
## Video playback limitations
Video support is new in Rerun, and has a few limitations:

* [#7298](https://github.com/rerun-io/rerun/issues/7298): Video playback only works in the web viewer
* [#7354](https://github.com/rerun-io/rerun/issues/7354): Only the MP4 container format is supported
* [#7298](https://github.com/rerun-io/rerun/issues/7298): Only the AV1 codec is supported in the native viewer
* [#5181](https://github.com/rerun-io/rerun/issues/5181): There is no audio support
* [#7594](https://github.com/rerun-io/rerun/issues/7594): HDR video is not supported
* There is no video encoder in the Rerun SDK, so you need to create the video file yourself
* Only a limited sets of codecs are supported (see below)
* A limited sets of codecs are supported on web (see below)

## Web viewer support
As of writing, playback of `AssetVideo` is only supported on the web viewer.
Native video playback is coming, and can be tracked [in this GitHub issue](https://github.com/rerun-io/rerun/issues/7298).

Video playback is done using the browser's own video decoder, so the supported codecs depend on your browser.

Overall, we recommend using Chrome or another Chromium-based browser, as it seems to have the best video support as of writing.
## Streaming video
Rerun does not yet support streaming video support. For scenarios where you don't need live video, you can work around this limitation by logging many small `AssetVideo`s to the same Entity Path. See [#7484](https://github.com/rerun-io/rerun/issues/7484) for more.

## Codec support
When choosing a codec, we recommend [AV1](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#av1),
as it seems to have the best overall playback support while also having very high compression quality.
Since AV1 is patent-free, it is also likely the first codec we will support in the native viewer.
AV1 is also patent-free, and is the only codec we currently support in the native viewer (see [#7298](https://github.com/rerun-io/rerun/issues/7298)). H.264/avc is another popular choice, and native support for that is coming soon.

### Native viewer
In the native viewer, AV1 is the only supported codec. H.264 is coming soon ([#7298](https://github.com/rerun-io/rerun/issues/7298)).


### Web viewer
Video playback in the Rerun Web Viewer is done using the browser's own video decoder, so the supported codecs depend on your browser.

Overall, we recommend using Chrome or another Chromium-based browser, as it seems to have the best video support as of writing.

For decoding video in the Web Viewer, we use the [WebCodecs API](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API).
This API enables us to take advantage of the browser's hardware accelerated video decoding capabilities.
Expand All @@ -58,7 +64,7 @@ With that in mind, here are the browsers which we have tested and verified to ge

[^1]: Firefox on Linux has been observed to [stutter when playing back H.264 video](https://github.com/rerun-io/rerun/issues/7532).
[^2]: Any Chromium-based browser should work, but we don't test all of them.
[^3]: Chrome on Windows has been observed to stutter on playback. It can be mitigated by [using software decoding](https://rerun.io/docs/getting-started/troubleshooting#video-stuttering), but this may lead to high memory usage. See also https://github.com/rerun-io/rerun/issues/7595.
[^3]: Chrome on Windows has been observed to stutter on playback. It can be mitigated by [using software decoding](https://rerun.io/docs/getting-started/troubleshooting#video-stuttering), but this may lead to high memory usage. See [#7595](https://github.com/rerun-io/rerun/issues/7595).

When it comes to codecs, we aim to support any codec which the browser supports, but
we currently cannot guarantee that all of them will work. For more information about
Expand All @@ -77,7 +83,6 @@ At the moment, we test the following codecs:
[^6]: Safari/WebKit has been observed suttering when playing `hvc1` but working fine with `hevc1`. Despite support being advertised Safari 16.5 has been observed not support H.265 decoding.
[^7]: Only supported if hardware encoding is available. Therefore always affected by Windows stuttering issues, see [^3].

(TODO(#7594))[https://github.com/rerun-io/rerun/issues/7594]: HDR video is generally not supported at this point.

## Links
* [Web video codec guide, by Mozilla](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs)
6 changes: 3 additions & 3 deletions rerun_cpp/src/rerun/archetypes/asset_video.hpp

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion rerun_cpp/src/rerun/archetypes/video_frame_reference.hpp

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions rerun_py/rerun_sdk/rerun/archetypes/asset_video.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 9f34cb7

Please sign in to comment.