Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve video doc page #8007

Merged
merged 2 commits into from
Nov 6, 2024
Merged

Improve video doc page #8007

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1be1b98
Update Video doc page
Wumpf Nov 5, 2024
6b70b2d
formulation fix
Wumpf Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 35 additions & 26 deletions docs/content/reference/video.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,62 +27,71 @@ snippet: archetypes/video_auto_frames
Video support is new in Rerun, and has a few limitations:

* [#7354](https://github.com/rerun-io/rerun/issues/7354): Only the MP4 container format is supported
* [#7298](https://github.com/rerun-io/rerun/issues/7298): On native, only the AV1 codec is supported
* [#7755](https://github.com/rerun-io/rerun/issues/7755): No AV1 support on Linux ARM
* [#5181](https://github.com/rerun-io/rerun/issues/5181): There is no audio support
* [#7594](https://github.com/rerun-io/rerun/issues/7594): HDR video is not supported
* There is no video encoder in the Rerun SDK, so you need to create the video file yourself
* A limited sets of codecs are supported on web (see below)

## Streaming video
Rerun does not yet support streaming video support. For scenarios where you don't need live video, you can work around this limitation by logging many small `AssetVideo`s to the same Entity Path. See [#7484](https://github.com/rerun-io/rerun/issues/7484) for more.

## Codec support

### Overview

Codec support varies in the web & native viewer:

| | Browser | Native |
| ---------- | ------- | ------ |
| AV1 | ✅ | ✅ |
| H.264/avc | ✅ | ✅ |
| H.265/hevc | 🚧 | ❌ |
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe 🚧 doesn't quite cut it. It's not like we're working on it actively (we can't - hevc support is a legal issue for mozilla). But 🟧 is worse

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trying 🔲 instead

| VP8 | ✅ | ❌ |
| VP9 | ✅ | ❌ |

<!-- for web codecs see https://www.w3.org/TR/webcodecs-codec-registry/#video-codec-registry -->

Details see below.

When choosing a codec, we recommend [AV1](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#av1),
as it seems to have the best overall playback support while also having very high compression quality.
AV1 is also patent-free, and is the only codec we currently support in the native viewer (see [#7298](https://github.com/rerun-io/rerun/issues/7298)). H.264/avc is another popular choice, and native support for that is coming soon.

Since AV1 can have very long encoding times, it is often not yet suitable for streaming.
Wumpf marked this conversation as resolved.
Show resolved Hide resolved
In those cases where encoding time matters, we recommend H.264/avc.

### Native viewer
In the native viewer, AV1 is the only supported codec. H.264 is coming soon ([#7298](https://github.com/rerun-io/rerun/issues/7298)).

* AV1 is supported out of the box using a software decoder paired with gpu based image conversion
* H.264/avc is supported via a system installed `FFmpeg` binary
* When no binary is found, Rerun offers a download link to a build of `FFmpeg` for your platform in the selection panel upon selecting the failed video

### Web viewer
Video playback in the Rerun Web Viewer is done using the browser's own video decoder, so the supported codecs depend on your browser.
Video playback in the Rerun Web Viewer is done using the browser's own video decoder, so the exact supported codecs depend on your browser.

Overall, we recommend using Chrome or another Chromium-based browser, as it seems to have the best video support as of writing.

For decoding video in the Web Viewer, we use the [WebCodecs API](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API).
This API enables us to take advantage of the browser's hardware accelerated video decoding capabilities.
It is implemented by all modern browsers, but with varying levels of support for different codecs, and varying levels of quality.

With that in mind, here are the browsers which we have tested and verified to generally work:

| | Linux | macOS | Windows |
| ---------- | ----- | ----- | ------- |
| Firefox | ✅[^1] | ✅ | ✅ |
| Chrome[^2] | ✅ | ✅ | ✅[^3] |
| Safari | | ✅ | |

[^1]: Firefox on Linux has been observed to [stutter when playing back H.264 video](https://github.com/rerun-io/rerun/issues/7532).
[^2]: Any Chromium-based browser should work, but we don't test all of them.
[^3]: Chrome on Windows has been observed to stutter on playback. It can be mitigated by [using software decoding](https://rerun.io/docs/getting-started/troubleshooting#video-stuttering), but this may lead to high memory usage. See [#7595](https://github.com/rerun-io/rerun/issues/7595).

When it comes to codecs, we aim to support any codec which the browser supports, but
we currently cannot guarantee that all of them will work. For more information about
which codecs are supported by which browser, see [Video codecs on MDN](https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Video_codecs#codec_details).

At the moment, we test the following codecs:
We tested the following codecs in more detail:

| | Linux Firefox | Linux Chrome | macOS Firefox | macOS Chrome | macOS Safari | Windows Firefox | Windows Chrome |
| ---------- | ------------- | ------------ | ------------- | ------------ | ------------ | --------------- | -------------- |
| AV1 | ✅ | ✅ | ✅ | ✅ | 🚧[^4] | ✅ | ✅ |
| H.264/avc | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| H.265/hevc | ❌ | ❌ | ❌ | ✅ | 🚧[^6] | ❌ | 🚧[^7] |
| | Linux Firefox | Linux Chrome[^2] | macOS Firefox | macOS Chrome | macOS Safari | Windows Firefox | Windows Chrome[^3] |
| ---------- | ------------- | ---------------- | ------------- | ------------ | ------------ | --------------- | ------------------ |
| AV1 | ✅ | ✅ | ✅ | ✅ | 🚧[^4] | ✅ | ✅ |
| H.264/avc | ✅[^4] | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| H.265/hevc | ❌ | ❌ | ❌ | ✅ | 🚧[^6] | ❌ | 🚧[^7] |

[^1]: Firefox on Linux has been observed to [stutter when playing back H.264 video](https://github.com/rerun-io/rerun/issues/7532).
[^2]: Any Chromium-based browser should work, but we don't test all of them.
[^3]: Chrome on Windows has been observed to stutter on playback. It can be mitigated by [using software decoding](https://rerun.io/docs/getting-started/troubleshooting#video-stuttering), but this may lead to high memory usage. See [#7595](https://github.com/rerun-io/rerun/issues/7595).
[^4]: Safari/WebKit does not support AV1 decoding except on [Apple Silicon devices with hardware support](https://webkit.org/blog/14445/webkit-features-in-safari-17-0/).
[^5]: Firefox does not support H.265 decoding on any platform.
[^6]: Safari/WebKit has been observed suttering when playing `hvc1` but working fine with `hevc1`. Despite support being advertised Safari 16.5 has been observed not support H.265 decoding.
[^7]: Only supported if hardware encoding is available. Therefore always affected by Windows stuttering issues, see above.
[^5]: Safari/WebKit has been observed suttering when playing `hvc1` but working fine with `hevc1`. Despite support being advertised Safari 16.5 has been observed not support H.265 decoding.
[^6]: Only supported if hardware encoding is available. Therefore always affected by Windows stuttering issues, see above.

Beyond this, for best compatibility we recommend:
* prefer YUV over RGB & monochrome formats
Expand Down
Loading