Skip to content

Commit

Permalink
Passthrough yuv format for ffmpeg-h264 decoder when possible (#8002)
Browse files Browse the repository at this point in the history
### What

* part of #7607

Recognizing ahead of time what format a piece of encoded data is
unfortunately not that easy. To accomplish that I fell into the rabbit
hole of parsing the Sequence Parameter Set (SPS). Which will hopefully
be useful for other things in the future!

Other than that, it turned out that determining the color space & color
range are again different cans of worms, so I'm being now a lot more
explicit with ffmpeg on what we expect.
Which.. is yet another rabbit hole! Pretty much no matter what I tried I
couldn't replicate exact colors some sample videos compared to VLC
(unlike we did with AV1 where we have a lot more solid grasp on all
these things and get satisfying parity!). But also here different video
players seem to do slightly different things, so I decided to not follow
this thread further for the moment as this whole affair already cost me
a lot more time that I wanted to invest originally.

~Partially disabled unfortunately since we need this fix:~
* nathanbabcock/ffmpeg-sidecar#61

<img width="390" alt="image"
src="https://github.com/user-attachments/assets/c340544a-0c32-406e-ba20-2acc630c0238">

(note that this uses 422, the video source material is 420 though)


Fixes slow 4k video handling:

Before:
<img width="512" alt="image"
src="https://github.com/user-attachments/assets/82a0f0a1-03e7-4eac-8a24-a17ba185a8b5">


After:
<img width="548" alt="image"
src="https://github.com/user-attachments/assets/fb8315f4-5a85-4c2e-83bb-7e7e0c8ded33">


### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/8002?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/8002?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!
* [x] If have noted any breaking changes to the log API in
`CHANGELOG.md` and the migration guide

- [PR Build Summary](https://build.rerun.io/pr/8002)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.
  • Loading branch information
Wumpf authored Nov 6, 2024
1 parent 6b3d7fd commit 4ef1be8
Show file tree
Hide file tree
Showing 9 changed files with 627 additions and 146 deletions.
4 changes: 2 additions & 2 deletions Cargo.lock
Original file line number Diff line number Diff line change
Expand Up @@ -2379,9 +2379,9 @@ dependencies = [

[[package]]
name = "ffmpeg-sidecar"
version = "1.1.2"
version = "2.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4bd1e249e0ceeb0f5c9f84a3c6941c3bde3ebc2815f4b94531a7e806af61c4c0"
checksum = "bb737515ca6d4a2e4e7d941d70b9a580e27f4c47e986c8a39aab2f9c80ca860f"
dependencies = [
"anyhow",
]
Expand Down
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ econtext = "0.2" # Prints error contexts on crashes
ehttp = "0.5.0"
enumset = "1.0.12"
env_logger = { version = "0.10", default-features = false }
ffmpeg-sidecar = "1.1.2"
ffmpeg-sidecar = { version = "2.0.2", default-features = false }
fixed = { version = "1.28", default-features = false }
flatbuffers = "23.0"
futures-channel = "0.3"
Expand Down

Large diffs are not rendered by default.

5 changes: 5 additions & 0 deletions crates/store/re_video/src/decode/ffmpeg_h264/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
mod ffmpeg;
mod nalu;
mod sps;

pub use ffmpeg::{Error, FfmpegCliH264Decoder};
92 changes: 92 additions & 0 deletions crates/store/re_video/src/decode/ffmpeg_h264/nalu.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
/// In Annex-B before every NAL unit is a nal start code.
///
/// Can also be 2 bytes of 0x00 and 1 byte of 0x01.
///
/// This is used in Annex-B byte stream formats such as h264 files.
/// Packet transform systems (RTP) may omit these.
pub const NAL_START_CODE: &[u8] = &[0x00, 0x00, 0x00, 0x01];

/// Possible values for `nal_unit_type` field in `nal_unit`.
///
/// Encodes to 5 bits.
/// Via:
/// * <https://docs.rs/less-avc/0.1.5/src/less_avc/nal_unit.rs.html#232/>
/// * <https://github.com/FFmpeg/FFmpeg/blob/87068b9600daa522e3f45b5501ecd487a3c0be57/libavcodec/h264.h#L33>
#[derive(PartialEq, Eq)]
#[non_exhaustive]
#[repr(u8)]
#[derive(Copy, Clone, Debug)]
pub enum NalUnitType {
/// Unspecified
Unspecified = 0,

/// Coded slice of a non-IDR picture
CodedSliceOfANonIDRPicture = 1,

/// Coded slice data partition A
CodedSliceDataPartitionA = 2,

/// Coded slice data partition B
CodedSliceDataPartitionB = 3,

/// Coded slice data partition C
CodedSliceDataPartitionC = 4,

/// Coded slice of an IDR picture
CodedSliceOfAnIDRPicture = 5,

/// Supplemental enhancement information (SEI)
SupplementalEnhancementInformation = 6,

/// Sequence parameter set
SequenceParameterSet = 7,

/// Picture parameter set
PictureParameterSet = 8,

/// Signals the end of a NAL unit.
AccessUnitDelimiter = 9,

EndSequence = 10,
EndStream = 11,
FillerData = 12,
SequenceParameterSetExt = 13,

/// Header type not listed here.
Other,
}

/// Header of the "Network Abstraction Layer" unit that is used by H.264/AVC & H.265/HEVC.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct NalHeader(pub u8);

impl NalHeader {
pub const fn new(unit_type: NalUnitType, ref_idc: u8) -> Self {
Self((unit_type as u8) | (ref_idc << 5))
}

pub fn unit_type(self) -> NalUnitType {
match self.0 & 0b11111 {
0 => NalUnitType::Unspecified,
1 => NalUnitType::CodedSliceOfANonIDRPicture,
2 => NalUnitType::CodedSliceDataPartitionA,
3 => NalUnitType::CodedSliceDataPartitionB,
4 => NalUnitType::CodedSliceDataPartitionC,
5 => NalUnitType::CodedSliceOfAnIDRPicture,
6 => NalUnitType::SupplementalEnhancementInformation,
7 => NalUnitType::SequenceParameterSet,
8 => NalUnitType::PictureParameterSet,
9 => NalUnitType::AccessUnitDelimiter,
10 => NalUnitType::EndSequence,
11 => NalUnitType::EndStream,
12 => NalUnitType::FillerData,
13 => NalUnitType::SequenceParameterSetExt,
_ => NalUnitType::Other,
}
}

/// Ref idc is a value from 0-3 that tells us how "important" the frame/sample is.
pub fn ref_idc(self) -> u8 {
(self.0 >> 5) & 0b11
}
}
Loading

0 comments on commit 4ef1be8

Please sign in to comment.