Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix VP9 decoding on iOS #269

Merged
merged 1 commit into from
Apr 29, 2024
Merged

Fix VP9 decoding on iOS #269

merged 1 commit into from
Apr 29, 2024

Conversation

aler9
Copy link
Member

@aler9 aler9 commented Apr 27, 2024

Description

The current implementation of the VP9 payloader produces payloads that are not compatible with iOS. This is because the payloader provides only the muxing strategy called "flexible mode".

According to the VP9 RFC draft, there are two ways to wrap VP9 frames into RTP packets: the "flexible mode" and the "non-flexible mode", with the latter being the preferred one for live-streaming applications. In particular, all browsers encodes VP9 RTP packets in the "non-flexible mode", while iOS supports decoding RTP packets in this mode only, and this is probably a problem shared by other implementations.

This patch improves the VP9 payloader by adding support for the "non-flexible mode". The "flexible mode" is retained and a flag is provided to perform the switch between the two modes.

Reference issue

bluenviron/mediamtx#3060

Copy link

codecov bot commented Apr 27, 2024

Codecov Report

Attention: Patch coverage is 67.27273% with 72 lines in your changes are missing coverage. Please review.

Project coverage is 84.10%. Comparing base (12646b6) to head (251d31c).

Files Patch % Lines
codecs/vp9/header.go 40.90% 47 Missing and 18 partials ⚠️
codecs/vp9/bits.go 88.57% 2 Missing and 2 partials ⚠️
codecs/vp9_packet.go 96.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #269      +/-   ##
==========================================
- Coverage   86.22%   84.10%   -2.12%     
==========================================
  Files          22       24       +2     
  Lines        1749     1950     +201     
==========================================
+ Hits         1508     1640     +132     
- Misses        204      253      +49     
- Partials       37       57      +20     
Flag Coverage Δ
go 84.10% <67.27%> (-2.12%) ⬇️
wasm 83.48% <65.45%> (-2.28%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@aler9 aler9 changed the title fix VP9 decoding on iOS Fix VP9 decoding on iOS Apr 27, 2024
@aler9 aler9 force-pushed the fix/vp9 branch 3 times, most recently from 0532c6a to fdfd0c8 Compare April 27, 2024 19:21
@aler9 aler9 marked this pull request as draft April 27, 2024 19:27
@Sean-Der
Copy link
Member

Thank you @aler9! I am in full support of this, happy to review w/e you are ready.

If Flexible mode doesn't have value (and is a burden) we could drop it.

@aler9 aler9 force-pushed the fix/vp9 branch 4 times, most recently from d7dea6e to 417bf25 Compare April 28, 2024 19:56
The current implementation of the VP9 payloader produces payloads that
are not compatible with iOS. This is because the payloader provides
only the muxing strategy called "flexible mode".

According to the VP9 RFC draft, there are two ways to wrap VP9 frames
into RTP packets: the "flexible mode" and the "non-flexible mode", with
the latter being the preferred one for live-streaming applications. In
particular, all browsers encodes VP9 RTP packets in the "non-flexible
mode", while iOS supports decoding RTP packets in this mode only, and
this is probably a problem shared by other implementations.

This patch improves the VP9 payloader by adding support for the
"non-flexible mode". The "flexible mode" is retained and a flag is
provided to perform the switch between the two modes.
@aler9 aler9 marked this pull request as ready for review April 28, 2024 19:59
@aler9 aler9 requested a review from Sean-Der April 28, 2024 19:59
@aler9
Copy link
Member Author

aler9 commented Apr 28, 2024

Hello @Sean-Der, now everything is ready for review, thanks for being available.
Regarding flexible mode, i don't know whether it has a value or not, keeping it is not a burden and avoids breakages in existing implementations.

@Sean-Der
Copy link
Member

LGTM @aler9! Thank you so much :)

@Sean-Der Sean-Der merged commit bc5124c into pion:master Apr 29, 2024
12 of 14 checks passed
@aler9 aler9 deleted the fix/vp9 branch April 29, 2024 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants