Skip to content

Commit

Permalink
Add AudioEncoderCocoa class, an AudioToolbox implementation of WebCod…
Browse files Browse the repository at this point in the history
…ec's AudioEncoder

https://bugs.webkit.org/show_bug.cgi?id=284019
rdar://140889671

Reviewed by NOBODY (OOPS!).

We add AudioEncoderCocoa, AudioToolbox implementation of WebCodec's AudioEncoder. We only support
encoding to Opus and AAC as the framework doesn't provide encoder for Flac, mp3 and vorbis (decoder only)

Enabled WPT tests.

* LayoutTests/TestExpectations:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any.worker-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any-expected.txt:
* LayoutTests/platform/ios/imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any-expected.txt: Added. The
only failure remaining is related to different default for Opus encoding which yield slightly different values to what expected.
The test performs very rough float comparisons.
* LayoutTests/platform/ios/imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any-expected.txt: Added.
* LayoutTests/platform/mac/imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any-expected.txt: Added.
* LayoutTests/platform/mac/imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any-expected.txt: Added.
* Source/WebCore/Modules/webcodecs/OpusEncoderConfig.h:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::isSupportedEncoderCodec): Add additional checks to ensure that the values provided in the Opus config are sane.
While this isn't probably the best place to do so, as this is common for both GStreamer and Cocoa AudioEncoder, it's the simplest.
(WebCore::isValidEncoderConfig): Make sure sampleRate and numberOfChannels aren't 0.
(WebCore::createAudioEncoderConfig): Increase code readability by using struct's named member initialisation.
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): Both Firefox and Chromes rejects AudioData that has a different sampleRate or numberOfChannels.
While we did the same, we didn't process the errors as per spec which requires that we also change the state to Closed just before
queueing the error.

(WebCore::WebCodecsAudioEncoder::isConfigSupported):
* Source/WebCore/PlatformMac.cmake:
* Source/WebCore/SourcesCocoa.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/platform/AudioEncoder.cpp:
(WebCore::AudioEncoder::create):  Plumb AudioEncoderCocoa.
* Source/WebCore/platform/AudioEncoder.h:
* Source/WebCore/platform/AudioEncoderActiveConfiguration.h: Add default initialiser to comply with more recent clang version.
It allows to make the struct member initialisation optional.
* Source/WebCore/platform/audio/cocoa/AudioDecoderCocoa.cpp:
(WebCore::InternalAudioDecoderCocoa::initialize): Change default AudioData creation to use interleaved frames instead of planar. A few
tests has this expectations, even though this isn't mandated by the specs.
* Source/WebCore/platform/audio/cocoa/AudioEncoderCocoa.cpp: Added.
(WebCore::InternalAudioEncoderCocoa::create):
(WebCore::InternalAudioEncoderCocoa::reset):
(WebCore::InternalAudioEncoderCocoa::converter):
(WebCore::InternalAudioEncoderCocoa::queueSingleton):
(WebCore::AudioEncoderCocoa::create):
(WebCore::AudioEncoderCocoa::AudioEncoderCocoa):
(WebCore::AudioEncoderCocoa::~AudioEncoderCocoa):
(WebCore::AudioEncoderCocoa::encode):
(WebCore::AudioEncoderCocoa::flush):
(WebCore::AudioEncoderCocoa::reset):
(WebCore::AudioEncoderCocoa::close):
(WebCore::InternalAudioEncoderCocoa::InternalAudioEncoderCocoa):
(WebCore::InternalAudioEncoderCocoa::initialize):
(WebCore::InternalAudioEncoderCocoa::compressedAudioOutputBufferCallback):
(WebCore::InternalAudioEncoderCocoa::generateDecoderDescriptionFromSample const):
(WebCore::InternalAudioEncoderCocoa::activeConfiguration const):
(WebCore::InternalAudioEncoderCocoa::processEncodedOutputs):
(WebCore::InternalAudioEncoderCocoa::encode):
(WebCore::InternalAudioEncoderCocoa::flush):
(WebCore::InternalAudioEncoderCocoa::close):
* Source/WebCore/platform/audio/cocoa/AudioEncoderCocoa.h: Copied from Source/WebCore/platform/AudioEncoderActiveConfiguration.h.
  • Loading branch information
jyavenard committed Dec 10, 2024
1 parent 3e90d88 commit 9224027
Show file tree
Hide file tree
Showing 20 changed files with 631 additions and 98 deletions.
4 changes: 0 additions & 4 deletions LayoutTests/TestExpectations
Original file line number Diff line number Diff line change
Expand Up @@ -6834,10 +6834,6 @@ imported/w3c/web-platform-tests/html/dom/render-blocking/element-render-blocking

webkit.org/b/278192 imported/w3c/web-platform-tests/html/dom/render-blocking/element-render-blocking-033.html [ Failure ]

# AudioEncoder implementation missing.
imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any.html [ Failure ]
imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any.html [ Failure ]

webkit.org/b/258192 imported/w3c/web-platform-tests/webcodecs/full-cycle-test.https.any.worker.html?h264_avc [ Pass Failure ]

# These tests are timing out since their import.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

PASS Test the Opus DTX flag works.
FAIL Test the Opus bitrateMode flag works. assert_less_than: expected a number less than 40330 but got 80660
PASS Test the Opus bitrateMode flag works.
PASS Test the AAC bitrateMode flag works.

Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Empty codec
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big assert_unreached: Should have rejected: undefined Reached unreachable code
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus frame duration too small
Expand All @@ -15,15 +15,9 @@ PASS Test that AudioEncoder.configure() rejects invalid config: Missing codec
PASS Test that AudioEncoder.configure() rejects invalid config: Empty codec
PASS Test that AudioEncoder.configure() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero channels assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Bit rate too big assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
PASS Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Zero channels
PASS Test that AudioEncoder.configure() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus frame duration too small
Expand All @@ -43,11 +37,11 @@ PASS Test that AudioEncoder.configure() doesn't support config: Sample rate is t
PASS Test that AudioEncoder.configure() doesn't support config: Way too many channels
PASS Test that AudioEncoder.configure() doesn't support config: Possible future opus codec string
PASS Test that AudioEncoder.configure() doesn't support config: Possible future aac codec string
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}} assert_true: expected true got false
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}}

Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Empty codec
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big assert_unreached: Should have rejected: undefined Reached unreachable code
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus frame duration too small
Expand All @@ -15,15 +15,9 @@ PASS Test that AudioEncoder.configure() rejects invalid config: Missing codec
PASS Test that AudioEncoder.configure() rejects invalid config: Empty codec
PASS Test that AudioEncoder.configure() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero channels assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Bit rate too big assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
PASS Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Zero channels
PASS Test that AudioEncoder.configure() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus frame duration too small
Expand All @@ -43,11 +37,11 @@ PASS Test that AudioEncoder.configure() doesn't support config: Sample rate is t
PASS Test that AudioEncoder.configure() doesn't support config: Way too many channels
PASS Test that AudioEncoder.configure() doesn't support config: Possible future opus codec string
PASS Test that AudioEncoder.configure() doesn't support config: Possible future aac codec string
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}} assert_true: expected true got false
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}}

Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@
PASS Simple audio encoding
PASS Test reset during flush
PASS Encode audio with negative timestamp
FAIL Channel number variation: 1 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Channel number variation: 2 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 3000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 13000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 23000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 33000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 43000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 53000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 63000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 73000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 83000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 93000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Encoding and decoding assert_true: expected true got false
PASS Channel number variation: 1
PASS Channel number variation: 2
PASS Sample rate variation: 3000
PASS Sample rate variation: 13000
PASS Sample rate variation: 23000
PASS Sample rate variation: 33000
PASS Sample rate variation: 43000
PASS Sample rate variation: 53000
PASS Sample rate variation: 63000
PASS Sample rate variation: 73000
PASS Sample rate variation: 83000
PASS Sample rate variation: 93000
PASS Encoding and decoding
PASS Emit decoder config and extra data.
PASS encodeQueueSize test
PASS Test encoding Opus with additional parameters: Empty Opus config
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

FAIL Test the Opus DTX flag works. assert_less_than: expected a number less than 250.5 but got 501
PASS Test the Opus bitrateMode flag works.
PASS Test the AAC bitrateMode flag works.

Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@

PASS Simple audio encoding
PASS Test reset during flush
PASS Encode audio with negative timestamp
PASS Channel number variation: 1
PASS Channel number variation: 2
PASS Sample rate variation: 3000
PASS Sample rate variation: 13000
PASS Sample rate variation: 23000
PASS Sample rate variation: 33000
PASS Sample rate variation: 43000
PASS Sample rate variation: 53000
PASS Sample rate variation: 63000
PASS Sample rate variation: 73000
PASS Sample rate variation: 83000
PASS Sample rate variation: 93000
FAIL Encoding and decoding assert_approx_equals: Difference between input and output is too large. index: 100 channel: 0 input: 0.9659258127212524 output: 0.4632495939731598 expected 0.4632495939731598 +/- 0.5 but got 0.9659258127212524
PASS Emit decoder config and extra data.
PASS encodeQueueSize test
PASS Test encoding Opus with additional parameters: Empty Opus config
PASS Test encoding Opus with additional parameters: Opus with frameDuration
PASS Test encoding Opus with additional parameters: Opus with complexity
PASS Test encoding Opus with additional parameters: Opus with useinbandfec
PASS Test encoding Opus with additional parameters: Opus with usedtx
PASS Test encoding Opus with additional parameters: Opus mixed parameters

Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@

FAIL Test the Opus DTX flag works. assert_less_than: expected a number less than 250.5 but got 501
PASS Test the Opus bitrateMode flag works.
PASS Test the AAC bitrateMode flag works.

Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@

PASS Simple audio encoding
PASS Test reset during flush
PASS Encode audio with negative timestamp
PASS Channel number variation: 1
PASS Channel number variation: 2
PASS Sample rate variation: 3000
PASS Sample rate variation: 13000
PASS Sample rate variation: 23000
PASS Sample rate variation: 33000
PASS Sample rate variation: 43000
PASS Sample rate variation: 53000
PASS Sample rate variation: 63000
PASS Sample rate variation: 73000
PASS Sample rate variation: 83000
PASS Sample rate variation: 93000
FAIL Encoding and decoding assert_approx_equals: Difference between input and output is too large. index: 100 channel: 0 input: 0.9659258127212524 output: 0.4632495939731598 expected 0.4632495939731598 +/- 0.5 but got 0.9659258127212524
PASS Emit decoder config and extra data.
PASS encodeQueueSize test
PASS Test encoding Opus with additional parameters: Empty Opus config
PASS Test encoding Opus with additional parameters: Opus with frameDuration
PASS Test encoding Opus with additional parameters: Opus with complexity
PASS Test encoding Opus with additional parameters: Opus with useinbandfec
PASS Test encoding Opus with additional parameters: Opus with usedtx
PASS Test encoding Opus with additional parameters: Opus mixed parameters

2 changes: 1 addition & 1 deletion Source/WebCore/Modules/webcodecs/OpusEncoderConfig.h
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ struct OpusEncoderConfig {
using BitstreamFormat = OpusBitstreamFormat;
OpusBitstreamFormat format { OpusBitstreamFormat::Opus };
uint64_t frameDuration { 20000 };
std::optional<size_t> complexity;
std::optional<size_t> complexity { 9 };
size_t packetlossperc { 0 };
bool useinbandfec { false };
bool usedtx { false };
Expand Down
Loading

0 comments on commit 9224027

Please sign in to comment.