Skip to content

Commit

Permalink
Add AudioEncoderCocoa class, an AudioToolbox implementation of WebCod…
Browse files Browse the repository at this point in the history
…ec's AudioEncoder

https://bugs.webkit.org/show_bug.cgi?id=284019
rdar://140889671

Reviewed by NOBODY (OOPS!).

We add AudioEncoderCocoa, AudioToolbox implementation of WebCodec's AudioEncoder. We only support
encoding to Opus and AAC as the framework doesn't provide encoder for Flac, mp3 and vorbis (decoder only)

Enabled WPT tests.

* LayoutTests/TestExpectations:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any.worker-expected.txt:
* LayoutTests/imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any-expected.txt:
* LayoutTests/platform/glib/TestExpectations: Remove passing expectations, add the remaining ones that do fail.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any-expected.txt: Removed.
* LayoutTests/platform/glib/imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any.worker-expected.txt: Removed.
* LayoutTests/platform/ios/imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any-expected.txt: Added. The
only failure remaining is related to different default for Opus encoding which yield slightly different values to what expected.
The test performs very rough float comparisons.
* LayoutTests/platform/ios/imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any-expected.txt: Added.
* LayoutTests/platform/mac/imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any-expected.txt: Added.
* LayoutTests/platform/mac/imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any-expected.txt: Added.
* Source/WebCore/Modules/webcodecs/OpusEncoderConfig.h:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::isSupportedEncoderCodec): Add additional checks to ensure that the values provided in the Opus config are sane.
While this isn't probably the best place to do so, as this is common for both GStreamer and Cocoa AudioEncoder, it's the simplest.
(WebCore::isValidEncoderConfig): Make sure sampleRate and numberOfChannels aren't 0.
(WebCore::createAudioEncoderConfig): Increase code readability by using struct's named member initialisation.
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): Both Firefox and Chromes rejects AudioData that has a different sampleRate or numberOfChannels.
While we did the same, we didn't process the errors as per spec which requires that we also change the state to Closed just before
queueing the error.

(WebCore::WebCodecsAudioEncoder::isConfigSupported):
* Source/WebCore/PlatformMac.cmake:
* Source/WebCore/SourcesCocoa.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
* Source/WebCore/platform/AudioEncoder.cpp:
(WebCore::AudioEncoder::create):  Plumb AudioEncoderCocoa.
* Source/WebCore/platform/AudioEncoder.h:
* Source/WebCore/platform/AudioEncoderActiveConfiguration.h: Add default initialiser to comply with more recent clang version.
It allows to make the struct member initialisation optional.
* Source/WebCore/platform/audio/cocoa/AudioDecoderCocoa.cpp:
(WebCore::InternalAudioDecoderCocoa::initialize): Change default AudioData creation to use interleaved frames instead of planar. A few
tests has this expectations, even though this isn't mandated by the specs.
* Source/WebCore/platform/audio/cocoa/AudioEncoderCocoa.cpp: Added.
(WebCore::InternalAudioEncoderCocoa::create):
(WebCore::InternalAudioEncoderCocoa::reset):
(WebCore::InternalAudioEncoderCocoa::converter):
(WebCore::InternalAudioEncoderCocoa::queueSingleton):
(WebCore::AudioEncoderCocoa::create):
(WebCore::AudioEncoderCocoa::AudioEncoderCocoa):
(WebCore::AudioEncoderCocoa::~AudioEncoderCocoa):
(WebCore::AudioEncoderCocoa::encode):
(WebCore::AudioEncoderCocoa::flush):
(WebCore::AudioEncoderCocoa::reset):
(WebCore::AudioEncoderCocoa::close):
(WebCore::InternalAudioEncoderCocoa::InternalAudioEncoderCocoa):
(WebCore::InternalAudioEncoderCocoa::initialize):
(WebCore::InternalAudioEncoderCocoa::compressedAudioOutputBufferCallback):
(WebCore::InternalAudioEncoderCocoa::generateDecoderDescriptionFromSample const):
(WebCore::InternalAudioEncoderCocoa::activeConfiguration const):
(WebCore::InternalAudioEncoderCocoa::processEncodedOutputs):
(WebCore::InternalAudioEncoderCocoa::encode):
(WebCore::InternalAudioEncoderCocoa::flush):
(WebCore::InternalAudioEncoderCocoa::close):
* Source/WebCore/platform/audio/cocoa/AudioEncoderCocoa.h: Copied from Source/WebCore/platform/AudioEncoderActiveConfiguration.h.
  • Loading branch information
jyavenard committed Dec 12, 2024
1 parent f52abbb commit 2b0ba8e
Show file tree
Hide file tree
Showing 25 changed files with 650 additions and 216 deletions.
4 changes: 0 additions & 4 deletions LayoutTests/TestExpectations
Original file line number Diff line number Diff line change
Expand Up @@ -6410,10 +6410,6 @@ imported/w3c/web-platform-tests/html/dom/render-blocking/element-render-blocking

webkit.org/b/278192 imported/w3c/web-platform-tests/html/dom/render-blocking/element-render-blocking-033.html [ Failure ]

# AudioEncoder implementation missing.
imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any.html [ Failure ]
imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any.html [ Failure ]

webkit.org/b/258192 imported/w3c/web-platform-tests/webcodecs/full-cycle-test.https.any.worker.html?h264_avc [ Pass Failure ]

# These tests are timing out since their import.
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

PASS Test the Opus DTX flag works.
FAIL Test the Opus bitrateMode flag works. assert_less_than: expected a number less than 40330 but got 80660
PASS Test the Opus bitrateMode flag works.
PASS Test the AAC bitrateMode flag works.

Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Empty codec
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big assert_unreached: Should have rejected: undefined Reached unreachable code
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus frame duration too small
Expand All @@ -15,15 +15,9 @@ PASS Test that AudioEncoder.configure() rejects invalid config: Missing codec
PASS Test that AudioEncoder.configure() rejects invalid config: Empty codec
PASS Test that AudioEncoder.configure() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero channels assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Bit rate too big assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
PASS Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Zero channels
PASS Test that AudioEncoder.configure() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus frame duration too small
Expand All @@ -43,11 +37,11 @@ PASS Test that AudioEncoder.configure() doesn't support config: Sample rate is t
PASS Test that AudioEncoder.configure() doesn't support config: Way too many channels
PASS Test that AudioEncoder.configure() doesn't support config: Possible future opus codec string
PASS Test that AudioEncoder.configure() doesn't support config: Possible future aac codec string
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}} assert_true: expected true got false
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}}

Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Empty codec
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big assert_unreached: Should have rejected: undefined Reached unreachable code
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Zero channels
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.isConfigSupported() rejects invalid config: Opus frame duration too small
Expand All @@ -15,15 +15,9 @@ PASS Test that AudioEncoder.configure() rejects invalid config: Missing codec
PASS Test that AudioEncoder.configure() rejects invalid config: Empty codec
PASS Test that AudioEncoder.configure() rejects invalid config: Missing sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Missing numberOfChannels
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Zero channels assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
FAIL Test that AudioEncoder.configure() rejects invalid config: Bit rate too big assert_throws_js: function "() => {
codec.configure(entry.config);
}" did not throw
PASS Test that AudioEncoder.configure() rejects invalid config: Zero sampleRate
PASS Test that AudioEncoder.configure() rejects invalid config: Zero channels
PASS Test that AudioEncoder.configure() rejects invalid config: Bit rate too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus complexity too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus packetlossperc too big
PASS Test that AudioEncoder.configure() rejects invalid config: Opus frame duration too small
Expand All @@ -43,11 +37,11 @@ PASS Test that AudioEncoder.configure() doesn't support config: Sample rate is t
PASS Test that AudioEncoder.configure() doesn't support config: Way too many channels
PASS Test that AudioEncoder.configure() doesn't support config: Possible future opus codec string
PASS Test that AudioEncoder.configure() doesn't support config: Possible future aac codec string
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}} assert_true: expected true got false
FAIL AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}} assert_true: expected true got false
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":8000,"numberOfChannels":1}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"constant","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"bitrate":128000,"bitrateMode":"variable","bogus":123}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"complexity":5,"frameDuration":20000,"packetlossperc":10,"useinbandfec":true}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{"format":"opus","complexity":10,"frameDuration":60000,"packetlossperc":20,"usedtx":true,"bogus":456}}
PASS AudioEncoder.isConfigSupported() supports: {"codec":"opus","sampleRate":48000,"numberOfChannels":2,"opus":{}}

Original file line number Diff line number Diff line change
Expand Up @@ -2,19 +2,19 @@
PASS Simple audio encoding
PASS Test reset during flush
PASS Encode audio with negative timestamp
FAIL Channel number variation: 1 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Channel number variation: 2 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 3000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 13000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 23000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 33000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 43000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 53000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 63000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 73000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 83000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Sample rate variation: 93000 assert_unreached: Should have rejected: undefined Reached unreachable code
FAIL Encoding and decoding assert_true: expected true got false
PASS Channel number variation: 1
PASS Channel number variation: 2
PASS Sample rate variation: 3000
PASS Sample rate variation: 13000
PASS Sample rate variation: 23000
PASS Sample rate variation: 33000
PASS Sample rate variation: 43000
PASS Sample rate variation: 53000
PASS Sample rate variation: 63000
PASS Sample rate variation: 73000
PASS Sample rate variation: 83000
PASS Sample rate variation: 93000
PASS Encoding and decoding
PASS Emit decoder config and extra data.
PASS encodeQueueSize test
PASS Test encoding Opus with additional parameters: Empty Opus config
Expand Down
12 changes: 4 additions & 8 deletions LayoutTests/platform/glib/TestExpectations
Original file line number Diff line number Diff line change
Expand Up @@ -1301,14 +1301,10 @@ imported/w3c/web-platform-tests/webcodecs/audioDecoder-codec-specific.https.any.
# A bit flaky on Debug bot.
imported/w3c/web-platform-tests/webcodecs/audioDecoder-codec-specific.https.any.html?mp4_aac [ Pass Failure ]

imported/w3c/web-platform-tests/webcodecs/encoded-audio-chunk.any.html [ Pass ]
imported/w3c/web-platform-tests/webcodecs/encoded-audio-chunk.any.worker.html [ Pass ]
imported/w3c/web-platform-tests/webcodecs/encoded-audio-chunk.crossOriginIsolated.https.any.html [ Pass ]
imported/w3c/web-platform-tests/webcodecs/encoded-audio-chunk.crossOriginIsolated.https.any.worker.html [ Pass ]
imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any.html [ Pass ]
imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any.html [ Pass ]
imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any.worker.html [ Pass ]
imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any.html [ Pass DumpJSConsoleLogInStdErr ]
webkit.org/b/284426 imported/w3c/web-platform-tests/webcodecs/audio-encoder-codec-specific.https.any.html [ Failure ]
webkit.org/b/284428 imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any.html [ Failure ]
webkit.org/b/284428 imported/w3c/web-platform-tests/webcodecs/audio-encoder-config.https.any.worker.html [ Failure ]
webkit.org/b/284429 imported/w3c/web-platform-tests/webcodecs/audio-encoder.https.any.html [ Failure ]
http/wpt/webcodecs/encoder-task-failing.html [ Pass ]

# H.264 high-4:2:2 encoding is supported in the GStreamer ports, so this test (checking that profile
Expand Down
Loading

0 comments on commit 2b0ba8e

Please sign in to comment.