[WebCodecs] Limit the number of codec operations we can enqueue

https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by Youenn Fablet.

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode): FlyBy: we need to check the configuration of the AudioData
against the original configuration of the encoder, not the internal configuration which may not always match.
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueCodecControlMessageAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementCodecQueueSize):
(WebCore::WebCodecsBase::decrementCodecQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decrementCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::codecQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::incrementCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:

Canonical link: https://commits.webkit.org/287798@main

Source/WebCore/Headers.cmake

@@ -650,6 +650,7 @@ set(WebCore_PRIVATE_FRAMEWORK_HEADERS
     Modules/webcodecs/WebCodecsAlphaOption.h
     Modules/webcodecs/WebCodecsAudioData.h
     Modules/webcodecs/WebCodecsAudioInternalData.h
+    Modules/webcodecs/WebCodecsBase.h
     Modules/webcodecs/WebCodecsEncodedAudioChunk.h
     Modules/webcodecs/WebCodecsEncodedAudioChunkData.h
     Modules/webcodecs/WebCodecsEncodedAudioChunkType.h

Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp

@@ -31,16 +31,16 @@
 
 #include "ContextDestructionObserverInlines.h"
 #include "DOMException.h"
-#include "Event.h"
-#include "EventNames.h"
 #include "JSDOMPromiseDeferred.h"
 #include "JSWebCodecsAudioDecoderSupport.h"
 #include "ScriptExecutionContext.h"
 #include "WebCodecsAudioData.h"
 #include "WebCodecsAudioDataOutputCallback.h"
+#include "WebCodecsControlMessage.h"
 #include "WebCodecsEncodedAudioChunk.h"
 #include "WebCodecsErrorCallback.h"
 #include "WebCodecsUtilities.h"
+
 #include <wtf/TZoneMallocInlines.h>
 
 WTF_ALLOW_UNSAFE_BUFFER_USAGE_BEGIN
@@ -57,7 +57,7 @@ Ref<WebCodecsAudioDecoder> WebCodecsAudioDecoder::create(ScriptExecutionContext&
 }
 
 WebCodecsAudioDecoder::WebCodecsAudioDecoder(ScriptExecutionContext& context, Init&& init)
-    : ActiveDOMObject(&context)
+    : WebCodecsBase(context)
     , m_output(init.output.releaseNonNull())
     , m_error(init.error.releaseNonNull())
 {
@@ -98,26 +98,26 @@ ExceptionOr<void> WebCodecsAudioDecoder::configure(ScriptExecutionContext&, WebC
     if (!isValidDecoderConfig(config))
         return Exception { ExceptionCode::TypeError, "Config is not valid"_s };
 
-    if (m_state == WebCodecsCodecState::Closed || !scriptExecutionContext())
+    if (state() == WebCodecsCodecState::Closed || !scriptExecutionContext())
         return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is closed"_s };
 
-    m_state = WebCodecsCodecState::Configured;
+    setState(WebCodecsCodecState::Configured);
     m_isKeyChunkRequired = true;
 
     bool isSupportedCodec = AudioDecoder::isCodecSupported(config.codec);
     queueControlMessageAndProcess({ *this, [this, config = WTFMove(config), isSupportedCodec, identifier = scriptExecutionContext()->identifier()]() mutable {
-        m_isMessageQueueBlocked = true;
+        blockControlMessageQueue();
 
         if (!isSupportedCodec) {
             postTaskToCodec<WebCodecsAudioDecoder>(identifier, *this, [] (auto& decoder) {
                 decoder.closeDecoder(Exception { ExceptionCode::NotSupportedError, "Codec is not supported"_s });
             });
-            return;
+            return WebCodecsControlMessageOutcome::Processed;
         }
 
         Ref createDecoderPromise = AudioDecoder::create(config.codec, createAudioDecoderConfig(config), [identifier, weakThis = ThreadSafeWeakPtr { *this }, decoderCount = ++m_decoderCount] (auto&& result) {
             postTaskToCodec<WebCodecsAudioDecoder>(identifier, weakThis, [result = WTFMove(result), decoderCount] (auto& decoder) mutable {
-                if (decoder.m_state != WebCodecsCodecState::Configured || decoder.m_decoderCount != decoderCount)
+                if (decoder.state() != WebCodecsCodecState::Configured || decoder.m_decoderCount != decoderCount)
                     return;
 
                 if (!result.has_value()) {
@@ -142,16 +142,16 @@ ExceptionOr<void> WebCodecsAudioDecoder::configure(ScriptExecutionContext&, WebC
             }
 
             protectedThis->setInternalDecoder(WTFMove(*result));
-            protectedThis->m_isMessageQueueBlocked = false;
-            protectedThis->processControlMessageQueue();
+            protectedThis->unblockControlMessageQueue();
         });
+        return WebCodecsControlMessageOutcome::Processed;
     } });
     return { };
 }
 
 ExceptionOr<void> WebCodecsAudioDecoder::decode(Ref<WebCodecsEncodedAudioChunk>&& chunk)
 {
-    if (m_state != WebCodecsCodecState::Configured)
+    if (state() != WebCodecsCodecState::Configured)
         return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is not configured"_s };
 
     if (m_isKeyChunkRequired) {
@@ -160,25 +160,28 @@ ExceptionOr<void> WebCodecsAudioDecoder::decode(Ref<WebCodecsEncodedAudioChunk>&
         m_isKeyChunkRequired = false;
     }
 
-    ++m_decodeQueueSize;
-    queueControlMessageAndProcess({ *this, [this, chunk = WTFMove(chunk)]() mutable {
-        --m_decodeQueueSize;
-        scheduleDequeueEvent();
-
+    queueCodecControlMessageAndProcess({ *this, [this, chunk = WTFMove(chunk)]() mutable {
+        incrementCodecOperationCount();
         protectedScriptExecutionContext()->enqueueTaskWhenSettled(Ref { *m_internalDecoder }->decode({ chunk->span(), chunk->type() == WebCodecsEncodedAudioChunkType::Key, chunk->timestamp(), chunk->duration() }), TaskSource::MediaElement, [weakThis = ThreadSafeWeakPtr { *this }, pendingActivity = makePendingActivity(*this)] (auto&& result) {
             RefPtr protectedThis = weakThis.get();
-            if (!protectedThis || !!result)
+            if (!protectedThis)
                 return;
 
-            protectedThis->closeDecoder(Exception { ExceptionCode::EncodingError, WTFMove(result.error()) });
+            if (!result) {
+                protectedThis->closeDecoder(Exception { ExceptionCode::EncodingError, WTFMove(result.error()) });
+                return;
+            }
+            protectedThis->decrementCodecOperationCountAndMaybeProcessControlMessageQueue();
         });
+
+        return WebCodecsControlMessageOutcome::Processed;
     } });
     return { };
 }
 
 ExceptionOr<void> WebCodecsAudioDecoder::flush(Ref<DeferredPromise>&& promise)
 {
-    if (m_state != WebCodecsCodecState::Configured)
+    if (state() != WebCodecsCodecState::Configured)
         return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is not configured"_s };
 
     m_isKeyChunkRequired = true;
@@ -189,6 +192,7 @@ ExceptionOr<void> WebCodecsAudioDecoder::flush(Ref<DeferredPromise>&& promise)
             if (RefPtr protectedThis = weakThis.get())
                 protectedThis->m_pendingFlushPromises.removeFirstMatching([&](auto& flushPromise) { return promise.ptr() == flushPromise.ptr(); });
         });
+        return WebCodecsControlMessageOutcome::Processed;
     } });
     return { };
 }
@@ -226,7 +230,7 @@ ExceptionOr<void> WebCodecsAudioDecoder::closeDecoder(Exception&& exception)
     auto result = resetDecoder(exception);
     if (result.hasException())
         return result;
-    m_state = WebCodecsCodecState::Closed;
+    setState(WebCodecsCodecState::Closed);
     m_internalDecoder = nullptr;
     if (exception.code() != ExceptionCode::AbortError)
         m_error->handleEvent(DOMException::create(WTFMove(exception)));
@@ -236,17 +240,13 @@ ExceptionOr<void> WebCodecsAudioDecoder::closeDecoder(Exception&& exception)
 
 ExceptionOr<void> WebCodecsAudioDecoder::resetDecoder(const Exception& exception)
 {
-    if (m_state == WebCodecsCodecState::Closed)
+    if (state() == WebCodecsCodecState::Closed)
         return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is closed"_s };
 
-    m_state = WebCodecsCodecState::Unconfigured;
+    setState(WebCodecsCodecState::Unconfigured);
     if (RefPtr internalDecoder = std::exchange(m_internalDecoder, { }))
         internalDecoder->reset();
-    m_controlMessageQueue.clear();
-    if (m_decodeQueueSize) {
-        m_decodeQueueSize = 0;
-        scheduleDequeueEvent();
-    }
+    clearControlMessageQueueAndMaybeScheduleDequeueEvent();
 
     auto promises = std::exchange(m_pendingFlushPromises, { });
     for (auto& promise : promises)
@@ -255,61 +255,23 @@ ExceptionOr<void> WebCodecsAudioDecoder::resetDecoder(const Exception& exception
     return { };
 }
 
-void WebCodecsAudioDecoder::scheduleDequeueEvent()
-{
-    if (m_dequeueEventScheduled)
-        return;
-
-    m_dequeueEventScheduled = true;
-    queueTaskKeepingObjectAlive(*this, TaskSource::MediaElement, [this]() mutable {
-        dispatchEvent(Event::create(eventNames().dequeueEvent, Event::CanBubble::No, Event::IsCancelable::No));
-        m_dequeueEventScheduled = false;
-    });
-}
-
 void WebCodecsAudioDecoder::setInternalDecoder(Ref<AudioDecoder>&& internalDecoder)
 {
     m_internalDecoder = WTFMove(internalDecoder);
 }
 
-void WebCodecsAudioDecoder::queueControlMessageAndProcess(WebCodecsControlMessage<WebCodecsAudioDecoder>&& message)
-{
-    if (m_isMessageQueueBlocked) {
-        m_controlMessageQueue.append(WTFMove(message));
-        return;
-    }
-    if (m_controlMessageQueue.isEmpty()) {
-        message();
-        return;
-    }
-
-    m_controlMessageQueue.append(WTFMove(message));
-    processControlMessageQueue();
-}
-
-void WebCodecsAudioDecoder::processControlMessageQueue()
-{
-    while (!m_isMessageQueueBlocked && !m_controlMessageQueue.isEmpty())
-        m_controlMessageQueue.takeFirst()();
-}
-
 void WebCore::WebCodecsAudioDecoder::suspend(ReasonForSuspension)
 {
 }
 
 void WebCodecsAudioDecoder::stop()
 {
-    m_state = WebCodecsCodecState::Closed;
+    setState(WebCodecsCodecState::Closed);
     m_internalDecoder = nullptr;
-    m_controlMessageQueue.clear();
+    clearControlMessageQueue();
     m_pendingFlushPromises.clear();
 }
 
-bool WebCodecsAudioDecoder::virtualHasPendingActivity() const
-{
-    return m_state == WebCodecsCodecState::Configured && (m_decodeQueueSize || m_isMessageQueueBlocked);
-}
-
 } // namespace WebCore
 
 WTF_ALLOW_UNSAFE_BUFFER_USAGE_END

Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h

@@ -28,16 +28,12 @@
 
 #if ENABLE(WEB_CODECS)
 
-#include "ActiveDOMObject.h"
 #include "AudioDecoder.h"
-#include "EventTarget.h"
 #include "JSDOMPromiseDeferredForward.h"
 #include "WebCodecsAudioDecoderConfig.h"
 #include "WebCodecsAudioDecoderSupport.h"
-#include "WebCodecsCodecState.h"
-#include "WebCodecsControlMessage.h"
+#include "WebCodecsBase.h"
 #include "WebCodecsEncodedAudioChunkType.h"
-#include <wtf/Deque.h>
 #include <wtf/Vector.h>
 
 namespace WebCore {
@@ -46,10 +42,7 @@ class WebCodecsEncodedAudioChunk;
 class WebCodecsErrorCallback;
 class WebCodecsAudioDataOutputCallback;
 
-class WebCodecsAudioDecoder
-    : public ThreadSafeRefCountedAndCanMakeThreadSafeWeakPtr<WebCodecsAudioDecoder>
-    , public ActiveDOMObject
-    , public EventTarget {
+class WebCodecsAudioDecoder : public WebCodecsBase {
     WTF_MAKE_TZONE_OR_ISO_ALLOCATED(WebCodecsAudioDecoder);
 public:
     ~WebCodecsAudioDecoder();
@@ -61,8 +54,7 @@ class WebCodecsAudioDecoder
 
     static Ref<WebCodecsAudioDecoder> create(ScriptExecutionContext&, Init&&);
 
-    WebCodecsCodecState state() const { return m_state; }
-    size_t decodeQueueSize() const { return m_decodeQueueSize; }
+    size_t decodeQueueSize() const { return codecQueueSize(); }
 
     ExceptionOr<void> configure(ScriptExecutionContext&, WebCodecsAudioDecoderConfig&&);
     ExceptionOr<void> decode(Ref<WebCodecsEncodedAudioChunk>&&);
@@ -72,10 +64,6 @@ class WebCodecsAudioDecoder
 
     static void isConfigSupported(ScriptExecutionContext&, WebCodecsAudioDecoderConfig&&, Ref<DeferredPromise>&&);
 
-    // ActiveDOMObject.
-    void ref() const final { ThreadSafeRefCountedAndCanMakeThreadSafeWeakPtr::ref(); }
-    void deref() const final { ThreadSafeRefCountedAndCanMakeThreadSafeWeakPtr::deref(); }
-
     WebCodecsAudioDataOutputCallback& outputCallbackConcurrently() { return m_output.get(); }
     WebCodecsErrorCallback& errorCallbackConcurrently() { return m_error.get(); }
 
@@ -85,32 +73,19 @@ class WebCodecsAudioDecoder
     // ActiveDOMObject.
     void stop() final;
     void suspend(ReasonForSuspension) final;
-    bool virtualHasPendingActivity() const final;
 
     // EventTarget
-    void refEventTarget() final { ref(); }
-    void derefEventTarget() final { deref(); }
     enum EventTargetInterfaceType eventTargetInterface() const final { return EventTargetInterfaceType::WebCodecsAudioDecoder; }
-    ScriptExecutionContext* scriptExecutionContext() const final { return ActiveDOMObject::scriptExecutionContext(); }
 
     ExceptionOr<void> closeDecoder(Exception&&);
     ExceptionOr<void> resetDecoder(const Exception&);
     void setInternalDecoder(Ref<AudioDecoder>&&);
-    void scheduleDequeueEvent();
-
-    void queueControlMessageAndProcess(WebCodecsControlMessage<WebCodecsAudioDecoder>&&);
-    void processControlMessageQueue();
 
-    WebCodecsCodecState m_state { WebCodecsCodecState::Unconfigured };
-    size_t m_decodeQueueSize { 0 };
     Ref<WebCodecsAudioDataOutputCallback> m_output;
     Ref<WebCodecsErrorCallback> m_error;
     RefPtr<AudioDecoder> m_internalDecoder;
-    bool m_dequeueEventScheduled { false };
     Vector<Ref<DeferredPromise>> m_pendingFlushPromises;
     bool m_isKeyChunkRequired { false };
-    Deque<WebCodecsControlMessage<WebCodecsAudioDecoder>> m_controlMessageQueue;
-    bool m_isMessageQueueBlocked { false };
     size_t m_decoderCount { 0 };
 };