Skip to content

Commit

Permalink
[WebCodec|AudioDecoder] Limit the number of codec operations we can e…
Browse files Browse the repository at this point in the history
…nqueue

https://bugs.webkit.org/show_bug.cgi?id=284448
rdar://141272065

Reviewed by NOBODY (OOPS!).

While the WebCodecs specs clearly allows for boundless queue for calculating
"Codec Saturdation", a side effect is that both decodeQueueSize and encodeQueueSize
can only ever be observed with a size of 0.
The value of decodeQueueSize and encodeQueueSize is used by some site (including W3C's WebCodecs's own example)
as a way to limit how much in advance it will attempt to encode or decode.

Both Chrome and Firefox have a limit on how many codecs operations (1 for Firefox, between 1 and 8 for Chrome)
they will enqueue concurrently.
To minimise web compatibility issue, in addition to lodging a spec bug (w3c/webcodecs#864)
we also introduce a limit on how many frames with submit to the internal decoder/encoder
before submitting more:
- 1 for Audio
- 4 for Video.

In order of not having to replicate 4 times the code in {Audio|Video}{Encoder|Decoder}
we create a new WebCodecsBase that handles all the control message queue operations
as well as handle how many codec operations have been submitted, and of which
all WebCodecs inherit from.
Doing so allows to remove a lot of similar code across all webcodecs and get
us closer to the verbiage of the specs.

Another benefit is that WebCodecsControlMessage no longer needs to be a templated class,
and we make it return a "Processed" or "Not Processed" value as the specs does.

No change in observable behaviours with existing WPT.
Tested that it allows the WebCodecs reference player to work.
No new tests as none of the work above is per spec.

* Source/WebCore/Headers.cmake:
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp:
(WebCore::WebCodecsAudioDecoder::WebCodecsAudioDecoder):
(WebCore::WebCodecsAudioDecoder::configure):
(WebCore::WebCodecsAudioDecoder::decode):
(WebCore::WebCodecsAudioDecoder::flush):
(WebCore::WebCodecsAudioDecoder::closeDecoder):
(WebCore::WebCodecsAudioDecoder::resetDecoder):
(WebCore::WebCodecsAudioDecoder::stop):
(WebCore::WebCodecsAudioDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h:
(WebCore::WebCodecsAudioDecoder::decodeQueueSize const):
(WebCore::WebCodecsAudioDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.cpp:
(WebCore::WebCodecsAudioEncoder::WebCodecsAudioEncoder):
(WebCore::WebCodecsAudioEncoder::configure):
(WebCore::WebCodecsAudioEncoder::encode):
(WebCore::WebCodecsAudioEncoder::flush):
(WebCore::WebCodecsAudioEncoder::closeEncoder):
(WebCore::WebCodecsAudioEncoder::resetEncoder):
(WebCore::WebCodecsAudioEncoder::stop):
(WebCore::WebCodecsAudioEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsAudioEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsAudioEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsAudioEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsAudioEncoder.h:
(WebCore::WebCodecsAudioEncoder::encodeQueueSize const):
(WebCore::WebCodecsAudioEncoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsBase.cpp: Added.
(WebCore::WebCodecsBase::WebCodecsBase):
(WebCore::WebCodecsBase::queueControlMessageAndProcess):
(WebCore::WebCodecsBase::queueControlMessageForCodecOperationAndProcess):
(WebCore::WebCodecsBase::scheduleDequeueEvent):
(WebCore::WebCodecsBase::processControlMessageQueue):
(WebCore::WebCodecsBase::incrementOperationQueueSize):
(WebCore::WebCodecsBase::decrementOperationQueueSizeAndScheduleDequeueEvent):
(WebCore::WebCodecsBase::decreaseCodecOperationCountAndMaybeProcessControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueue):
(WebCore::WebCodecsBase::clearControlMessageQueueAndMaybeScheduleDequeueEvent):
(WebCore::WebCodecsBase::blockControlMessageQueue):
(WebCore::WebCodecsBase::unblockControlMessageQueue):
(WebCore::WebCodecsBase::virtualHasPendingActivity const):
* Source/WebCore/Modules/webcodecs/WebCodecsBase.h: Copied from Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h.
(WebCore::WebCodecsBase::state const):
(WebCore::WebCodecsBase::setState):
(WebCore::WebCodecsBase::operationQueueSize const):
(WebCore::WebCodecsBase::maximumCodecOperationsEnqueued const):
(WebCore::WebCodecsBase::increaseCodecOperationCount):
(WebCore::WebCodecsBase::isCodecSaturated const):
* Source/WebCore/Modules/webcodecs/WebCodecsControlMessage.h:
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.cpp:
(WebCore::WebCodecsVideoDecoder::WebCodecsVideoDecoder):
(WebCore::WebCodecsVideoDecoder::configure):
(WebCore::WebCodecsVideoDecoder::decode):
(WebCore::WebCodecsVideoDecoder::flush):
(WebCore::WebCodecsVideoDecoder::closeDecoder):
(WebCore::WebCodecsVideoDecoder::resetDecoder):
(WebCore::WebCodecsVideoDecoder::stop):
(WebCore::WebCodecsVideoDecoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoDecoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoDecoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoDecoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoDecoder.h:
(WebCore::WebCodecsVideoDecoder::decodeQueueSize const):
(WebCore::WebCodecsVideoDecoder::state const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.cpp:
(WebCore::WebCodecsVideoEncoder::WebCodecsVideoEncoder):
(WebCore::WebCodecsVideoEncoder::updateRates):
(WebCore::WebCodecsVideoEncoder::configure):
(WebCore::WebCodecsVideoEncoder::encode):
(WebCore::WebCodecsVideoEncoder::flush):
(WebCore::WebCodecsVideoEncoder::closeEncoder):
(WebCore::WebCodecsVideoEncoder::resetEncoder):
(WebCore::WebCodecsVideoEncoder::stop):
(WebCore::WebCodecsVideoEncoder::scheduleDequeueEvent): Deleted.
(WebCore::WebCodecsVideoEncoder::queueControlMessageAndProcess): Deleted.
(WebCore::WebCodecsVideoEncoder::processControlMessageQueue): Deleted.
(WebCore::WebCodecsVideoEncoder::virtualHasPendingActivity const): Deleted.
* Source/WebCore/Modules/webcodecs/WebCodecsVideoEncoder.h:
(WebCore::WebCodecsVideoEncoder::encodeQueueSize const):
(WebCore::WebCodecsVideoEncoder::state const): Deleted.
* Source/WebCore/Sources.txt:
* Source/WebCore/WebCore.xcodeproj/project.pbxproj:
  • Loading branch information
jyavenard committed Dec 12, 2024
1 parent 27cefcf commit 9502a07
Show file tree
Hide file tree
Showing 14 changed files with 407 additions and 401 deletions.
1 change: 1 addition & 0 deletions Source/WebCore/Headers.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -649,6 +649,7 @@ set(WebCore_PRIVATE_FRAMEWORK_HEADERS
Modules/webcodecs/WebCodecsAlphaOption.h
Modules/webcodecs/WebCodecsAudioData.h
Modules/webcodecs/WebCodecsAudioInternalData.h
Modules/webcodecs/WebCodecsBase.h
Modules/webcodecs/WebCodecsEncodedAudioChunk.h
Modules/webcodecs/WebCodecsEncodedAudioChunkData.h
Modules/webcodecs/WebCodecsEncodedAudioChunkType.h
Expand Down
94 changes: 28 additions & 66 deletions Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -32,15 +32,16 @@
#include "ContextDestructionObserverInlines.h"
#include "DOMException.h"
#include "Event.h"
#include "EventNames.h"
#include "JSDOMPromiseDeferred.h"
#include "JSWebCodecsAudioDecoderSupport.h"
#include "ScriptExecutionContext.h"
#include "WebCodecsAudioData.h"
#include "WebCodecsAudioDataOutputCallback.h"
#include "WebCodecsControlMessage.h"
#include "WebCodecsEncodedAudioChunk.h"
#include "WebCodecsErrorCallback.h"
#include "WebCodecsUtilities.h"

#include <wtf/TZoneMallocInlines.h>

WTF_ALLOW_UNSAFE_BUFFER_USAGE_BEGIN
Expand All @@ -57,7 +58,7 @@ Ref<WebCodecsAudioDecoder> WebCodecsAudioDecoder::create(ScriptExecutionContext&
}

WebCodecsAudioDecoder::WebCodecsAudioDecoder(ScriptExecutionContext& context, Init&& init)
: ActiveDOMObject(&context)
: WebCodecsBase(context)
, m_output(init.output.releaseNonNull())
, m_error(init.error.releaseNonNull())
{
Expand Down Expand Up @@ -98,26 +99,26 @@ ExceptionOr<void> WebCodecsAudioDecoder::configure(ScriptExecutionContext&, WebC
if (!isValidDecoderConfig(config))
return Exception { ExceptionCode::TypeError, "Config is not valid"_s };

if (m_state == WebCodecsCodecState::Closed || !scriptExecutionContext())
if (state() == WebCodecsCodecState::Closed || !scriptExecutionContext())
return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is closed"_s };

m_state = WebCodecsCodecState::Configured;
setState(WebCodecsCodecState::Configured);
m_isKeyChunkRequired = true;

bool isSupportedCodec = AudioDecoder::isCodecSupported(config.codec);
queueControlMessageAndProcess({ *this, [this, config = WTFMove(config), isSupportedCodec, identifier = scriptExecutionContext()->identifier()]() mutable {
m_isMessageQueueBlocked = true;
blockControlMessageQueue();

if (!isSupportedCodec) {
postTaskToCodec<WebCodecsAudioDecoder>(identifier, *this, [] (auto& decoder) {
decoder.closeDecoder(Exception { ExceptionCode::NotSupportedError, "Codec is not supported"_s });
});
return;
return WebCodecsControlMessageOutcome::Processed;
}

Ref createDecoderPromise = AudioDecoder::create(config.codec, createAudioDecoderConfig(config), [identifier, weakThis = ThreadSafeWeakPtr { *this }, decoderCount = ++m_decoderCount] (auto&& result) {
postTaskToCodec<WebCodecsAudioDecoder>(identifier, weakThis, [result = WTFMove(result), decoderCount] (auto& decoder) mutable {
if (decoder.m_state != WebCodecsCodecState::Configured || decoder.m_decoderCount != decoderCount)
if (decoder.state() != WebCodecsCodecState::Configured || decoder.m_decoderCount != decoderCount)
return;

if (!result.has_value()) {
Expand All @@ -142,16 +143,16 @@ ExceptionOr<void> WebCodecsAudioDecoder::configure(ScriptExecutionContext&, WebC
}

protectedThis->setInternalDecoder(WTFMove(*result));
protectedThis->m_isMessageQueueBlocked = false;
protectedThis->processControlMessageQueue();
protectedThis->unblockControlMessageQueue();
});
return WebCodecsControlMessageOutcome::Processed;
} });
return { };
}

ExceptionOr<void> WebCodecsAudioDecoder::decode(Ref<WebCodecsEncodedAudioChunk>&& chunk)
{
if (m_state != WebCodecsCodecState::Configured)
if (state() != WebCodecsCodecState::Configured)
return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is not configured"_s };

if (m_isKeyChunkRequired) {
Expand All @@ -160,25 +161,27 @@ ExceptionOr<void> WebCodecsAudioDecoder::decode(Ref<WebCodecsEncodedAudioChunk>&
m_isKeyChunkRequired = false;
}

++m_decodeQueueSize;
queueControlMessageAndProcess({ *this, [this, chunk = WTFMove(chunk)]() mutable {
--m_decodeQueueSize;
scheduleDequeueEvent();

queueControlMessageForCodecOperationAndProcess({ *this, [this, chunk = WTFMove(chunk)]() mutable {
increaseCodecOperationCount();
protectedScriptExecutionContext()->enqueueTaskWhenSettled(Ref { *m_internalDecoder }->decode({ chunk->span(), chunk->type() == WebCodecsEncodedAudioChunkType::Key, chunk->timestamp(), chunk->duration() }), TaskSource::MediaElement, [weakThis = ThreadSafeWeakPtr { *this }, pendingActivity = makePendingActivity(*this)] (auto&& result) {
RefPtr protectedThis = weakThis.get();
if (!protectedThis || !!result)
if (!protectedThis)
return;

protectedThis->closeDecoder(Exception { ExceptionCode::EncodingError, WTFMove(result.error()) });
if (!result)
protectedThis->closeDecoder(Exception { ExceptionCode::EncodingError, WTFMove(result.error()) });
else
protectedThis->decreaseCodecOperationCountAndMaybeProcessControlMessageQueue();
});

return WebCodecsControlMessageOutcome::Processed;
} });
return { };
}

ExceptionOr<void> WebCodecsAudioDecoder::flush(Ref<DeferredPromise>&& promise)
{
if (m_state != WebCodecsCodecState::Configured)
if (state() != WebCodecsCodecState::Configured)
return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is not configured"_s };

m_isKeyChunkRequired = true;
Expand All @@ -189,6 +192,7 @@ ExceptionOr<void> WebCodecsAudioDecoder::flush(Ref<DeferredPromise>&& promise)
if (RefPtr protectedThis = weakThis.get())
protectedThis->m_pendingFlushPromises.removeFirstMatching([&](auto& flushPromise) { return promise.ptr() == flushPromise.ptr(); });
});
return WebCodecsControlMessageOutcome::Processed;
} });
return { };
}
Expand Down Expand Up @@ -226,7 +230,7 @@ ExceptionOr<void> WebCodecsAudioDecoder::closeDecoder(Exception&& exception)
auto result = resetDecoder(exception);
if (result.hasException())
return result;
m_state = WebCodecsCodecState::Closed;
setState(WebCodecsCodecState::Closed);
m_internalDecoder = nullptr;
if (exception.code() != ExceptionCode::AbortError)
m_error->handleEvent(DOMException::create(WTFMove(exception)));
Expand All @@ -236,17 +240,13 @@ ExceptionOr<void> WebCodecsAudioDecoder::closeDecoder(Exception&& exception)

ExceptionOr<void> WebCodecsAudioDecoder::resetDecoder(const Exception& exception)
{
if (m_state == WebCodecsCodecState::Closed)
if (state() == WebCodecsCodecState::Closed)
return Exception { ExceptionCode::InvalidStateError, "AudioDecoder is closed"_s };

m_state = WebCodecsCodecState::Unconfigured;
setState(WebCodecsCodecState::Unconfigured);
if (RefPtr internalDecoder = std::exchange(m_internalDecoder, { }))
internalDecoder->reset();
m_controlMessageQueue.clear();
if (m_decodeQueueSize) {
m_decodeQueueSize = 0;
scheduleDequeueEvent();
}
clearControlMessageQueueAndMaybeScheduleDequeueEvent();

auto promises = std::exchange(m_pendingFlushPromises, { });
for (auto& promise : promises)
Expand All @@ -255,61 +255,23 @@ ExceptionOr<void> WebCodecsAudioDecoder::resetDecoder(const Exception& exception
return { };
}

void WebCodecsAudioDecoder::scheduleDequeueEvent()
{
if (m_dequeueEventScheduled)
return;

m_dequeueEventScheduled = true;
queueTaskKeepingObjectAlive(*this, TaskSource::MediaElement, [this]() mutable {
dispatchEvent(Event::create(eventNames().dequeueEvent, Event::CanBubble::No, Event::IsCancelable::No));
m_dequeueEventScheduled = false;
});
}

void WebCodecsAudioDecoder::setInternalDecoder(Ref<AudioDecoder>&& internalDecoder)
{
m_internalDecoder = WTFMove(internalDecoder);
}

void WebCodecsAudioDecoder::queueControlMessageAndProcess(WebCodecsControlMessage<WebCodecsAudioDecoder>&& message)
{
if (m_isMessageQueueBlocked) {
m_controlMessageQueue.append(WTFMove(message));
return;
}
if (m_controlMessageQueue.isEmpty()) {
message();
return;
}

m_controlMessageQueue.append(WTFMove(message));
processControlMessageQueue();
}

void WebCodecsAudioDecoder::processControlMessageQueue()
{
while (!m_isMessageQueueBlocked && !m_controlMessageQueue.isEmpty())
m_controlMessageQueue.takeFirst()();
}

void WebCore::WebCodecsAudioDecoder::suspend(ReasonForSuspension)
{
}

void WebCodecsAudioDecoder::stop()
{
m_state = WebCodecsCodecState::Closed;
setState(WebCodecsCodecState::Closed);
m_internalDecoder = nullptr;
m_controlMessageQueue.clear();
clearControlMessageQueue();
m_pendingFlushPromises.clear();
}

bool WebCodecsAudioDecoder::virtualHasPendingActivity() const
{
return m_state == WebCodecsCodecState::Configured && (m_decodeQueueSize || m_isMessageQueueBlocked);
}

} // namespace WebCore

WTF_ALLOW_UNSAFE_BUFFER_USAGE_END
Expand Down
31 changes: 3 additions & 28 deletions Source/WebCore/Modules/webcodecs/WebCodecsAudioDecoder.h
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,12 @@

#if ENABLE(WEB_CODECS)

#include "ActiveDOMObject.h"
#include "AudioDecoder.h"
#include "EventTarget.h"
#include "JSDOMPromiseDeferredForward.h"
#include "WebCodecsAudioDecoderConfig.h"
#include "WebCodecsAudioDecoderSupport.h"
#include "WebCodecsCodecState.h"
#include "WebCodecsControlMessage.h"
#include "WebCodecsBase.h"
#include "WebCodecsEncodedAudioChunkType.h"
#include <wtf/Deque.h>
#include <wtf/Vector.h>

namespace WebCore {
Expand All @@ -46,10 +42,7 @@ class WebCodecsEncodedAudioChunk;
class WebCodecsErrorCallback;
class WebCodecsAudioDataOutputCallback;

class WebCodecsAudioDecoder
: public ThreadSafeRefCountedAndCanMakeThreadSafeWeakPtr<WebCodecsAudioDecoder>
, public ActiveDOMObject
, public EventTarget {
class WebCodecsAudioDecoder : public WebCodecsBase {
WTF_MAKE_TZONE_OR_ISO_ALLOCATED(WebCodecsAudioDecoder);
public:
~WebCodecsAudioDecoder();
Expand All @@ -61,8 +54,7 @@ class WebCodecsAudioDecoder

static Ref<WebCodecsAudioDecoder> create(ScriptExecutionContext&, Init&&);

WebCodecsCodecState state() const { return m_state; }
size_t decodeQueueSize() const { return m_decodeQueueSize; }
size_t decodeQueueSize() const { return operationQueueSize(); }

ExceptionOr<void> configure(ScriptExecutionContext&, WebCodecsAudioDecoderConfig&&);
ExceptionOr<void> decode(Ref<WebCodecsEncodedAudioChunk>&&);
Expand All @@ -72,10 +64,6 @@ class WebCodecsAudioDecoder

static void isConfigSupported(ScriptExecutionContext&, WebCodecsAudioDecoderConfig&&, Ref<DeferredPromise>&&);

// ActiveDOMObject.
void ref() const final { ThreadSafeRefCountedAndCanMakeThreadSafeWeakPtr::ref(); }
void deref() const final { ThreadSafeRefCountedAndCanMakeThreadSafeWeakPtr::deref(); }

WebCodecsAudioDataOutputCallback& outputCallbackConcurrently() { return m_output.get(); }
WebCodecsErrorCallback& errorCallbackConcurrently() { return m_error.get(); }

Expand All @@ -85,32 +73,19 @@ class WebCodecsAudioDecoder
// ActiveDOMObject.
void stop() final;
void suspend(ReasonForSuspension) final;
bool virtualHasPendingActivity() const final;

// EventTarget
void refEventTarget() final { ref(); }
void derefEventTarget() final { deref(); }
enum EventTargetInterfaceType eventTargetInterface() const final { return EventTargetInterfaceType::WebCodecsAudioDecoder; }
ScriptExecutionContext* scriptExecutionContext() const final { return ActiveDOMObject::scriptExecutionContext(); }

ExceptionOr<void> closeDecoder(Exception&&);
ExceptionOr<void> resetDecoder(const Exception&);
void setInternalDecoder(Ref<AudioDecoder>&&);
void scheduleDequeueEvent();

void queueControlMessageAndProcess(WebCodecsControlMessage<WebCodecsAudioDecoder>&&);
void processControlMessageQueue();

WebCodecsCodecState m_state { WebCodecsCodecState::Unconfigured };
size_t m_decodeQueueSize { 0 };
Ref<WebCodecsAudioDataOutputCallback> m_output;
Ref<WebCodecsErrorCallback> m_error;
RefPtr<AudioDecoder> m_internalDecoder;
bool m_dequeueEventScheduled { false };
Vector<Ref<DeferredPromise>> m_pendingFlushPromises;
bool m_isKeyChunkRequired { false };
Deque<WebCodecsControlMessage<WebCodecsAudioDecoder>> m_controlMessageQueue;
bool m_isMessageQueueBlocked { false };
size_t m_decoderCount { 0 };
};

Expand Down
Loading

0 comments on commit 9502a07

Please sign in to comment.