Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AudioDecoder should let you specify an format for resulting AudioData #859

Open
jyavenard opened this issue Dec 11, 2024 · 2 comments
Open

Comments

@jyavenard
Copy link
Member

The User Agent is free to choose how the decoding will be performed and what format it will use when creating the AudioData.

In theory, how the data is stored should indeed be an implementation details.

However, what format is chosen will impact on how AudioData.copyTo will behave and how it will interpret the planeIndex parameter.
Additionally, there's a few WPTs that relies on the decoded AudioData to be interleaved and error otherwise (such as https://wpt.live/webcodecs/audio-encoder.https.any.js and its channelNumberVariationTests)

AudioData::copyTo
"The output AudioSampleFormat for the destination data. If not provided, the resulting copy will use this AudioData’s [[format]"

With the default option, if we were to call copyTo with a planeIndex value of 0 and the AudioData was interleaved, we would copy all planes.
While if the AudioData was planar, copyTo with a planeIndex of 0 would copy only that particular plane.

That the internal UA choice on what to use impact the default behaviour for later created objects and change the behaviour means that it cannot be considered an implementation detail, it's not transparent and is directly observable.
And real-life experience already shows web compatibility between Chrome and Firefox (such as https://w3c.github.io/webcodecs/samples/audio-video-player/audio_video_player.html where the code expects the decoded AudioData to be planar and will have an exception as it attempts to copy the planar data in an array of float) as it doesn't specify what format to use with copyTo)

So we should specify what the default format the AudioDecoder should use and provide the ability to override it in the configuration.

@dalecurtis
Copy link
Contributor

IMHO, I don't think we should do this, since it's not always possible to control the output format without conversion. We have the same issues with VideoFrame. Our original ideal with the WebCodecs types was to try to avoid implicit conversions since they've been performance issues historically. Developers should rely on explicit format conversion through copyTo()

@jyavenard
Copy link
Member Author

IMHO, I don't think we should do this, since it's not always possible to control the output format without conversion. We have the same issues with VideoFrame. Our original ideal with the WebCodecs types was to try to avoid implicit conversions since they've been performance issues historically. Developers should rely on explicit format conversion through copyTo()

As I mentioned in my original description, the idea that we have the default format as an implementation detail is a good one,

but when it affects the output and behaviour of AudioData.copyTo later, it throws that idea out of the window.

Like in one case it will throw (if the AudioData internal format was interleaved) because now you need twice as big of an array to receive the result.

Or at least a big warning that web developers should ensure of the format before using the AudioData and not rely on using the default original format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants