Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MediaStream Destination Node #91

Open
justin0mcateer opened this issue Mar 4, 2024 · 7 comments
Open

MediaStream Destination Node #91

justin0mcateer opened this issue Mar 4, 2024 · 7 comments

Comments

@justin0mcateer
Copy link

You mention the MediaStreams API is somewhat out of scope for this library, but there is an implementation (wrapper) of the MediaStream Source Node. Also, the upstream/wrapped crate appears to implement the MediaStream Destination node:
https://github.com/orottier/web-audio-api-rs/blob/main/src/node/media_stream_destination.rs

Aside from perhaps needing to be written manually (vs generated) what are the challenges with creating a MediaStream Desitnation Node wrapper? Is there anyone able to take this on if it were to be sponsored?

@b-ma
Copy link
Collaborator

b-ma commented Mar 4, 2024

Hey, as said in #27 , the issue is more about what it would mean to have a handle to a stream in a JS environment (meaning that another library would be able to use the handle and underlying stream properly), rather than just providing a wrapper for the upstream object.
A possible solution would be consider these just as Node streams (which are not really MediaStreams neither), but then we still need to wait for napi-rs/napi-rs#1483 .
In any case, there is no simple solution right now unfortunately, what would be your use case?

@justin0mcateer
Copy link
Author

I don't actually want to handle the MediaStream directly in the JavaScript, I need to hand it off to another library (WebRTC) that is expecting specifically a MediaStream (not a NodeJS Stream).

@orottier
Copy link
Collaborator

orottier commented Mar 5, 2024

Hey @justin0mcateer, thanks for bringing this up.

The problem is, like @b-ma states, there is no such thing as the MediaStream in NodeJs. So even if we would provide an implementation here, it will not be compatible with other libraries. (The w3c only specifies the API surface and relation to other APIs such as webaudio and webrtc, not the inner workings). See e.g. node-webrtc/node-webrtc#728 (comment) for the issues that will arise.

On the web-audio-api-rs side of things I acknowledge this issue, and we solve it by provinding a real-time safe MediaStream shim that allows you to extract the audio samples ( https://docs.rs/web-audio-api/latest/web_audio_api/media_streams/struct.MediaStreamTrack.html#method.iter ) and it allows you to construct an instance from raw audio data ( https://docs.rs/web-audio-api/latest/web_audio_api/media_streams/struct.MediaStreamTrack.html#method.from_iter ).

The NodeJs ecosystem should provide the same so application authors will be able to glue libraries such as node-web-audio-api and node-webrtc manually. So perhaps this issue could be rewritten to "How to extract raw audio without resorting to AudioWorkletProcessor"?

@b-ma
Copy link
Collaborator

b-ma commented Mar 5, 2024

Thanks @orottier for the further details

I guess the best solution would be to have some compliant C++/Rust/Node implementation of https://www.w3.org/TR/mediacapture-streams/ that everyone agrees to use as middleware and build upon... but this looks like really non-trivial (and a bit like daydreaming for now :)

@b-ma b-ma mentioned this issue Apr 7, 2024
@medemi68
Copy link

Would it be possible to change the audiocontext destination and convert it to a readable stream? How do we extract a stream from node-web-audio-api without it playing from the speakers of the device? If we can extract the writable stream then in @roamhq/wrtc we could probably use audioSource.onData and pump in the raw chunks.

@b-ma
Copy link
Collaborator

b-ma commented May 22, 2024

Hey, thanks for re-opening the discussion

Would it be possible to change the audiocontext destination and convert it to a readable stream?

I'm really not convinced that's the way to go actually. In my opinion, this question must be solved at the lowest possible level and not on the JS side (maybe it could, but it would probably create some kind of problems that nobody wants to spend time on...)

As @orottier mentioned, on the rust side this is quite opened as MediaStream acts as an iterator that is "a real-time safe MediaStream shim that allows you to extract the audio samples". So if you have some kind of C++ implementation of MediaStream in @roamhq/wrtc, I guess the first step would be make a bridge working at the C++/Rust level (I have personally no idea of how this can be achieved, but happy to help)

Then, once done creating a JS facade over that should be rather straightforward (hopefully)

@medemi68
Copy link

medemi68 commented Jun 7, 2024

I'm pretty sure that the whole implementation of @roamhq/wrtc is just a wrapper around a C++ API with bindings for Node.

@ircam-ismm ircam-ismm deleted a comment Jun 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants