From 6e98975569cd94733e5be06fd81e50fdddb72a7b Mon Sep 17 00:00:00 2001 From: Palak Agarwal Date: Wed, 2 Aug 2023 13:48:58 +0200 Subject: [PATCH 1/6] Add sample code to create-encoded-explainer --- create-encoded-explainer.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/create-encoded-explainer.md b/create-encoded-explainer.md index edd2f13..48e5094 100644 --- a/create-encoded-explainer.md +++ b/create-encoded-explainer.md @@ -48,3 +48,30 @@ This may include the need for a number of signals and appropriate treatment ther * If the source is capable of producing I-frames on demand, the handler taking data from it must be able to make those requests appropriately It is harmful to the model if there are couplings between the source of encoded data and the sink for encoded data that are not exposed to the handler. + + +## Sample code + +Let's take a scenario where P2P relays are used to forward frames. Depending solely on a local peer for the stream is not very reliable in this setup. For consistent, glitch-free, low latency streaming, a redundant PeerConnection delivering the same stream is also setup(`recvPc1` and `recvPc2`). A peer can then relay the stream to the next peer in the network (`relayPc`). We pass the readable stream of the incoming PeerConnections to `transferFrames` function which reads the frames and modifies the frame's metadata such that two frames with the same payload have the same metadata. We then write one of these frames to the `relayPcWriter`. + +``` +// Let recvPc1, recvPc2 be the receiving PCs. +recvPc{1|2}.ontrack = evt => { + transferFrames(evt.receiver.createEncodedStreams().readable.getReader()); +}; + +// Let relayPc be the PC used to relay frames to the next peer. +relayPcWriter = relayPc.sender.createEncodedStreams().writable.getWriter(); + +async function transferFrames(reader) { + while (true) { + const {frame, done} = await reader.read(); + if (done) return; + + frame.setMetadata(getUnifiedMetadata(frame)); + if(!isDuplicate(frame)) { + relayPcWriter.write(frame); + } + } +} +``` From d66f0bbbf641e65727335f2e5a19a6d61e8907ee Mon Sep 17 00:00:00 2001 From: Palak Agarwal Date: Thu, 3 Aug 2023 10:59:36 +0200 Subject: [PATCH 2/6] Update create-encoded-explainer.md --- create-encoded-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/create-encoded-explainer.md b/create-encoded-explainer.md index 48e5094..c6fe184 100644 --- a/create-encoded-explainer.md +++ b/create-encoded-explainer.md @@ -52,7 +52,7 @@ It is harmful to the model if there are couplings between the source of encoded ## Sample code -Let's take a scenario where P2P relays are used to forward frames. Depending solely on a local peer for the stream is not very reliable in this setup. For consistent, glitch-free, low latency streaming, a redundant PeerConnection delivering the same stream is also setup(`recvPc1` and `recvPc2`). A peer can then relay the stream to the next peer in the network (`relayPc`). We pass the readable stream of the incoming PeerConnections to `transferFrames` function which reads the frames and modifies the frame's metadata such that two frames with the same payload have the same metadata. We then write one of these frames to the `relayPcWriter`. +Let's take a scenario where P2P relays are used to forward frames. Depending solely on a local peer for the stream is not very reliable in this setup. For consistent, glitch-free, low latency streaming, a redundant PeerConnection delivering the same stream is also setup(`recvPc1` and `recvPc2`). A peer can then choose to relay this stream to the next peer in the network (`relayPc`). We pass the readable stream of the incoming PeerConnections to `transferFrames` function which reads the frames and modifies the frame's metadata such that two frames with the same payload have the same metadata (`getUnifiedMetadata`). We then write one of these frames(`isDuplicate`) to the `relayPcWriter`. `getUnifiedMetadata` and `isDuplicate` are application-specific functions. ``` // Let recvPc1, recvPc2 be the receiving PCs. From f559ec4b5e13b61e50300ed3d00cb5f90c25a09b Mon Sep 17 00:00:00 2001 From: Palak Agarwal Date: Wed, 27 Sep 2023 18:26:05 +0200 Subject: [PATCH 3/6] Update create-encoded-explainer.md --- create-encoded-explainer.md | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/create-encoded-explainer.md b/create-encoded-explainer.md index c6fe184..8333354 100644 --- a/create-encoded-explainer.md +++ b/create-encoded-explainer.md @@ -55,23 +55,45 @@ It is harmful to the model if there are couplings between the source of encoded Let's take a scenario where P2P relays are used to forward frames. Depending solely on a local peer for the stream is not very reliable in this setup. For consistent, glitch-free, low latency streaming, a redundant PeerConnection delivering the same stream is also setup(`recvPc1` and `recvPc2`). A peer can then choose to relay this stream to the next peer in the network (`relayPc`). We pass the readable stream of the incoming PeerConnections to `transferFrames` function which reads the frames and modifies the frame's metadata such that two frames with the same payload have the same metadata (`getUnifiedMetadata`). We then write one of these frames(`isDuplicate`) to the `relayPcWriter`. `getUnifiedMetadata` and `isDuplicate` are application-specific functions. ``` +// code in main.js file +const worker = new Worker('worker.js'); + // Let recvPc1, recvPc2 be the receiving PCs. recvPc{1|2}.ontrack = evt => { - transferFrames(evt.receiver.createEncodedStreams().readable.getReader()); + evt.receiver.transform = new RTCRtpScriptTransform(worker, { name: "receiverTransform" }); }; // Let relayPc be the PC used to relay frames to the next peer. -relayPcWriter = relayPc.sender.createEncodedStreams().writable.getWriter(); +relayPc.sender.transform = new RTCRtpScriptTransform(worker, { name: "senderTransform" }); + +``` -async function transferFrames(reader) { +``` +// code in worker.js file +async function transferFrames(reader, writer) { + if(!reader || !writer){ + return; + } while (true) { const {frame, done} = await reader.read(); if (done) return; frame.setMetadata(getUnifiedMetadata(frame)); if(!isDuplicate(frame)) { - relayPcWriter.write(frame); + writer.write(frame); } } } + +// Code to instantiate reader and writer from the RTPReceiver and RTPSender. +onrtctransform = (event) => { + if (event.transformer.options.name == "senderTransform") + writer = event.transformer.writable; + else if (event.transformer.options.name == "receiverTransform") + reader = event.transformer.readable; + else + return; + + transferFrames(reader,writer); +}; ``` From 8a27594df76ae64dc06a3e745fd0af1ea8b0a07a Mon Sep 17 00:00:00 2001 From: guidou Date: Mon, 29 Jan 2024 09:21:56 +0100 Subject: [PATCH 4/6] Update create-encoded-explainer.md --- create-encoded-explainer.md | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/create-encoded-explainer.md b/create-encoded-explainer.md index 8333354..feba9d5 100644 --- a/create-encoded-explainer.md +++ b/create-encoded-explainer.md @@ -13,12 +13,11 @@ The use cases include: Addressing these cases in the webrtc-encoded-transform API requires the ability to: * Create encoded frames -* Clone encoded frames * Manipulate encoded frame metadata to conform with the requirements of its destination * Enqueue those frames on a sending or receiving stream that is not the same stream as the frame originated on These are not the only functions that are needed to handle the use cases, but this explainer -focuses on the frame creation, cloning and manipulation functions. +focuses on the frame creation and manipulation functionality. ## Approach @@ -63,37 +62,43 @@ recvPc{1|2}.ontrack = evt => { evt.receiver.transform = new RTCRtpScriptTransform(worker, { name: "receiverTransform" }); }; -// Let relayPc be the PC used to relay frames to the next peer. -relayPc.sender.transform = new RTCRtpScriptTransform(worker, { name: "senderTransform" }); +// Let relayPc be the PC used to relay frames to the next peer. +worker.onmessage = (e) => { + relayPc.replaceTrack(e.data); +}; ``` ``` // code in worker.js file -async function transferFrames(reader, writer) { - if(!reader || !writer){ +async function transferFrames(reader, writer, encodedSource) { + if(!reader || !writer || !encodedSource){ return; } while (true) { const {frame, done} = await reader.read(); if (done) return; - frame.setMetadata(getUnifiedMetadata(frame)); - if(!isDuplicate(frame)) { - writer.write(frame); + let newFrame = new RTCRtpEncodedVideoFrame(frame, getUnifiedMetadata(frame)); + if(!isDuplicate(newFrame)) { + encodedSource.enqueue(newFrame); } + // Put the original frame back in the receiver PC + writer.write(frame); } } // Code to instantiate reader and writer from the RTPReceiver and RTPSender. onrtctransform = (event) => { - if (event.transformer.options.name == "senderTransform") - writer = event.transformer.writable; - else if (event.transformer.options.name == "receiverTransform") + if (event.transformer.options.name == "receiverTransform") { reader = event.transformer.readable; - else + writer = event.transformer.writable; + encodedSource = new RTCRtpSenderEncodedSource(); + postMessage(encodedSource.handle); + } else { return; + } - transferFrames(reader,writer); + transferFrames(reader, writer, encodedSource); }; ``` From ea81d0d1b645f077c3a7d76cf365c163b5b427cf Mon Sep 17 00:00:00 2001 From: guidou Date: Mon, 29 Jan 2024 09:37:15 +0100 Subject: [PATCH 5/6] Update create-encoded-explainer.md --- create-encoded-explainer.md | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/create-encoded-explainer.md b/create-encoded-explainer.md index feba9d5..d9872dd 100644 --- a/create-encoded-explainer.md +++ b/create-encoded-explainer.md @@ -14,7 +14,7 @@ Addressing these cases in the webrtc-encoded-transform API requires the ability * Create encoded frames * Manipulate encoded frame metadata to conform with the requirements of its destination -* Enqueue those frames on a sending or receiving stream that is not the same stream as the frame originated on +* Enqueue those frames on a sending peer connection that is not the same peer connection where the frame originated on These are not the only functions that are needed to handle the use cases, but this explainer focuses on the frame creation and manipulation functionality. @@ -24,15 +24,7 @@ focuses on the frame creation and manipulation functionality. The approach proposed as a minimum viable API consists of 3 functions: * A constructor that takes encoded data + metadata and creates an encoded frame -* A setter that is able to modify some of the values returned from the existing getMetadata method -* A clone operator that takes an existing encoded frame and creates a new, independent encoded frame - -The clone operator may be thought of as a convenient shorthand for the constructor: - -``` -frame.clone = constructor(frame.getMetadata(), frame.data) - -``` +* A class that allows enqueing encoded frames into a sender peer connection ## Relevant other needed changes @@ -51,7 +43,7 @@ It is harmful to the model if there are couplings between the source of encoded ## Sample code -Let's take a scenario where P2P relays are used to forward frames. Depending solely on a local peer for the stream is not very reliable in this setup. For consistent, glitch-free, low latency streaming, a redundant PeerConnection delivering the same stream is also setup(`recvPc1` and `recvPc2`). A peer can then choose to relay this stream to the next peer in the network (`relayPc`). We pass the readable stream of the incoming PeerConnections to `transferFrames` function which reads the frames and modifies the frame's metadata such that two frames with the same payload have the same metadata (`getUnifiedMetadata`). We then write one of these frames(`isDuplicate`) to the `relayPcWriter`. `getUnifiedMetadata` and `isDuplicate` are application-specific functions. +Let's take a scenario where P2P relays are used to forward frames. Depending solely on a local peer for the stream is not very reliable in this setup. For consistent, glitch-free, low latency streaming, a redundant PeerConnection delivering the same stream is also setup(`recvPc1` and `recvPc2`). A peer can then choose to relay this stream to the next peer in the network (`relayPc`). We pass the readable stream of the incoming PeerConnections to `transferFrames` function which reads the frames and creates new frames with updated metadata that conforms to the sender PeerConnection's requirements (`getUnifiedMetadata`). We then write one of these frames(`isDuplicate`) to the relay peer PerConnection. `getUnifiedMetadata` and `isDuplicate` are application-specific functions. ``` // code in main.js file @@ -64,8 +56,8 @@ recvPc{1|2}.ontrack = evt => { // Let relayPc be the PC used to relay frames to the next peer. -worker.onmessage = (e) => { - relayPc.replaceTrack(e.data); +worker.onmessage = evt => { + relayPc.replaceTrack(evt.data); }; ``` From d7d32ae80941227211cf65afe4195dd4c6b70974 Mon Sep 17 00:00:00 2001 From: guidou Date: Mon, 29 Jan 2024 10:08:17 +0100 Subject: [PATCH 6/6] Update create-encoded-explainer.md --- create-encoded-explainer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/create-encoded-explainer.md b/create-encoded-explainer.md index d9872dd..cf428b5 100644 --- a/create-encoded-explainer.md +++ b/create-encoded-explainer.md @@ -24,7 +24,7 @@ focuses on the frame creation and manipulation functionality. The approach proposed as a minimum viable API consists of 3 functions: * A constructor that takes encoded data + metadata and creates an encoded frame -* A class that allows enqueing encoded frames into a sender peer connection +* A class that allows enqueing encoded frames into a sender peer connection and exposing control signals ## Relevant other needed changes