Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update encoded explainer to include sample code for relay P2Ps #198

Open
wants to merge 6 commits into
base: create-encoded-explainer
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 58 additions & 12 deletions create-encoded-explainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,27 +13,18 @@ The use cases include:
Addressing these cases in the webrtc-encoded-transform API requires the ability to:

* Create encoded frames
* Clone encoded frames
* Manipulate encoded frame metadata to conform with the requirements of its destination
* Enqueue those frames on a sending or receiving stream that is not the same stream as the frame originated on
* Enqueue those frames on a sending peer connection that is not the same peer connection where the frame originated on

These are not the only functions that are needed to handle the use cases, but this explainer
focuses on the frame creation, cloning and manipulation functions.
focuses on the frame creation and manipulation functionality.

## Approach

The approach proposed as a minimum viable API consists of 3 functions:

* A constructor that takes encoded data + metadata and creates an encoded frame
* A setter that is able to modify some of the values returned from the existing getMetadata method
* A clone operator that takes an existing encoded frame and creates a new, independent encoded frame

The clone operator may be thought of as a convenient shorthand for the constructor:

```
frame.clone = constructor(frame.getMetadata(), frame.data)

```
* A class that allows enqueing encoded frames into a sender peer connection and exposing control signals

## Relevant other needed changes

Expand All @@ -48,3 +39,58 @@ This may include the need for a number of signals and appropriate treatment ther
* If the source is capable of producing I-frames on demand, the handler taking data from it must be able to make those requests appropriately

It is harmful to the model if there are couplings between the source of encoded data and the sink for encoded data that are not exposed to the handler.


## Sample code

Let's take a scenario where P2P relays are used to forward frames. Depending solely on a local peer for the stream is not very reliable in this setup. For consistent, glitch-free, low latency streaming, a redundant PeerConnection delivering the same stream is also setup(`recvPc1` and `recvPc2`). A peer can then choose to relay this stream to the next peer in the network (`relayPc`). We pass the readable stream of the incoming PeerConnections to `transferFrames` function which reads the frames and creates new frames with updated metadata that conforms to the sender PeerConnection's requirements (`getUnifiedMetadata`). We then write one of these frames(`isDuplicate`) to the relay peer PerConnection. `getUnifiedMetadata` and `isDuplicate` are application-specific functions.

```
// code in main.js file
const worker = new Worker('worker.js');

// Let recvPc1, recvPc2 be the receiving PCs.
recvPc{1|2}.ontrack = evt => {
evt.receiver.transform = new RTCRtpScriptTransform(worker, { name: "receiverTransform" });
};


// Let relayPc be the PC used to relay frames to the next peer.
worker.onmessage = evt => {
relayPc.replaceTrack(evt.data);
};
```

```
// code in worker.js file
async function transferFrames(reader, writer, encodedSource) {
if(!reader || !writer || !encodedSource){
return;
}
while (true) {
const {frame, done} = await reader.read();
if (done) return;

let newFrame = new RTCRtpEncodedVideoFrame(frame, getUnifiedMetadata(frame));
if(!isDuplicate(newFrame)) {
encodedSource.enqueue(newFrame);
}
// Put the original frame back in the receiver PC
writer.write(frame);
}
}

// Code to instantiate reader and writer from the RTPReceiver and RTPSender.
onrtctransform = (event) => {
if (event.transformer.options.name == "receiverTransform") {
reader = event.transformer.readable;
writer = event.transformer.writable;
encodedSource = new RTCRtpSenderEncodedSource();
postMessage(encodedSource.handle);
} else {
return;
}

transferFrames(reader, writer, encodedSource);
};
```
Loading