Questions about Media API

[Sean-Der]

@at-wat I have started sketching out examples using the new media API (it looks really good!) I have some questions.

How do packets flow from RTPLocalTrack -> RTPSender

I have done a simple example here is this what the public API should look like? How do packets flow?

What do you want to do about Samples/Packetization

This is a pretty common use case. What do you think of having a LocalRTPTrack implementer that only does samples? It will handle packetization, and push into the RTPSender

How do you want to handle codec been chosen at signaling time?

This is play-files-from-disk-dynamic mentioned in the README.

Lets say I have two streams H264 and VP8. I want to make an offer that contains both codecs, and then the remote will answer with all the codecs they support.

Does WriteInterceptor need more info?

Should we be doing CongestionControl here? That needs to happen even before the RTP Packets are generated I believe.

How does the WriteInterceptor know when to send a rtx? How does it know the SSRC/PayloadType to send?

Does ReadInterceptor need more info?

How does the ReadInterceptor send RTCP back?

What about LocalMediaStreamTrack and RemoteMediaStreamTrack ?

Would that make more sense for users mapping them to the Javascript API?

[at-wat]

Does WriteInterceptor need more info?

Should we be doing CongestionControl here? That needs to happen even before the RTP Packets are generated I believe.

How does the WriteInterceptor know when to send a rtx? How does it know the SSRC/PayloadType to send?

Does ReadInterceptor need more info?

How does the ReadInterceptor send RTCP back?

I think interceptors like CongestionControl should be constructed and inserted to the pipeline by PeerConnection who has all information including SSRC and PayloadType.
Read/WriteInterceptor chains rtpengine.Reader/Writer which has ReadRTP, WriteRTCP/WriteRTP, ReadRTCP and they are finally connected to RTPSender/Receiver which actually receive/send RTCP packets.

[at-wat]

What about LocalMediaStreamTrack and RemoteMediaStreamTrack ?

As pion/webrtc couldn't actually handle raw media data, it's not same as MediaStreamTrack in WebAPI, I think.

[at-wat]

How do you want to handle codec been chosen at signaling time?

In my understanding of WebRTC API, PeerConnection should provide codec preference information via RTPSender.SetParameter() to RTPSender.

webrtc-v3-design/webrtc/track.go

Lines 65 to 68 in 02ae911

	// SetParameters sets information about how the data is to be encoded.
	// This will be called by PeerConnection according to the result of
	// SDP based negotiation.
	SetParameters(RTPParameters) error

[at-wat]

What do you want to do about Samples/Packetization

#13 is my thought about sample based media processing.
I'm not very sure but maybe there can be an another mutation of the LocalTrack which packetize and feed it to RTPSender.

[at-wat]

How do packets flow from RTPLocalTrack -> RTPSender

Opened a PR to realize this.
Please see #24

[Sean-Der]

@at-wat What do you think of doing Name + Direction instead of Direction + Name to make it easier for users to find things? I think it matches the Go and WebRTC pattern as well

type ByteReader
type ByteScanner
type ByteWriter

RTPSender
RTPReceiver

So we would do TrackRemote and TrackLocal instead.

---

Done in 618a6fe feel free to revert @at-wat if you disagree!

[Sean-Der]

Hey @at-wat I am trying to play with this, but it isn't clicking at all for me :/

Would you mind writing the code that ties this all together? I can fill everything after.

[Sean-Der]

How do you want to handle codec been chosen at signaling time?

In my understanding of WebRTC API, PeerConnection should provide codec preference information via RTPSender.SetParameter() to RTPSender.

webrtc-v3-design/webrtc/track.go

Lines 65 to 68 in 02ae911

	// SetParameters sets information about how the data is to be encoded.
	// This will be called by PeerConnection according to the result of
	// SDP based negotiation.
	SetParameters(RTPParameters) error

So this isn't about preference, but reacting to the offerer. Lets say I have two streams of video foo.h264 and foo.vp8`. As the answerer I should be ready to send other, but how do I know which one gets sent?

[Sean-Der]

I think you have some really good ideas around API design here! I am having a really hard time following along, how can I empower you to get them all out?

I really want to implement these, but having a hard time seeing the path forward. Tell me what I can do to help and would really like to implement all of the https://github.com/pion/webrtc-v3-design/tree/master/examples

I think if we can get these 'working' we can then iterate on the API. We just don't have enough yet to implement them (I think?)

[Sean-Der]

@at-wat would you be up for working together synchronously via Slack sometime on this? If we could spend maybe 30 minutes discussing we could get this done super fast I think!

[at-wat]

So this isn't about preference, but reacting to the offerer. Lets say I have two streams of video foo.h264 and foo.vp8`. As the answerer I should be ready to send other, but how do I know which one gets sent?

I thought following procedure

1. Answerer PeerConnection sets codecs selected by the offerer by RTPSender.SetParameter()
2. RTPSender select one
3. PeerConnection reads selected codec info by RTPSender.Parameters().SelectedCodec
4. PeerConnection generates and send an answer

---

@at-wat would you be up for working together synchronously via Slack sometime on this? If we could spend maybe 30 minutes discussing we could get this done super fast I think!

Sure, I'll be online as possible.

[at-wat]

So we would do TrackRemote and TrackLocal instead.

So, LocalRTPTrack and PassthroughRTPSender should be TrackLocalRTP and RTPSenderPassthrough.