Provide quality decision-making

[jvdrhoof]

Provide quality decision-making for the existing adaptive bitrate heuristic used for LL-DASH. Decisions on the quality of the tiles should happen in the following format:

<clientID_0>,<qualTile_0>,<qualTile_1>,…,<qualTile_{n_0 - 1}>;
<clientID_1 >,<qualTile_0>,<qualTile_1>,…,<qualTile_{n_1 - 1}>;
…

E.g., 0,1,1;2,1,−1,1,0;3,1 sent by client 1 indicates that:

• for client 0, the two tiles should be sent at high quality
• for client 1, tiles 0 and 2 should be sent at high quality, tile 3 should be sent at low quality for tile 3, and tile 1 should not be sent at all
• for client 3, the only tile should be sent at low quality

Messages can be sent using the send_control_packet function, made available through the WebRTCConnector DLL. It is not obligatory to send decisions on multiple clients or point cloud objects per time.

[jackjansen]

The place where the high level decision is communicated to the lower layers is PointCloudPipelineOther::SelectTileQualities().

At the moment the implementation of this is a bit of a mess: for each of the transport protocols that implemented multi-quality we try to cast this.reader to the type, and if it works we call the method needed on that casted object.

The quickest solution is to simply add another cast and if statement for the WebRTC reader.

Then later we can look at the pattern and add a general method to the ITransportProtocolReader_tiled interface.

Note that this will give the WebRTC reader the decision for one client_id only, but I think @jvdrhoof said that was good enough for now. Especially for the mmsys measurements where we're initially only interested in the 2-user case anyway.

[jackjansen]

Oh yes: note that the tile and quality indexes are into the IncomingTileDescription[] and its IncomingStreamDescription[] that you got passed as a parameter during the Init() call.

And I think that the streamIndex inside that last structure is a Dash implementation detail, but it may be reusable for WebRTC. Ultimately all the information here is gotten from the PointCloudNetworkTileDescription which is created on the sender side, and then communicated to all receivers through the VR2Gather session communication.

The point cloud pipelines will delay creating the receivers until this information has been received, so in the transport protocol Init() method you already know everything about all streams/tiles/qualities that will be available for this sender.

[jvdrhoof]

I extended RegisterTransmitter so that the number of tiles and quality representations can be passed as input argument to the WebRTC client. When additional encoders are added in config.json, the number of available quality representations is now correctly registered.

I also managed to retrieve the index and quality of a particular tile/stream in WebRTCStreamDescription, so that this information can be propagated. While maybe not the cleanest approach, the AsyncWebRTCWriter now has all information needed to send tiles out at a specific quality, fully compatible with our WebRTC DLL/Go code.

I further managed to include an AsyncWebRTCReader_Tiled in PointCloudPipelineOther.cs, which contains setTileQualities with decisions for every tile passed as a list. This function forwards the decisions through setTileQualities in TransportProtocolWebRTC. The latter generates a control message to the SFU to update the quality decisions.

What I do notice, is that dynamic quality decision-making is not triggered by default. To this end, BaseTileSelector should be run inside a thread, with the Update function calling StartCoroutine(_doSelectTileQualities(selectedTileQualities)); where needed. What configurations do I have to modify to get this to work for WebRTC?

[jackjansen]

Linked to https://github.com/jvdrhoof/mmsys_2025/issues/7

[jackjansen]

First thing I noticed when looking at this code again, after a few months: the TileSelector is disabled in the P_Player prefab and never enabled in code. That may be the basis of our problems: we disabled it when Shishir's tiling experiments were over and done and forgot all about it. Will address that first.

[jackjansen]

That bit is working again. The tile selector is making decisions, and those are implemented (at least: tested with tcp transport).

Unfortunately the decisions are wrong (tested with interactive mode first).

Need to find out where the logic error is. Checked on the sending side: it seems the tiles are ordered by the way they are ordered in reverse of the config LocalUser.PCSelfConfig.Encoders order. So maybe simply putting the qualities in worst-to-best order there is the solution.

But: that will result in the streams starting up in worst quality mode.