Proposal: Ensemble Protocol

[sr229]

Rationale

As we move on to remote interactions, it still poses a challenge to remotely "connect" virtual avatars. While facilities like VRChat exist, there are users who only need a "multiplayer" like feature in their avatar system. This is the main reason for the Ensemble Project.

Problems

• Real-time communication between networked clients, both locally and in remote cases.
• Allowing non-Vignette clients to transmit over the wire, while maintaining open protocol compliance.
• Allow a first-party method to coordinate and stream collab streams between virtual streamers.

Objectives

• Allow real-time communication between two parties, think "Zoom" for collabs.
• Performant, secure and low-usage communication layer between clients.
• Using existing open protocols to allow seamless communication.

The Ensemble Protocol in view

The protocol is a modified version of VMC over a Websocket/WebRTC tunnel, which allows streaming Model assets and tracking data between clients in a efficient manner. As VMC only defines a way to transmit tracking data to a Marionette provider, Ensemble is designed to transmit both the puppet and the data over the network. Textures streamed over the network are compressed and are rendered utilizing data from the peer.

As we mostly operate in a vendor-neutral space, Transmitting the Model and its metadata should use the Vignette representation, if available, to properly render the puppet in the client's view.

If available, the Ensemble protocol will always choose the fastest route towards the client, which means, if two Ensemble clients are on the same network, they will attempt to connect via LAN first.

Security

The Ensemble Protocol inherits both the speed and security of WireGuard and the Noise protocol, assuring a safe, fast, and secure experience to all compatible clients.

Limitations

Ensemble does not intend to replace VMC or other contemporaries and is intended for multiplayer scenarios. VMC shall be used still for communication between Vignette and VMC-compatible mobile clients.