docs: ADR-204 new comms architecture #272

pentreathm · 2024-10-21T19:19:27Z

No description provided.

cloudflare-workers-and-pages · 2024-10-21T19:19:53Z

Deploying adr with Cloudflare Pages

Latest commit:	`888d9d9`
Status:	✅ Deploy successful!
Preview URL:	https://d4f2014e.adr-cvq.pages.dev
Branch Preview URL:	https://adr-comms-ea.adr-cvq.pages.dev

View logs

nachomazzara

Great work @pentreathm

content/ADR-204-comms-architecture.md

nachomazzara · 2024-10-24T19:24:14Z

content/ADR-204-comms-architecture.md

+
+#### Design Considerations 
+
+The establishment of the connection with the island remains unchanged, as that portion of the protocol is the same. The key difference lies in the types of messages sent through the channel. To establish a connection with a scene-room, a new service called [Gatekeeper](https://github.com/decentraland/comms-gatekeeper/) will be introduced to manage token permissions for the transport layer ([LiveKit](https://livekit.io/)). Only one scene room can be active at a time, as a user can only be present in one scene at any given moment. The scene room operates similarly to the communication service implementation used in worlds, where each scene has its own dedicated room. The Gatekeeper service will also allow scene owners to create authorizations for users, which will be reflected in the LiveKit token. This capability will enable them to moderate voice interactions and data streams within the scene effectively.


Only one scene room can be active at a time,

What is the limit of connected peers and what happpens if that limit is achieved?

LiveKit does not have a hard limit on the number of participants in a room, allowing us to define our own limits as needed based on client needs.

The client receives information about users through the island and scene rooms. It must determine how many avatars it can render efficiently, especially in scenarios with a high number of participants. An approach could be prioritizing the processes of messages from players based on their proximity, rendering only the closest 200 avatars and ignoring the rest. This selective rendering can reduce processing load by discarding non-essential messages. This requires fine-tuning but these behaviors were deferred to a later phase as it is not a problem in the current state of the platform and the door is opened to different possible solutions. The key point is that based on LiveKit, there could be no limit on the amount of players connected to the room and we would probably find some bottlenecks in the bandwidth and the processing of the messages

I will add a comment in the ADR.

kuruk-mm · 2024-10-25T13:21:30Z

Hi, how are you? Great work @pentreathm! I have just a few questions:

Is it possible to connect to multiple scene rooms at the same time, or is only one scene room connection allowed at once?
With this design, are we going to lose the ability to walk around with a friend, explore Decentraland, and voice chat together? It seems like you'll be reconnecting with each scene change, and I think that's one of the best features of Islands. Has this been considered in the design? Are there any future plans for private rooms where we can stay with our friends to mitigate this issue?
I read in the "Consequences" section that you're not going to hear someone who is near you but in a different scene. If the reason for this is to give the scene owner control over the voice chat content, shouldn't the nearby chat be treated the same way?

content/ADR-204-comms-architecture.md

aleortega · 2024-10-25T13:10:03Z

content/ADR-204-comms-architecture.md

+
+![comms](/resources/ADR-204/comms.png)
+
+In the illustrated scenario, Peers 1, 2, and 3 are connected to the scene room `E1`. Additionally, Peer 1 establishes a connection to the island `I1`, while Peers 2 and 3 are linked to the island `I2`. This example serves as a simplified illustration. Users may be far apart within the same scene, but this aims to demonstrate that if a peer is at the edge of a scene, they will be able to see other players who are not within the scene but are within viewing range. On the other hand, the scene channel will be there to share consistent data among the visitors, like a stream of video on a texture or the state of an object like a door that can be opened or closed besides also sharing information about the users visiting the scene. 


Just curious about this scenario:

User A is on scene 101

User B is on scene 102 which borders on scene 101

Both users shares a single channel because of proximity but also are connected to different channels because of the scene distinct (one channel for scene 101 and another one for scene 102)

If User A opens a door in scene 101

And User B after the door was opened, joins scene 101, will it see the door open or close?

The question is if when a user enters a new scene it replicates the current state of the scene that has been changing by another users who were already on that scene.

Technically, yes. The state of the scene (CRDT State) should be saved, and when it enters the scene room, it should receive the state and synchronize the scene.

So User B will see the door opened when it joins the scene and the state gets syncronized.

yes, what @kuruk-mm said is correct.

content/ADR-204-comms-architecture.md

Co-authored-by: Alejo Thomas Ortega <[email protected]> Signed-off-by: pentreathm <[email protected]>

Co-authored-by: Ignacio Mazzara <[email protected]> Signed-off-by: pentreathm <[email protected]>

pentreathm · 2024-10-28T14:59:12Z

Is it possible to connect to multiple scene rooms at the same time, or is only one scene room connection allowed at once?

A user should always be connected to only one scene at a time, only one scene connection should remain open.

With this design, are we going to lose the ability to walk around with a friend, explore Decentraland, and voice chat together? It seems like you'll be reconnecting with each scene change, and I think that's one of the best features of Islands. Has this been considered in the design? Are there any future plans for private rooms where we can stay with our friends to mitigate this issue?

Indeed, this feature will be lost in this initial version, it was a necessary trade-off in the design to give scene owners greater control over audio streams and the ability to moderate content. That said, there is potential for improvements. For instance, if a user is connected to Archipelago but not to a specific Scene, they could leverage that channel to share audio, enabling this use case. Additionally, a future "party mode" could be developed using the comms gatekeeper, allowing users to create a private communication channel with friends. This would facilitate sharing audio and text chat, irrespective of other connections in the environment.

I read in the "Consequences" section that you're not going to hear someone who is near you but in a different scene. If the reason for this is to give the scene owner control over the voice chat content, shouldn't the nearby chat be treated the same way?

yes, and I agree with your point, however, the decision was made because text chat is generally less intrusive for other users compared to voice chat. With voice, individuals can disrupt the experience for others, whereas text chat can simply be muted or turned off when needed.

cc @kuruk-mm

Co-authored-by: Alejo Thomas Ortega <[email protected]> Signed-off-by: pentreathm <[email protected]>

content/ADR-204-comms-architecture.md

* Describe Compression and Encoding algorithms Signed-off-by: Mikhail Agapov <[email protected]>

content/ADR-204-comms-architecture.md

leanmendoza · 2024-12-02T09:58:38Z

content/ADR-204-comms-architecture.md

+**Parcel Linearization**
+
+```csharp
+        public static class GenesisCityData
+        {
+            public static readonly Vector2Int MIN_PARCEL = -150 * Vector2Int.one;
+            public static readonly Vector2Int MAX_PARCEL = new (163, 158);
+        }
+
+        public int MinX => GenesisCityData.MIN_PARCEL.x - terrainData.borderPadding;
+        public int MinY => GenesisCityData.MIN_PARCEL.y - terrainData.borderPadding;
+        public int MaxX => GenesisCityData.MAX_PARCEL.x + terrainData.borderPadding;
+        public int MaxY => GenesisCityData.MAX_PARCEL.y + terrainData.borderPadding;
+
+        public int Encode(Vector2Int parcel) =>
+            parcel.x - MinX + ((parcel.y - MinY) * width);
+
+        public Vector2Int Decode(int index) =>
+            new ((index % width) + MinX, (index / width) + MinY);
+```
+
+`terrainData.borderPadding = 2` in the new client.


I think this definitely MUST NOT be hard-coded here in the ADR. It could be assumed with the actual map size or included in another message with the part of the constants.

Anyway, we should talk about another way to set up the relative parcel you are. Setting 17 bits to this index is a hard-limit that we already are close with Genesis City.

cc @mikhail-dcl

Let's try to avoid referencing new client and make it more generic

If we want to ensure that that different clients can properly decode the message we need to have this included in ADR.
I agree that all the information should be here in the ADR. But we can approach an identical implementation even if we remove the genesis city shape&size from here.

Setting different origins for player to refer the relative position is a good approach. Then, we can take advantage of that for sending less data. But I think it's a bad idea to put Genesis City shape and size here to index parcels. I also think 17bits is a hard-limit and really close to what we currently have for Genesis-City.

I can add the idea of using another message to send the references/settings instead of hard-coding it.

I see, we could expose it in the /about under comms if needed. @mikhail-dcl what do you think about these options?

The number of bits needed to encode parcels can be dynamic but in practice, we never expanded it

I have no objections to having an endpoint to provide us with the Genesis city description.

Still, the size should fit the structure defined as int32 or int64 can't grow per se

We have that data available in the /about endpoint right now.

Example: https://realm-provider-ea.decentraland.org/main/about

We can process the sizes in the configurations.map.sizes and get the minimum bottom, maximum top, minimum left, and maximum right values. Then, we can use these values.

... "configurations": { "map": { ... "sizes": [ { "left": -150, "top": 150, "right": 150, "bottom": -150 }, { "left": 62, "top": 158, "right": 162, "bottom": 151 }, { "left": 151, "top": 150, "right": 163, "bottom": 59 } ], ... } },

If you query Genesis Plaza, you will get:

Maximum right: 163 Minimum left: -150 Maximum top: 158 Minimum bottom: -150

which is the same data used in your definition:

public static readonly Vector2Int MIN_PARCEL = -150 * Vector2Int.one; public static readonly Vector2Int MAX_PARCEL = new (163, 158);

here is the ADR of that spec: https://adr.decentraland.org/adr/ADR-250

We decided to move parcel information to Metadata: it will no longer be compressed and will be set only when the parcel is changed. Thus, your concern will be covered and no extra info is needed.

We need to try it out first in E@ to understand how reliable it is before moving on.

@mikhail-dcl could you please update the ADR so that we can resolve the comments and merge it?

content/ADR-204-comms-architecture.md

leanmendoza · 2024-12-02T10:14:18Z

content/ADR-204-comms-architecture.md

+### Movement (Compressed)
+
+```protobuf
+message MovementCompressed {
+  int32 temporal_data = 1; // bit-compressed: timestamp + animations 
+  int64 movement_data = 2; // bit-compressed: position + velocity 
+}
+```


Hey folks, since adding all the compressed complexity (and the assumption in each part) is not cheap, I imagine you have an analysis report to share the trade-off here. Adding that analysis to understand the real motivation behind this would be interesting. Also, it'd be interesting to have the comparison in the practice (maybe benchmarks or theoretical scenarios):

We have a 48-byte message (uncompressed) vs. a 12-byte message (MovementCompreesed). You could reduce the 48-byte message to 40-byte or fewer with significantly less commitment and assumptions. But let's take 48/12=4; this is the gain ratio. Have you considered the LiveKit overhead that this is sent on top of that? This is directly affected, and it results in a lower gain ratio. Then you also have the WebRTC overhead and then the UDP/IP layer (I imagine this is sent by the unreliable way)

What are the rationale behind each constant? I can guess some of them, but let's not leave it to the imagination.

Instead of having the constant, would it be more suitable to have a MovementHeader (to be announced with the profile maybe) where all the constant are set it up for that profile? It'd be far flexible and you avoid explain the constant here, only the limitations. For instance, you send the minValues, maxValues, nBits, etc.

Co-authored-by: Aga <[email protected]> Signed-off-by: pentreathm <[email protected]>

content/ADR-204-comms-architecture.md

Signed-off-by: pentreathm <[email protected]>

content/ADR-204-comms-architecture.md

aixaCode · 2024-12-03T10:19:06Z

content/ADR-204-comms-architecture.md

+**Parcel Linearization**
+
+```csharp
+        public static class GenesisCityData
+        {
+            public static readonly Vector2Int MIN_PARCEL = -150 * Vector2Int.one;
+            public static readonly Vector2Int MAX_PARCEL = new (163, 158);
+        }
+
+        public int MinX => GenesisCityData.MIN_PARCEL.x - terrainData.borderPadding;
+        public int MinY => GenesisCityData.MIN_PARCEL.y - terrainData.borderPadding;
+        public int MaxX => GenesisCityData.MAX_PARCEL.x + terrainData.borderPadding;
+        public int MaxY => GenesisCityData.MAX_PARCEL.y + terrainData.borderPadding;
+
+        public int Encode(Vector2Int parcel) =>
+            parcel.x - MinX + ((parcel.y - MinY) * width);
+
+        public Vector2Int Decode(int index) =>
+            new ((index % width) + MinX, (index / width) + MinY);
+```
+
+`terrainData.borderPadding = 2` in the new client.


Let's try to avoid referencing new client and make it more generic

aixaCode · 2024-12-03T10:25:12Z

Should we also mark the older comms ADR as deprecated as we introduced the new state?

leanmendoza · 2024-12-03T10:36:24Z

Should we also mark the older comms ADR as deprecated as we introduced the new state?

Only after this had lived for at least year (or a period that makes sense) I'd set the current comms as deprecated. There is no rush; this is still experimental and not implemented by other clients except by the new Unity Explorer.

Co-authored-by: Aga <[email protected]> Signed-off-by: pentreathm <[email protected]>

aixaCode · 2024-12-04T15:42:04Z

Should we also mark the older comms ADR as deprecated as we introduced the new state?

Only after this had lived for at least year (or a period that makes sense) I'd set the current comms as deprecated. There is no rush; this is still experimental and not implemented by other clients except by the new Unity Explorer.

Agree, there is definitely no rush, just would like to avoid situation when potential new clients start using the old version of the comms.

kevinszuchet · 2024-12-25T11:51:27Z

content/ADR-204-comms-architecture.md

+- **Stats**: A service that exposes users' positions and islands to display realm statistics, providing insights into hot scenes and identifying where crowds are gathering.
+- **[NATS](https://nats.io/)**: Message broker to exchange information between services. 
+
+**Rom Limits**: The transport layer (LiveKit) does not impose a strict limit on the number of participants in a single room. The real constraint comes from the end user’s bandwidth and the client’s ability to process data efficiently to render nearby players. In Archipelago, islands are capped at a maximum of 100 players, a limitation originally introduced to prevent performance degradation when rendering large numbers of avatars simultaneously.


Suggested change

**Rom Limits**: The transport layer (LiveKit) does not impose a strict limit on the number of participants in a single room. The real constraint comes from the end user’s bandwidth and the client’s ability to process data efficiently to render nearby players. In Archipelago, islands are capped at a maximum of 100 players, a limitation originally introduced to prevent performance degradation when rendering large numbers of avatars simultaneously.

**Room Limits**: The transport layer (LiveKit) does not impose a strict limit on the number of participants in a single room. The real constraint comes from the end user’s bandwidth and the client’s ability to process data efficiently to render nearby players. In Archipelago, islands are capped at a maximum of 100 players, a limitation originally introduced to prevent performance degradation when rendering large numbers of avatars simultaneously.

kevinszuchet · 2024-12-25T11:53:07Z

public/resources/ADR-204/comms-services.png

Is it possible to fix the typo in WebScoket Connector -> WebSocket Connector? 🙏

pentreathm added 2 commits August 30, 2024 14:46

docs: ADR comms WIP

eda99f8

docs: wip new comms architecture ADR

b1139d9

pentreathm requested a review from a team as a code owner October 21, 2024 19:19

pentreathm requested a review from LautaroPetaccio October 21, 2024 19:19

pentreathm marked this pull request as draft October 21, 2024 19:19

pentreathm changed the title ~~Adr comms ea~~ docs: ADR-204 new comms architecture Oct 21, 2024

pentreathm added 4 commits October 21, 2024 16:24

docs: wip new comms architecture ADR

c033dcc

docs: wip new comms architecture ADR

2177d6d

docs: wip new comms architecture ADR

afdd571

docs: wip new comms architecture ADR

dc40af7

nachomazzara reviewed Oct 24, 2024

View reviewed changes

aleortega reviewed Oct 25, 2024

View reviewed changes

pentreathm and others added 3 commits October 28, 2024 11:02

Update content/ADR-204-comms-architecture.md

d664b47

Co-authored-by: Alejo Thomas Ortega <[email protected]> Signed-off-by: pentreathm <[email protected]>

Update content/ADR-204-comms-architecture.md

251b957

Co-authored-by: Ignacio Mazzara <[email protected]> Signed-off-by: pentreathm <[email protected]>

docs: wip new comms architecture ADR

dbb1655

pentreathm and others added 10 commits October 28, 2024 12:00

Update content/ADR-204-comms-architecture.md

d0ae8de

Co-authored-by: Alejo Thomas Ortega <[email protected]> Signed-off-by: pentreathm <[email protected]>

Update content/ADR-204-comms-architecture.md

c09bb4f

Co-authored-by: Alejo Thomas Ortega <[email protected]> Signed-off-by: pentreathm <[email protected]>

docs: wip new comms architecture ADR

f6ff355

docs: wip new comms architecture ADR

7aee68e

docs: wip new comms architecture ADR

f041a13

docs: wip new comms architecture ADR

0821e4c

docs: wip new comms architecture ADR

5b4e99f

docs: wip new comms architecture ADR

af6c888

docs: wip new comms architecture ADR

d0291c4

docs: wip new comms architecture ADR

f475f62

leanmendoza reviewed Nov 1, 2024

View reviewed changes

content/ADR-204-comms-architecture.md Show resolved Hide resolved

Merge branch 'main' into adr-comms-ea

70a24cf

mikhail-dcl added 2 commits November 29, 2024 13:17

Describe Protocol Messages and Metadata

b7cc3b8

* Describe Compression and Encoding algorithms Signed-off-by: Mikhail Agapov <[email protected]>

Suppress JSON Format

60685e2

aixaCode reviewed Nov 29, 2024

View reviewed changes

content/ADR-204-comms-architecture.md Outdated Show resolved Hide resolved

leanmendoza requested changes Dec 2, 2024

View reviewed changes

Update content/ADR-204-comms-architecture.md

fe16385

Co-authored-by: Aga <[email protected]> Signed-off-by: pentreathm <[email protected]>

pentreathm commented Dec 2, 2024

View reviewed changes

content/ADR-204-comms-architecture.md Outdated Show resolved Hide resolved

Update content/ADR-204-comms-architecture.md

251278e

Signed-off-by: pentreathm <[email protected]>

aixaCode reviewed Dec 3, 2024

View reviewed changes

Update content/ADR-204-comms-architecture.md

888d9d9

Co-authored-by: Aga <[email protected]> Signed-off-by: pentreathm <[email protected]>

kevinszuchet reviewed Dec 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: ADR-204 new comms architecture #272

docs: ADR-204 new comms architecture #272

pentreathm commented Oct 21, 2024

cloudflare-workers-and-pages bot commented Oct 21, 2024 •

edited

Loading

nachomazzara left a comment

nachomazzara Oct 24, 2024

pentreathm Oct 28, 2024

kuruk-mm commented Oct 25, 2024

aleortega Oct 25, 2024

kuruk-mm Oct 26, 2024

pentreathm Oct 28, 2024

pentreathm commented Oct 28, 2024 •

edited

Loading

leanmendoza Dec 2, 2024

leanmendoza Dec 2, 2024

pentreathm Dec 2, 2024

aixaCode Dec 3, 2024

leanmendoza Dec 3, 2024

pentreathm Dec 3, 2024

mikhail-dcl Dec 6, 2024

kuruk-mm Dec 7, 2024

mikhail-dcl Dec 12, 2024

pentreathm Dec 12, 2024

leanmendoza Dec 2, 2024 •

edited

Loading

aixaCode Dec 3, 2024

aixaCode commented Dec 3, 2024

leanmendoza commented Dec 3, 2024

aixaCode commented Dec 4, 2024

kevinszuchet Dec 25, 2024

kevinszuchet Dec 25, 2024


		#### Design Considerations

		The establishment of the connection with the island remains unchanged, as that portion of the protocol is the same. The key difference lies in the types of messages sent through the channel. To establish a connection with a scene-room, a new service called [Gatekeeper](https://github.com/decentraland/comms-gatekeeper/) will be introduced to manage token permissions for the transport layer ([LiveKit](https://livekit.io/)). Only one scene room can be active at a time, as a user can only be present in one scene at any given moment. The scene room operates similarly to the communication service implementation used in worlds, where each scene has its own dedicated room. The Gatekeeper service will also allow scene owners to create authorizations for users, which will be reflected in the LiveKit token. This capability will enable them to moderate voice interactions and data streams within the scene effectively.


		![comms](/resources/ADR-204/comms.png)

		In the illustrated scenario, Peers 1, 2, and 3 are connected to the scene room `E1`. Additionally, Peer 1 establishes a connection to the island `I1`, while Peers 2 and 3 are linked to the island `I2`. This example serves as a simplified illustration. Users may be far apart within the same scene, but this aims to demonstrate that if a peer is at the edge of a scene, they will be able to see other players who are not within the scene but are within viewing range. On the other hand, the scene channel will be there to share consistent data among the visitors, like a stream of video on a texture or the state of an object like a door that can be opened or closed besides also sharing information about the users visiting the scene.

docs: ADR-204 new comms architecture #272

Are you sure you want to change the base?

docs: ADR-204 new comms architecture #272

Conversation

pentreathm commented Oct 21, 2024

cloudflare-workers-and-pages bot commented Oct 21, 2024 • edited Loading

Deploying adr with Cloudflare Pages

nachomazzara left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuruk-mm commented Oct 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pentreathm commented Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leanmendoza Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aixaCode commented Dec 3, 2024

leanmendoza commented Dec 3, 2024

aixaCode commented Dec 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloudflare-workers-and-pages bot commented Oct 21, 2024 •

edited

Loading

pentreathm commented Oct 28, 2024 •

edited

Loading

leanmendoza Dec 2, 2024 •

edited

Loading