pion-http-proxy (p2p REST/JSON/GRPC over WebRTC)

[ivelin]

Summary

There are a lot of use cases for easy p2p communication between browser and edge devices. Most stemming from giving users ultimate privacy and control over their own data. Breaking free of big brother cloud providers that offer convenient collection and "care" of user data.

There are many http client and server libraries, a vast body of knowledge and established best practices (e.g REST, JSON, GRPC) over HTTP(1,2,websockets). Implementations are available for most OS platforms and languages. However there is currently no well established project that solves the combination of the two issues: easy p2p over http APIs.

Developers looking to build web browser supported p2p APIs have to dive deep and master the complicated, unfamiliar, low level WebRTC APIs that are also hard to debug and test.

Motivation

A number of projects in the awesome-pion section approach this problem from slightly different perspectives.
https://github.com/jsmouret/grpc-over-webrtc
https://github.com/duality-solutions/web-bridge
https://github.com/rtctunnel/rtctunnel
https://github.com/keroserene/snowflake
https://github.com/decentraland/webrtc-broker
https://github.com/ailabstw/webrtc-socket-proxy
https://github.com/muka/peerjs-go

Maybe it's worth having a community conversation on joining forces for a minimum viable foundation for http over webrtc that enables other higher level solutions to be built on top.

There are two big sub-topics to cover:

1. Signaling
2. Message payload

Signaling

It seems like there is a potential to have a highly available community supported global p2p signaling network of servers.

For example for Ambianic.ai, we rely on the incredibly simple and powerful PeerJS Server that provides an ephemeral, privacy preserving, easy to scale, easy to maintain and extend foundation. We added a peer-room plugin for LAN peer discovery, which solves our use case allowing browser apps to discover and connect to other browser apps or IoT devices on the same WiFi/LAN. The idea and implementation was inspired ShareDrop. More on our particular use case and solution here.

The PeerJS Server deployment has had no unscheduled downtime in 2020 and was last patched over 6 months ago. It's been that reliable, useful and stable.

There are well known community networks for torrents , Tor, Matrix and blockchain support. It only makes sense to have one that has the minimal viable protocol for p2p connectivity that makes no assumptions about how peers will exchange messages once they connect. Nor should it make assumptions about what type of messages will be exchanged.

There is also a growing set of independently maintained public STUN servers, which is the other critical missing piece for p2p comms.

Here is a strawman for a minimal signaling server functionality:

1. Allow peers to register with a crypto safe PeerID (maybe uuid4)
2. Ephemeral (memory or cached) storage for registered peers that expires after a minute unless a peer renews its registration.
3. Allows a peers to send/receive RTC invite/response to/from another peer.

That's essentially what PeerJS Server does today. It has a plug-in architecture that allows higher level signaling protocols to be built on top. Similarly to STUN servers, PeerJS Servers are standalone and can be deployed for a private p2p network or shared with the community for public access.

Can we designate PeerJS Server as the minimum signaling server reference implementation or we need to design something different?

HTTP over WebRTC Payload Format

This problem can be split into stages:

1. Chunking HTTP/1 request and response payloads into pieces that safely travel over webrtc messages between all recent browser end points and pion. This would solve the vast majority of HTTP API use cases (REST, JSON, GRPC).
2. Emulate WebSockets sessions.
3. Emulate HTTP/2 pub/sub notifications.
4. Other protocol mappings as needed.

An initial primitive implementation for stage 1 on the browser side is available here. A corresponding http-over-webrtc-proxy implementation is available here (in Python).

Describe alternatives you've considered

There a few other OSS projects that have overlapping goals, but broader scope.

libp2p (an IPFS sub-project) has a webrtc subgroup with http over webrtc layer.

• https://github.com/libp2p/js-libp2p-webrtc-star
• https://github.com/libp2p/go-libp2p-http

The Matrix team is experimenting with p2p comms over http with browser support:

• https://matrix.org/blog/2020/06/02/introducing-p-2-p-matrix

[ivelin]

@Sean-Der @muka, thoughts?

[jfey]

Interesting post. I am in the process to evaluate options for an industrial project.

Use case goes along this line: Embedded edge device sends data via WebRTC and multiple channels to one or a small number of destinations. It may receive back data via data channel. Not all peers are in the same local network. STUN and TURN will be part of the setup.

To initiate the communication a simple central signalling server, which acts as registration and "matchmaker" providing the SDP offer-answer task. PeerJS sounds good to me, with a few add-ons. Was the topic going forward in the meantime? Any reactions so far?

HTTP over WebRTC lead me to the question if it would make sense to MQTT over WebRTC. Maybe i had a bad moment ..

[ivelin]

Interesting post. I am in the process to evaluate options for an industrial project.

Use case goes along this line: Embedded edge device sends data via WebRTC and multiple channels to one or a small number of destinations. It may receive back data via data channel. Not all peers are in the same local network. STUN and TURN will be part of the setup.

To initiate the communication a simple central signalling server, which acts as registration and "matchmaker" providing the SDP offer-answer task. PeerJS sounds good to me, with a few add-ons. Was the topic going forward in the meantime? Any reactions so far?

HTTP over WebRTC lead me to the question if it would make sense to MQTT over WebRTC. Maybe i had a bad moment ..

This may be relevant for your project:
https://github.com/ambianic/peerfetch

[meduar]

any news?