WebSockets Integration

[victorgarcia98]

Motivation

In order to support more reactive ways to do flexibility (FCR, aFRR, congestion management), which require faster response times, it would be interesting to support WebSockets in FlexMeasures core.

What are WebSockets?

from Mozilla:

The WebSocket API is an advanced technology that makes it possible to open a two-way interactive communication session between the user's browser and a server. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

In essence, WebSockets clients establish the connection with a WebSockets server through a HTTP handshake. Once the connection is established, a TCP is open between the two peers which is used for the actual communication of the messages.

⚠️ WebSockets != Socket.IO: Socket.IO uses websockets as one of its potential (actually, preferred) underlying protocols and it implements extra functionalities such as connect/disconnect messages and Namespaces.

Benefits of WebSockets

In summary, the main benefits of websockets are: (1) bidirectional communication, (2) less resourceful than REST, (3) Broadcast (5)

(1) Bidirectional communications

We don't have to pull from the client to the server to 'asking for a new event'.

This is ideal for several applications:

• Watchdog: we can ask from the server directly to the clients the current status and not having to wait for the client to report its status periodically.
• Direct control: we can send control signals directly to the energy assets.
• Update schedules on finishing: we can push the schedules to the gateways after finishing the computation in a "callback fashion".

Benefits: responsiveness and less resourceful (we don't need constant pulling).

Drawbacks: requires a complex architecture to handle the events.

(2) Less Resourceful

It turns out that websockets have less overhead because the first connection is established through http and the following messages are send trough tcp (less headers).

(3) Broadcast

Inline with the bidirectional communication, we can broadcast messages to the whole pool of users or to a part of them. This feature would be very interesting to control pools of energy assets, thanks to which, we'll be able to send control signals to the whole portfolios at once.

Modules

There are a lot of python package that implement the websocket protocol but many appear to be unmaintained. Among all of them, I've tried so far the following two:

• Flask-Sock, from the creator of Flask-SocketIO
• websocket, a really popular library

(list last updated on 05/04/2023)

Flask-Sock - 185 ⭐

In bullet-points:

• does not require a greenlet based server (gevent, eventlet) to work.
• flask integrated (blueprints)
• not very mature

Integration

I manage to integrate this into a flexmeasures plugin and It works, but It's a little bit slow.

file: flexmeasures_s2/__init__.py

from flask_sock import Sock

sock = Sock()

flexmeasures_s2_websockets : Blueprint = Blueprint(
    "flexmeasures-s2 Websockets",
    __name__,
    url_prefix="/flexmeasures-s2/websockets/"
) # important: url_prefix needs to finish with a slash

ensure_bp_routes_are_loaded_fresh("events.event")

from flexmeasures_s2.events import event

file: flexmeasures_s2/events/event.py

from flexmeasures_s2 import sock
from flexmeasures_s2 import flexmeasures_s2_websockets

@sock.route('/echo', bp=flexmeasures_s2_websockets)
def echo(ws):
    while True:
        data = ws.receive()
        if data == 'close':
            break
        ws.send(data)

Testing Client:

import asyncio
from websockets.sync.client import connect

def run_client():
    with connect("ws://localhost:5000/flexmeasures-s2/websockets/echo") as websocket:
        websocket.send("Hello world!")
        message = websocket.recv()
        print(f"Received: {message}")

run_client()

Websockets - 4k ⭐

In bulletpoints:

• mature project
• supports both threading and async modes
• rich documentation (e.g. a guide on how to work out a server inside Docker)

There are some features we would like to have and that are not directly covered by websockets protocol but that are covered in the documentation:

• Send Message to Single User
• How do I access the request path?: This is important for Token authentication
• How do I access HTTP headers?: Also important for some kinds of authentication/authorization.

Actually, there's a whole topic on Authentication, so we can take some things from there.

Integration into FlexMeasures

With all that, I have identified the following implementation decisions that we need to address, feel free to add your ideas and concerns:

• What package to use?
  • Already flask-integrated or more established module.
  • Potential (future) maitenance problems.
  • Difficulty to put the solution into production.
• How to handle plugins making use of this functionality?
  • We can make plugins to use a url_prefix for their topic names, i.e: /<plugin-name>/<topic>
  • Create a common topic with an event handler where several plugins can subscribe (pub-sub architecture).
• Standalone server vs Integrated one: that is running in differents port or in the same (through the WSGI).
• Should we make the flexmeasures core run trough websockets alone as well?
• In case we go with the Flask-Sock package, we can just use blueprints ❤️
• In case we go with websockets, we can create a thread from the main app that runs a server or just have a new flexmeasures command to run the server like flexmeasures run.