Flask-SocketIO client connections failing at just over 1000 connections

[DouglasLapsley]

Ubuntu 20.04 on EC2 instance

Flask-SocketIO 5.1.1
python-engineio 4.2.1
python-socketio 5.4.0
simple-websocket 0.3.0

I'm doing some load testing on this server to maximise the number of client connections I can get out of it. The server python code includes the following main elements:

from flask import Flask, request
from flask_socketio import SocketIO, emit, join_room, leave_room, send

app = Flask(__name__)
app.config['SECRET_KEY'] = appConfig.SECRET_KEY
socketio = SocketIO(app, cors_allowed_origins="*")

@socketio.on('json')
def handle_json(json):
   emit("json", json, broadcast=True, include_self=False)
   

if __name__ == '__main__':
    socketio.run(
        app, 
        host='0.0.0.0', 
        port=443, 
        keyfile='MYPATH/privkey.pem', 
        certfile='MYPATH/fullchain.pem', 
        max_size=100000
    )

Note the max_size arg in socketio.run() passing through to the max_size eventlet param.

I have set the file descriptor limits with sudo nano /etc/security/limits.conf as follow:

* soft nproc 100000
* hard nproc 100000
* soft nofile 100000
* hard nofile 100000

I can confirm this with ulimit -a which gives output:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1806
max locked memory       (kbytes, -l) 65536
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 100000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I am load testing with the following code (reduced for brevity) which creates 2000 connections running from my local machine:

import socketio

URL = "MY_SOCKETIO_SERVER_URL"

clients = []
for k in range(2000):
    sio = socketio.Client()
    sio.connect(URL)
    clients.append(sio)
    
for client in clients:
    client.disconnect()

When I run this, around 1176 connections are made successfully, and then they fail to be created after that point and I get a high CPU process in htop of process /usr/lib/snapd/snapd and the socket.io python app terminates printing the message "Killed" after a few seconds.

I need to support around 5000 connections minimum on this server. I understand that it is possible, but how do I do that please?

[miguelgrinberg]

This package does not manage client connections, and does not impose any limits on how many of them can be active at a time. I think this is something that you need to figure out on the web server side, it might be an issue with eventlet.

[DouglasLapsley]

Thank you so much for the quick reply.

I think you are right. Running the test locally without eventlet installed (which I believe would therefore default to the Flask development server) can connect 5000 clients no problem.

If I install eventlet and re-run it picks up the eventlet install and starts dropping clients at around 500.

Uninstall eventlet and run it again and it connects the 5000 clients again no problem.

I've set the max_size property per suggestions in other posts, but that has no impact. I notice the following in the WSGI part of the eventlet documentation.

"Warning At the moment server() will always wait for active connections to finish before exiting, even if there’s an exception raised inside it (all exceptions are handled the same way, including greenlet.GreenletExit and those inheriting from BaseException).

While this may not be an issue normally, when it comes to long running HTTP connections (like eventlet.websocket) it will become problematic and calling wait() on a thread that runs the server may hang, even after using kill(), as long as there are active connections."

I don't really understand eventlet at all, but might this be something to do with it? Any other thoughts on how to tweak it?

Thanks very much Miguel.

[miguelgrinberg]

I think this is about exiting the server, right? So I don't think it is related to your problem. As I said above, this is really not my domain, you will need to take it up with the eventlet people, or try another web server.

[DouglasLapsley]

OK thanks.

In case it's useful, in the eventlet documentation I notice

server = eventlet.listen(('0.0.0.0', 6000))
pool = eventlet.GreenPool(10000)

Which they say "pool = eventlet.GreenPool(10000) creates a pool of green threads that could handle ten thousand clients."

So I'm wondering if that might be something for me to look at. I can't see the eventlet.GreenPool in your implementation though, so not sure how I'd achieve that.

I'll try the eventlet community and see if they can shed any light on this.

[miguelgrinberg]

In case it isn't clear, you don't have to use app.run, that is provided as a convenience, but you can run your server with your own code if you like. There is nothing special about the server that requires app.run.