Unable to get working with Redis & Gevent

[amks1]

I was successful in getting Flask-SocketIO to work with my Flask+React application running inside a container with nginx as the reverse proxy. I added the websocket upgrade lines to the nginx configs and all works fine.

The problem starts when I want to start using Redis queue for websockets. I need this because I want a separate remote application emitting data to my rooms. Here's my basic code:

# run.py
from factory_app import create_app
from  factory_utils import socketio

app = create_app()
socketio.run(app, host='0.0.0.0', port=8000, debug=False)

# factory_utils.py
from flask_socketio import SocketIO

socketio = SocketIO(cors_allowed_origins=["<my_domain>", "http://localhost:5000"])

# factory_app.py
from flask import Flask

def create_app(app_name=__name__, **kwargs):

	app = Flask(app_name, template_folder="./templates")

	#* Flask-RQ2 Configuration
	from factory_utils import rq
	app.config['RQ_REDIS_URL'] = REDIS_URI_RQ
	rq.init_app(app)

	#* SocketIO Configuration
	from factory_utils import socketio
	socketio.init_app(app,  message_queue=REDIS_URI_SOCKETIO)	

	#* SMTP Configuration

	#* General Configuration

	#* Login Manager Configuration
	
	#* Blueprint Configuration
	
	return app

When I run this code on my server, I got this error:

connect handler error
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/engineio/server.py", line 603, in _trigger_event
    return self.handlers[event](*args)
  File "/usr/local/lib/python3.8/site-packages/socketio/server.py", line 725, in _handle_eio_connect
    self.manager.initialize()
  File "/usr/local/lib/python3.8/site-packages/socketio/redis_manager.py", line 65, in initialize
    raise RuntimeError(
RuntimeError: Redis requires a monkey patched socket library to work with gevent

So I changed my run.py to the following:

# run.py
from gevent import monkey
monkey.patch_all()

from factory_app import create_app
from  factory_utils import socketio

app = create_app()
socketio.run(app, host='0.0.0.0', port=8000, debug=False)

Now nginx starts throwing this error:
[error] 10#10: *28 connect() failed (111: Connection refused) while connecting to upstream

The webpage just goes on loading and the server doesn't return any response.

Interestingly, the same code works when I tried on my local machine, and emitting data from a different unrelated process - the React client receives the data. The difference on my local machine is that there's no nginx, and I didn't need to monkey patch anything.

Can someone help me get this working?

[miguelgrinberg]

Why were you able to skip the monkey patching on your local setup? There shouldn't be any difference. Try your production setup locally, just removing nginx but leaving everything else the same.

[amks1]

No. Redis uses network sockets (and it might also use threads). These are blocking operations, they cannot work in a greenlet-based application without monkey patching.

I was using (still using) Python RQ for worker functions using the FLASK-RQ2 package, and that works with Redis without monkey patching. Why the difference with Flask-socketio?

My app is IO based, so I need to go with a greenlet option for my app

I don't understand this. Most web applications are I/O based. What's special about yours that requires greenlets?

To be honest I was told it was better to use greenlets, and frankly I don't understand it well enough. So if I wanted to use threads, can I start a production server with socketio.run(app)?

Or do I need to use something like gunicorn or uwsgi?

[miguelgrinberg]

Have you confirmed 100% that it is RQ that fails to work under gevent with monkey patching? I have never heard of any problems regarding this.

As I said before, the documentation includes all the deployment configurations that are supported. If you want to use threads, you can use socketio.run(app) for development (with the Flask dev server) and Gunicorn in threaded mode for production.

[amks1]

Have you confirmed 100% that it is RQ that fails to work under gevent with monkey patching? I have never heard of any problems regarding this.

What I meant is, I am using a redis queue for Python RQ without monkey patching and it's working perfectly. Only with Flask-socketio it's not working without monkey patching.

[miguelgrinberg]

No, it doesn't actually work perfectly, I guarantee you that. As I said above, Redis uses network sockets, which block the gevent loop. You just don't notice it because RQ isn't constantly reading/writing from Redis.

[amks1]

No, it doesn't actually work perfectly, I guarantee you that. As I said above, Redis uses network sockets, which block the gevent loop. You just don't notice it because RQ isn't constantly reading/writing from Redis.

Ah, I see. It's clear now.

Thanks a lot @miguelgrinberg for your help. I got my app working for now using gunicorn + threads + simple-websocket using the command given in the documentation: gunicorn -w 1 --threads 100 module:app.

The app hanged once and I had to restart the container but since then it's been working ok under low loads. I'm cautiously optimistic that it was some one-off issue which caused it to hang.

[amks1]

I tried it, you're right. Monkey patching breaks my app.

[amks1]

Hi @miguelgrinberg .. I ran into issues with gunicorn. After several hours of idling, it was taking up 100% CPU. And a search showed many people were facing similar issues with gunicorn. So I decided to try uWSGI. But now I'm not able to get websockets to work, it returns 400.

Here's the uWSGI command that I'm running: uwsgi --http :8000 --processes 2 --threads 50 --http-websockets --master --wsgi-file run.py --callable app

My docker-compose logs show the following 2 lines going on forever as long as the client is connected:

nginx            | <my IP> - - [02/Aug/2021:12:23:39 +0000] "GET /socket.io/?EIO=4&transport=websocket HTTP/1.1" 400 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 Edg/92.0.902.55"
backend_app      | [pid: 9|app: 0|req: 1840/1859] 172.18.0.7 () {48 vars in 1420 bytes} [Mon Aug  2 12:22:21 2021] GET /socket.io/?EIO=4&transport=websocket => generated 13 bytes in 0 msecs (HTTP/1.1 400) 3 headers in 156 bytes (1 switches on core 49)

Other than websockets everything else is working fine. What could be the issue?

[miguelgrinberg]

@amks1 you really have to check the documentation to learn what are the valid ways to deploy a Socket.IO server. If it is not documented, chances are it is not a supported solution. uWSGI is only supported with gevent, that's what you missed by not going to the docs.

[amks1]

Hi Miguel thanks for the response but I've been poring over your docs for the past several days.
I did see this line:

When using the uWSGI server in combination with gevent, the Socket.IO server can take advantage of uWSGI’s native WebSocket support.

but it doesn't say explicitly that uWSGI can only be used with gevent. Sorry if I missed it from somewhere else.

[miguelgrinberg]

@amks1 the supported configurations all have detailed instructions in the documentation. The section from where you copied the uWSGI sentence has instructions, actually. You've done something completely different that is not based on those instructions. Also, if the documentation says "When using A with B ..." you can't deduce that it is also okay to use A without B (unless there is also a mention of that explicitly, along with instructions).

[amks1]

Finally I found a solution to my woes, I was able to gevent monkey patch my code using the thread=False argument. Then I am able to use uWSGI and redis queue for SocketIO.

[miguelgrinberg]

And do you know which part of your application requires real threads instead of monkey patched ones? If it was me, I would like to know and understand it well, because in general it is not a good idea to mix real threads with gevent. As I mentioned before, sometimes you get the impression that everything works, but there is blocking that goes unnoticed due to testing with low traffic. Then when you run the application with more users things start to fall apart.

[amks1]

Good point. If it is part of an external library it's going to be very difficult to know. But in my code there are very few places with threads. What approach would you recommend to me, to figure out why monkey patching doesn't work without thread=False?

[miguelgrinberg]

There is no special process to do this, just a standard debugging approach or replacing or removing each instance until you find where the problem is.