WebSockets connections from common JS clients fail on 1.5.4

[vonnieda]

This is related to this issue in the Paho client: eclipse-paho/paho.mqtt.javascript#169

It seems that any JavaScript WebSockets client connection to Mosquitto 1.5.4 goes into an infinite loop after the connection. On the client side, it seems to see the connection received message an infinite amount of times, eventually crashing the browser.

On the Mosquitto side I logged the following:

1542755237: v13 hdrs completed
1542755237: libwebsocket_parse sees parsing complete
1542755237: lws_parse calling handshake_04
1542755237: WSI_TOKEN_EXTENSIONS = 'permessage-deflate; client_max_window_bits'
1542755237: issuing resp pkt 163 len
1542755237: Allocating RX buffer 4118
1542755237: accepted v13 connection
1542755237: spill on mqttv3.1
1542755237: New client connected from 23.228.140.75 as mqttjs_66bc96cc9bf68a19 (c1, k60).
1542755237: No will message specified.
1542755237: Sending CONNACK to mqttjs_66bc96cc9bf68a19 (0, 0)
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
1542755237: written 6 bytes to client
...

The written 6 bytes to client message repeats forever, until you kill the connection.

The server is running on Ubuntu 14.04 and the config is below:

persistent_client_expiration 1h

# default listener
#port 1883

# TLS listener
#listener 8883
#cafile /etc/mosquitto/certs/ca.crt
#certfile /etc/mosquitto/certs/server.crt
#keyfile /etc/mosquitto/certs/server.key

# WebSockets listener
listener 8000             
protocol websockets

websockets_log_level 255
log_type all

I have tested and reproduced this issue with the Paho client, MQTT.js, and the Hive client.

Edit: It seems Mosquitto is sending an infinite number of CONNACK responses. Here is a snap from a WireShark capture:

[vonnieda]

A bit more info:

• mosquitto 1.5.4 on MacOS installed via brew does not seem to have this problem.
• I performed two WireShark captures - one against the Ubuntu server I'm having problems with and one against MacOS. The two dumps are basically identical except that the Ubuntu just keeps sending the CONNACK over and over with no further traffic from the client.

So, for the MacOS one the dump is:

1. Connect.
2. Upgrade websockets connection.
3. Send CONNECT.
4. Receive CONNACK.

On Ubuntu I get:

1. Connect.
2. Upgrade websockets connection.
3. Send CONNECT.
4. Receive CONNACK over and over until client crashes.

I suspect the problem is actually in libwebsockets. The MacOS binary appears to be linked against version "12" (maybe 1.2?):

> otool -L /usr/local/sbin/mosquitto 
/usr/local/sbin/mosquitto:
	/usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
	/usr/local/opt/libwebsockets/lib/libwebsockets.12.dylib (compatibility version 12.0.0, current version 0.0.0)

While the Ubuntu one is linked against 3:

ubuntu@asdasd:~$ ldd /usr/sbin/mosquitto 
	linux-vdso.so.1 =>  (0x00007ffdb0d7b000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8b4d2ce000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8b4cfc8000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8b4cdc0000)
	libwrap.so.0 => /lib/x86_64-linux-gnu/libwrap.so.0 (0x00007f8b4cbb6000)
	libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f8b4c957000)
	libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f8b4c57b000)
	libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f8b4c376000)
	libanl.so.1 => /lib/x86_64-linux-gnu/libanl.so.1 (0x00007f8b4c172000)
	libwebsockets.so.3 => /usr/lib/x86_64-linux-gnu/libwebsockets.so.3 (0x00007f8b4bf61000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8b4bb98000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f8b4d4d2000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8b4b97a000)
	libnsl.so.1 => /lib/x86_64-linux-gnu/libnsl.so.1 (0x00007f8b4b760000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8b4b547000)

And finally, I found a thread on the libwebsockets mailing list alluding to an infinite loop, but I am not familiar enough with the library to know if it's related: https://libwebsockets.org/pipermail/libwebsockets/2018-August/003811.html

[vonnieda]

Okay, final word for tonight: I recompiled the Ubuntu version against libwebsockets 2.4.2 (like the MacOS) one and it works! So, seems there is a bug related to libwebsockets 3.0.

[ralight]

The library SO version (12 and 3 in what you've quoted above) should be a binary compatibility version unrelated to the actual version of the project. If the developer changes the ABI of the library, it should be incremented. That means that SO version 3 isn't anything to do with version 3.0 of libwebsockets, it's actually a much much older version - version 1.2.2, as is present in ubuntu 14.04. This version has many problems unfortunately.

If you want to use websockets on 14.04 the only realistic option is to compile with a more recent version of libwebsockets, exactly as you have done.

[vonnieda]

Thanks @ralight, that was helpful. I checked the version of the websockets package that was installed when I apt install mosquitto and indeed it is "websockets3" version 1.2.2-1. So perhaps libwebsockets version 3.0 is actually fine - it seems I have not truly tested against it.

I think we could probably close this unless you think it warrants further investigation.

[ralight]

Great, will do!