FusionAuth locks up on Ubuntu - SSL memory error

[DaveInAus]

FusionAuth locks up on Ubuntu - SSL memory error

Description

FusionAuth lockups and needs to be restarted due to SSL handshake error. Everything work fine when the server is first started and requests are successfully handled by FusionAuth only for it to lockup randomly - usually after a few hours.

---

2024-11-27 05:27:09.189 AM ERROR io.fusionauth.http.server.HTTPServerThread - An exception was thrown during processing
javax.net.ssl.SSLException: Unrecognized record version (D)TLS-0.0 , plaintext connection?
at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:97)
at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:64)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:612)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
at io.fusionauth.http.server.HTTPS11Processor.decrypt(HTTPS11Processor.java:393)
at io.fusionauth.http.server.HTTPS11Processor.read(HTTPS11Processor.java:199)
at io.fusionauth.http.server.HTTPServerThread.read(HTTPServerThread.java:338)
at io.fusionauth.http.server.HTTPServerThread.run(HTTPServerThread.java:148)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 7"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 4"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP Server Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP Server Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 3"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 5"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-2-thread-1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 2"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 1"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 8"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "I/O dispatcher 6"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP Server Thread"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HttpClient-1-SelectorManager"

---

Observed versions

1.53.3

Affects versions

Steps to reproduce

Steps to reproduce the behavior:

1. Install FusionAuth on Ubuntu with PostgresSQL database
2. Add Lets Encrypt SSL certificate with Certbot
3. FusionAuth randomly locks up after a couple hours

Expected behavior

FusionAuth should not lockup- incorrect error handling?

Screenshots

Platform

(Please complete the following information)

• Device: N/A
• OS: Ubuntu
• Browser N/A
• Database PostgresSQL (Ubuntu 16.4-0ubuntu0.24.04.2)

Community guidelines

All issues filed in this repository must abide by the FusionAuth community guidelines.

Additional context

Ubuntu Server is sitting on a public IP address and all https requests are forwarded to this server on port 9013.
I've setup the Let's Encrypt certificate with certbot (certonly) - I believe the issue is I haven't integrated this certificate with FusionAuth correctly even though everything seems to work correctly in the beginning.
The installation is a fresh install of Ubuntu with FusionAuth and Postgres with no other applications.
The SSL plaintext error can occur when no clients are attempting to log into FusionAuth.

[robotdan]

The first exception javax.net.ssl.SSLException: Unrecognized record version (D)TLS-0.0 , plaintext connection? - looks like the JDK is unable to handle the TLS connection.

Is it possible you are trying to make an http connection on this TLS port?

The second exception java.lang.OutOfMemoryError - generally when this exception is thrown it means the system does not have enough capacity. Can you provide your configuration to show how much RAM you have assigned to FusionAuth, and the amount of memory available on the host?

Is there something specific that would indicate this is a bug in FusionAuth and not just that the VM doesn't have enough RAM to process the request?

Or is the symptom resolved by increasing the amount of memory assigned to the VM?

[DaveInAus]

Hi @robotdan

The fusionauth.properties has the following for HTTPS/TLS

fusionauth-app.https.enabled=true 
fusionauth-app.https.port=9013 
fusionauth-app.https.certificate-file=fullchain.pem file generated from certbot in manual mode 
fusionauth-app.https.private-key-file=privacy.pem file generated from certbot in manual mode 

running certbot in manual like this
certbot certonly –manual -d *.exampledomain.com -d exampledomain.com –agree-tos –manual-public-ip-logging-ok –preferred-challenges dns-01 –server https://acme-v02.api.letsencrypt.org/directory –register-unsafely-without-email –rsa-key-size 4096

Memory
fusionauth-app.memory=512M

Current Setup:
Ubuntu on Development PC. Router only routes traffic on port 9013 from a public facing IP to this development PC.
to connect https://auth.exampledomain.com:9013

As stated all seems to work fine at first only then for plaintext errors to start followed by memory errors and then a lockup.

I have decided to go with an IIS reverse proxy to route traffic to this Ubuntu Developement PC and everything works 100% now. It's a cleaner solution however still curious to understand why the routing on port 9013 with a manual cert is creating an issue.

Thanks