Got Stuck on STREAM: Recovering the state...

[ErrorCode996]

Startup is stuck on

[1] 2024/01/22 08:31:22.385297 [INF] STREAM: Starting nats-streaming-server[appetite-micro] version 0.25.6
[1] 2024/01/22 08:31:22.385345 [INF] STREAM: ServerID: EodFr7Xob7MS7e62c1N7IF
[1] 2024/01/22 08:31:22.385346 [INF] STREAM: Go version: go1.20.11
[1] 2024/01/22 08:31:22.385347 [INF] STREAM: Git commit: [d1a98ca]
[1] 2024/01/22 08:31:26.786728 [INF] Starting nats-server
[1] 2024/01/22 08:31:26.786774 [INF]   Version:  2.9.24
[1] 2024/01/22 08:31:26.786779 [INF]   Git:      [e43cfb4]
[1] 2024/01/22 08:31:26.786792 [INF]   Name:     ND2U7NKWY5Q5OQ36GEVGLPMFHOCYEEJSBBPVLFRFJHRIIT4N6IYZWHH4
[1] 2024/01/22 08:31:26.786795 [INF]   ID:       ND2U7NKWY5Q5OQ36GEVGLPMFHOCYEEJSBBPVLFRFJHRIIT4N6IYZWHH4
[1] 2024/01/22 08:31:26.793177 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2024/01/22 08:31:26.793387 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2024/01/22 08:31:26.793950 [INF] Server is ready
[1] 2024/01/22 08:31:26.816971 [INF] STREAM: Recovering the state...

deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nats-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nats
  template:
    metadata:
      labels:
        app: nats
    spec:
      containers:
        - name: nats
          image: nats-streaming:0.25.6
          args: [
              "-p",
              "4222",
              "-m",
              "8222",
              "-hbi",
              "5s",
              "-hbt",
              "5s",
              "-hbf",
              "2",
              "-SD",
              "-cid",
              "micro",
              "--sql_driver",
              "postgres",
              "--sql_source",
              "postgresql://<<username>>:<<password>>@<<host>>:<<port>>/<<dbname>>?sslmode=require",
              "--sql_max_open_conns",
              "1",
              "--store",
              "sql",
            ]

Connects few times successfully
but if restart the projects multiple times then it gets stock on STREAM: Recovering the state

DB pool size is 4

must be related to db connection

[kozlovic]

First, I have to remind you that this project is deprecated, so there won't be much that we can do to help.

You are mentioning that the DB pool size is 4, but as you can see in the parameters you are limiting it to 1 (--sql_max_open_conns 1). I am not sure if using more would help or not.

The important thing would be to know if the server is just taking a long time to recover the state, but is not stuck, or if it is really stuck, say due to a deadlock? Since you run with profiling -m 8222, you could capture stack traces by pointing your browser (or curl) to that port with the stacksz endpoint: http://<host>:8222/stacksz and do 2 or 3 captures are 1 minute interval.