JetStream: multiple messages per subject in a stream with "Maximum Per Subject: 1" [v2.10.22]

[ygabuev]

Observed behavior

We are observing multiple messages per subject in a (in-memory) stream that we set up to hold only 1 message per subject. It is not clear if that is an issue with our setup, or if it could be a bug in NATS/Jetstream. We would appreciate some help with debugging this and could provide more information if needed.

root@host # nats stream subjects our-stream | tail
│ our.stream.abc-1117                                                       │ 144   │
│ our.stream.abc-1112                                                       │ 146   │
│ our.stream.abc-1114                                                       │ 147   │
│ our.stream.abc-7777                                                       │ 154   │
│ our.stream.abc-1111                                                       │ 170   │
│ our.stream.abc-3333                                                       │ 172   │
│ our.stream.abc-1119                                                       │ 176   │
root@host # nats stream info our-stream
Information for Stream module-status created 2024-12-02 13:36:45

              Subjects: our.stream.>
              Replicas: 3
               Storage: Memory

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: 1
         Maximum Bytes: unlimited
           Maximum Age: unlimited
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: nats
                Leader: nats-1
               Replica: nats-0, current, seen 120ms ago
               Replica: nats-2, current, seen 120ms ago

State:

              Messages: 34,771
                 Bytes: 9.7 MiB
        First Sequence: 18,519 @ 2024-12-02 16:25:19 UTC
         Last Sequence: 22,024,595 @ 2024-12-10 11:16:21 UTC
      Deleted Messages: 21,971,306
      Active Consumers: 453
    Number of Subjects: 5,439

Expected behavior

We expect to have at most 1 message per subject in this stream.

Server and client version

Server: 2.10.22
Client: 0.1.4
NACK: 0.16.0

Host environment

NATS cluster in K3S with 3 replicas. The container image is 2.10.22-apline, in-memory Jetstream. The stream is created using NACK (0.16.0).

Steps to reproduce

We only observed this bug once and were not able to reproduce it.