Jetstream appears to not accept new messages when only one of two limits is breached [v2.10.23]

[rosscooperman]

Observed behavior

We have the following settings on a stream in Synadia Cloud:

Information for Stream match_messages created 2024-11-05 12:40:02

              Subjects: match.>
              Replicas: 3
               Storage: File

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
            Direct Get: true
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: true

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: 1.5 GiB
           Maximum Age: 2d0h0m0s
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: ngsprod-aws-useast2
                Leader: aws-useast2-natscj1-2
               Replica: aws-useast2-natscj1-1, current, seen 672ms ago
               Replica: aws-useast2-natscj1-3, current, seen 673ms ago

State:

              Messages: 72,982
                 Bytes: 1.5 GiB
        First Sequence: 4,230,426 @ 2024-12-01 22:13:58 UTC
         Last Sequence: 4,303,407 @ 2024-12-02 10:47:28 UTC
      Active Consumers: 1
    Number of Subjects: 5

With this configuration, even with consistent delivery of new messages to NATS, there are frequent "pauses" during which no new messages arrive in Jetstream. Messages delivered during these "pauses" never end up in the stream.

Expected behavior

Our expectation is that when total size exceeds 1.5GiB that old messages are discarded to make way for new even if those old messages are less than 2 days old.

Server and client version

We are using Synadia Cloud so can't get server version. Client is the TS/JS client, version 2.28.2 (but also the cloud dashboard watching activity in live mode).

Host environment

No response

Steps to reproduce

No response

[neilalexander]

Can you please confirm whether you're using JetStream publishes or core NATS publishes?

[rosscooperman]

core NATS publishing

[neilalexander]

You should probably be using JetStream publishes BTW. Core NATS publishes into a stream can fail silently.

A JetStream publish uses request-reply so you receive a PubAck confirmation from the server when JetStream processed the message. If you don't get a PubAck back, you can retry.

[rosscooperman]

You should probably be using JetStream publishes BTW. Core NATS publishes into a stream can fail silently.

A JetStream publish uses request-reply so you receive a PubAck confirmation from the server when JetStream processed the message. If you don't get a PubAck back, you can retry.

Ah, yeah, good point!