[Core NATS] Upgrading from v2.0.4 to v2.9.7

[chichunhua]

I upgraded the version from 2.0.4 to 2.9.7 and then it became not working well. It suppose to have MSG_PAYLOAD every 30s but not, the intervals vary a lot, sometimes 10 min sometimes 1min. NATs server is running, and all pub/sub client can connect to it, based on the monitoring endpoint, under connections, I can see in_msgs amount, but not out_msgs amount. also in NATs logs, I can see the msg_payload, not in regular interval though. Did I missing any config? or something else related? Looking forwards your kind reply. Thank you!

The config:

# Client port of 4222 on all interfaces
port: xxxx

# HTTP monitoring port
#monitor_port: xxxx

# This is for clustering multiple servers together.
cluster {
  # Route connections to be received on any interface on port 6222
  port: xxxx
  routes = []
 
 # number of retries to connect to a discovered route
  connect_retries: 25
}

# logging configurations
debug: true
trace: true
logtime: true

Subscriptions:

{
num_subscriptions: 312,
num_cache: 10,
num_inserts: 483,
num_removes: 171, // what does this mean?
num_matches: 113210,
cache_hit_rate: 0.00009716456143450225,
max_fanout: 4, // is this mean only 4 msgs were consumed?
avg_fanout: 3.2
}

In logs: I can see bunch of the trace logs below, and seems like my actual message is not there in regular interval.

[RMSG $SYS $SYS.SERVER.ACCOUNT.$G.CONNS 550]
[RMSG $SYS $SYS.SERVER.NDZGNWXWKBRFPFBHYI2MSNMPQEU535UMDHFQTIQCJHBGQASHGQWBOC5M.STATSZ 1060]
MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"IBNKsj1motiyiaTm6pTtRB\",\"timestamp\":\"2023-01-25T20:21:06.788445383Z\",\"server\":{\"name\":\"NDZGNWXWKBRFPFBHYI2MSNMPQEU535UMDHFQTIQCJHBGQASHGQWBOC5M\",\"host\":\"0.0.0.0\",\"id\":\"NDZGNWXWKBRFPFBHYI2MSNMPQEU535UMDHFQTIQCJHBGQASHGQWBOC5M\",\"cluster\":\"vJXipA5NGT31HzqFlLB7QY\",\"ver\":\"2.9.7\",\"seq\":1639,\"jetstream\":false,\"time\":\"2023-01-25T20:21:06.788460073Z\"},\"acc\":\"$G\",\"conns\":3,\"leafnodes\":0,\"total_conns\":3,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":5280,\"bytes\":1275711},\"slow_consumers\":0}"] | [15] 2023/01/25 20:21:06.788801 [TRC] 10.226.28.212:6222 - rid:6 - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"IBNKsj1motiyiaTm6pTtRB\",\"timestamp\":\"2023-01-25T20:21:06.788445383Z\",\"server\":{\"name\":\"NDZGNWXWKBRFPFBHYI2MSNMPQEU535UMDHFQTIQCJHBGQASHGQWBOC5M\",\"host\":\"0.0.0.0\",\"id\":\"NDZGNWXWKBRFPFBHYI2MSNMPQEU535UMDHFQTIQCJHBGQASHGQWBOC5M\",\"cluster\":\"vJXipA5NGT31HzqFlLB7QY\",\"ver\":\"2.9.7\",\"seq\":1639,\"jetstream\":false,\"time\":\"2023-01-25T20:21:06.788460073Z\"},\"acc\":\"$G\",\"conns\":3,\"leafnodes\":0,\"total_conns\":3,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":5280,\"bytes\":1275711},\"slow_consumers\":0}"]
MSG_PAYLOAD: ["{\"server\":{\"name\":\"NDV7U5LHSWC364O2X7YH7YC7YEVUNTMDJATZVAQXPVPN366VAUARR2QD\",\"host\":\"0.0.0.0\",\"id\":\"NDV7U5LHSWC364O2X7YH7YC7YEVUNTMDJATZVAQXPVPN366VAUARR2QD\",\"cluster\":\"vJXipA5NGT31HzqFlLB7QY\",\"ver\":\"2.9.7\",\"seq\":1642,\"jetstream\":false,\"time\":\"2023-01-25T20:21:05.972367605Z\"},\"statsz\":{\"start\":\"2023-01-25T15:53:03.893871113Z\",\"mem\":16752640,\"cores\":8,\"cpu\":0,\"connections\":5,\"total_connections\":27,\"active_accounts\":2,\"subscriptions\":307,\"sent\":{\"msgs\":3204,\"bytes\":2556471},\"received\":{\"msgs\":148519,\"bytes\":19015407},\"slow_consumers\":0,\"routes\":[{\"rid\":10,\"name\":\"NCGQGRWM54SJJK2SBF6GX7UAQJ2KIX3R2M2NRXNINNUGPMCZ323D744X\",\"sent\":{\"msgs\":1067,\"bytes\":851467},\"received\":{\"msgs\":936,\"bytes\":776151},\"pending\":0},{\"rid\":9,\"name\":\"NDZGNWXWKBRFPFBHYI2MSNMPQEU535UMDHFQTIQCJHBGQASHGQWBOC5M\",\"sent\":{\"msgs\":1067,\"bytes\":851467},\"received\":{\"msgs\":1067,\"bytes\":849792},\"pending\":0},{\"rid\":8,\"name\":\"NBRYMRJ6EC4VLQYHCLPCC2QTNN34I2ZTP3XR5WGSUY3RTUYL7ENP6GLY\",\"sent\":{\"msgs\":1070,\"bytes\":853537},\"received\":{\"msgs\":895,\"bytes\":753896},\"pending\":0}],\"active_servers\":4}}"] | [15] 2023/01/25 20:21:05.972672 [TRC] 10.226.28.42:6222 - rid:7 - <<- MSG_PAYLOAD: ["{\"server\":{\"name\":\"NDV7U5LHSWC364O2X7YH7YC7YEVUNTMDJATZVAQXPVPN366VAUARR2QD\",\"host\":\"0.0.0.0\",\"id\":\"NDV7U5LHSWC364O2X7YH7YC7YEVUNTMDJATZVAQXPVPN366VAUARR2QD\",\"cluster\":\"vJXipA5NGT31HzqFlLB7QY\",\"ver\":\"2.9.7\",\"seq\":1642,\"jetstream\":false,\"time\":\"2023-01-25T20:21:05.972367605Z\"},\"statsz\":{\"start\":\"2023-01-25T15:53:03.893871113Z\",\"mem\":16752640,\"cores\":8,\"cpu\":0,\"connections\":5,\"total_connections\":27,\"active_accounts\":2,\"subscriptions\":307,\"sent\":{\"msgs\":3204,\"bytes\":2556471},\"received\":{\"msgs\":148519,\"bytes\":19015407},\"slow_consumers\":0,\"routes\":[{\"rid\":10,\"name\":\"NCGQGRWM54SJJK2SBF6GX7UAQJ2KIX3R2M2NRXNINNUGPMCZ323D744X\",\"sent\":{\"msgs\":1067,\"bytes\":851467},\"received\":{\"msgs\":936,\"bytes\":776151},\"pending\":0},{\"rid\":9,\"name\":\"NDZGNWXWKBRFPFBHYI2MSNMPQEU535UMDHFQTIQCJHBGQASHGQWBOC5M\",\"sent\":{\"msgs\":1067,\"bytes\":851467},\"received\":{\"msgs\":1067,\"bytes\":849792},\"pending\":0},{\"rid\":8,\"name\":\"NBRYMRJ6EC4VLQYHCLPCC2QTNN34I2ZTP3XR5WGSUY3RTUYL7ENP6GLY\",\"sent\":{\"msgs\":1070,\"bytes\":853537},\"received\":{\"msgs\":895,\"bytes\":753896},\"pending\":0}],\"active_servers\":4}}"]

[wallyqs]

Try setting a cluster name and then retry, otherwise you would have some reconnections happening while the cluster is being assembled

# This is for clustering multiple servers together.
cluster {
  name: "example"
  ...
}

[derekcollison]

Also please run latest 2.9 release which is 2.9.11.

[chichunhua]

@wallyqs I added the cluster name, but seems like does not working. I deployed one seed service and 3 worker service in AWS ECS. By the way, the client is using nats.go v1.11 ang go-nats v1.7. will it not compatible with the new version of NATs server? The NATs server can receive msg_payload with v1.11, but can not send it out to subs with v1.7. However, when I update the version for the subs, still not working.

@derekcollison I am not using JetStream, is there unknown improvement for Core NATs in v2.9.11? It's still the same with the latest version though.

[derekcollison]

Regardless of JetStream we recommend all users to be current with the latest version and patched release. When we release a new minor version, an upgrade, customers will be supported on the previous version for some time and it will receive bug fixes and security updates.

[wallyqs]

@chichunhua it should all be compatible... go-nats is an archived read only copy of the nats.go client, make sure that you have these versions:

NATS Client: nats.go v1.23.0
NATS Server: nats-server v2.9.11

The import paths in your code should be using the nats.go version so github.com/nats-io/nats.go.

[chichunhua]

Tested it, found that it's not only the version issue, but also relates to the subject. is there any change regarding the subject? It seems like does not matching and working after change to the new version. Also is there a command line to list the subjects?

for example

aaa.bb.cc.dd.eee.fff.gg.hhhh
aaa.*.*.*.*.*.*.*  working
aaa.*.cc.*.*.*.*.*  working 
aaa.*.cc.dd.fff.*.*.*  does not working 
aaa.*.cc.dd.*.*.gg.* working 

[jnmoyne]

You can always use nats to subscribe to messages using a wildcard and to publish messages (and therefore check that they match the subject).

For example in one window do nats sub aaa.*.*.*.*.*.*.* and while it's running in another window do nats pub aaa.bb.cc.dd.eee.fff.gg.hhhh payload

You should see the published message appear in the nats sub window.

If you subscribe to aaa..cc.dd.fff... then you do not receive the message because the fff is out of place

[chichunhua]

Thank you so much @jnmoyne! just want to double confirm one more format. It was working in v2.0.4 with subject format below, but not working for v2.9.7 and v2.9.11. seems like you had some change on it. do you remember when and in which version made that change?

pub: aaa.bb.cc.dd.eee..gg.hhhh
sub aaa.bb.cc.dd.eee.fff.gg.hhhh

[ripienaar]

Maybe if you did sub aaa.bb.cc.dd.eee.*.gg.hhhh, but as you have it, it did not work. (easy to confirm using docker container nats:2.0.4).

Now on 2.9.x that would fail because aa.bb.cc.dd.eee..gg.hhhh is not a valid subject, there was a bug that would allow this, bugs been fixed.

[chichunhua]

Do you remember in which version was that bug fixed?

[ripienaar]

Your example pub aaa.bb.cc.dd.eee..gg.hhhh with sub aaa.bb.cc.dd.eee.fff.gg.hhhh never worked in that release.

sub aaa.bb.cc.dd.eee.*.gg.hhhh and publishing to aaa.bb.cc.dd.eee..gg.hhhh was fixed in 2.2.0

[chichunhua]

@ripienaar thanks you very much, it helps a lot!