delist dead client connections

[glycerine]

When clients disappear, or go inactive, it would be nice to be able to clear them from the nats-top display, either automatically (say after an hour of no contact?) or manually (say a keypress to say "show me only active clients").

Currently old clients continue to be listed even though their process has died.

Maybe this is actually a gnatsd issue instead?

[glycerine]

As an example, check out the screen scrape below. Can you tell at a glance which of these 2 processes are actually dead and need to be restarted? It's not obvious. The last activity tells you that 2 of them are 5 days old.

NATS server version 0.7.5 (uptime: 14d23h2m4s)
Server:
  Load: CPU:  1.6%  Memory: 11.0M  Slow Consumers: 0
  In:   Msgs: 158.5M  Bytes: 82.5G  Msgs/Sec: 8.0  Bytes/Sec: 8.2K
  Out:  Msgs: 208.8M  Bytes: 108.6G  Msgs/Sec: 31.8  Bytes/Sec: 32.7K

Connections: 8
  HOST                 CID      NAME            SUBS    PENDING     MSGS_TO     MSGS_FROM   BYTES_TO    BYTES_FROM  LANG     VERSION  UPTIME   LAST ACTIVITY
  10.88.88.99:29813    270      archiver        1       0           9.5M        0           5.1G        0           go       1.1.9    6d3h49m46s  2016-03-09 22:53:58.218747962 +000
  10.88.88.99:19926    271      archiver        1       0           9.5M        0           5.1G        0           go       1.1.9    6d3h21m13s  2016-03-09 22:53:58.218745747 +000
  10.88.88.99:11324    273      profiler        1       0           8.8M        0           4.7G        0           go       1.1.9    5d21h26m31s  2016-03-09 22:53:58.218750434 +00
  10.88.88.99:11669    277      replayer        1       0           1           1           63          2           go       1.1.9    5d18h38m57s  2016-03-04 04:15:09.345560427 +00
  10.88.88.99:11670    278      backfiller      1       0           0           0           0           0           go       1.1.9    5d18h38m57s  2016-03-04 04:15:01.810602423 +00
  10.123.123.123:12087 280      ps-job          0       0           0           1.8M        0           987.3M      go       1.1.9    2d3h46m18s  2016-03-09 22:53:57.862214631 +000
  10.123.123.45:15813  281      ps-job          0       0           0           1.8M        0           807.9M      go       1.1.9    2d3h44m45s  2016-03-09 22:53:58.218730614 +000
  10.45.156.132:8527   282      ps-job          0       0           0           112.6K      0           223.9M      go       1.1.9    2d3h43m12s  2016-03-09 22:53:57.456426391 +000

[derekcollison]

So the processes are confirmed dead correct? The server only reports active (from its standpoint) connections.

Do you have a test that can reproduce this?

[glycerine]

Ok, wierd. the process named replayer is dead, but I see another process holding that port open using lsof. My bad, there is a live process, and I must have mixed up nats name with the program name. Closing tickets, but noting that Adding a PID to the nats-top display would bypass some confusion; that is #30

[derekcollison]

ok no problem. We would need to add a PID field to the INFO proto message from the clients. Note that if you want, you can make "name" anything you want, so you could pick argv[0]-PID if you wanted.