Peer accessing db during shutdown causes crash

[gfodor]

I've noticed the following crash when shutting down:

github.com/dgraph-io/badger/v3.(*memTable).IncrRef(0x0)
        /home/gfodor/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/memtable.go:231 +0x19
github.com/dgraph-io/badger/v3.(*DB).getMemTables(0xc0a189a480)
        /home/gfodor/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:696 +0x1cc
github.com/dgraph-io/badger/v3.(*DB).get(0xc0a189a480, {0xc2bcc40120, 0x2d, 0x2d})
        /home/gfodor/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/db.go:730 +0x165
github.com/dgraph-io/badger/v3.(*Txn).Get(0xc2bcc4e000, {0xc2bcc400f0, 0x25, 0x30})
        /home/gfodor/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/txn.go:478 +0x4da
github.com/bitclout/core/lib.DbGetUtxoEntryForUtxoKeyWithTxn(0xc2bcc4e000, 0xc2bcc400c0)
        /home/gfodor/1729/bitclout/core/lib/db_utils.go:1991 +0xda
github.com/bitclout/core/lib.DbGetUtxoEntryForUtxoKey.func1(0xc2bcc4e000)
        /home/gfodor/1729/bitclout/core/lib/db_utils.go:2016 +0x45
github.com/dgraph-io/badger/v3.(*DB).View(0xc0a189a480, 0xc00b8b0b00)
        /home/gfodor/go/pkg/mod/github.com/dgraph-io/badger/[email protected]/txn.go:806 +0x162
github.com/bitclout/core/lib.DbGetUtxoEntryForUtxoKey(0xc0a189a480, 0xc2bcc400c0)
        /home/gfodor/1729/bitclout/core/lib/db_utils.go:2015 +0x7d
github.com/bitclout/core/lib.(*UtxoView).GetUtxoEntryForUtxoKey(0xc00fa40340, 0xc2bcc400c0)
        /home/gfodor/1729/bitclout/core/lib/block_view.go:1270 +0xd0
github.com/bitclout/core/lib.(*BitCloutMempool).tryAcceptTransaction(0xc00d8acf20, 0xc1f5a4e5b0, 0x0, 0x1, 0x1)
        /home/gfodor/1729/bitclout/core/lib/mempool.go:1229 +0x628
github.com/bitclout/core/lib.(*BitCloutMempool).processTransaction(0xc00d8acf20, 0xc1f5a4e5b0, 0x1, 0x0, 0x1, 0x1)
        /home/gfodor/1729/bitclout/core/lib/mempool.go:1942 +0x165
github.com/bitclout/core/lib.(*BitCloutMempool).ProcessTransaction(0xc00d8acf20, 0xc1f5a4e5b0, 0x1, 0x0, 0x1, 0x1)
        /home/gfodor/1729/bitclout/core/lib/mempool.go:1992 +0x156
github.com/bitclout/core/lib.(*Server).ProcessSingleTxnWithChainLock(0xc0023fd140, 0xc00adc4000, 0xc1f5a4e5b0)
        /home/gfodor/1729/bitclout/core/lib/server.go:1338 +0x150
github.com/bitclout/core/lib.(*Server)._processTransactions(0xc0023fd140, 0xc00adc4000, 0xc0c7c0cd98)
        /home/gfodor/1729/bitclout/core/lib/server.go:1358 +0x1e6
github.com/bitclout/core/lib.(*Peer).HandleTransactionBundleMessage(0xc00adc4000, 0xc0c7c0cd98)
        /home/gfodor/1729/bitclout/core/lib/peer.go:343 +0x4f2
github.com/bitclout/core/lib.(*Peer).StartBitCloutMessageProcessor(0xc00adc4000)
        /home/gfodor/1729/bitclout/core/lib/peer.go:556 +0x57a
created by github.com/bitclout/core/lib.(*Peer).Start
        /home/gfodor/1729/bitclout/core/lib/peer.go:1141 +0x213

I believe this is happening because the peer message processor is not being shut down, and if a transaction comes in during the end of shutdown after the database is closed this crash occurs. However after a brief investigation I'm not sure if/where this message processor is supposed to be gracefully terminated - the Stop() function in the connection manager just closes the listeners.

[maebeam]

Thanks for the report. We need to add a Stop() method to Peer that send a message to the quit channel. When ConnectionManager is stopped it should call Stop() for every GetAllPeers()