Consensus failures/Panics - pebbledb chunks >= 4.0Gb error messages

[effofxprime]

Since the last couple of upgrades, I started receiving these two errors in the two different flavors. I am not sure why I will see a consensus failure sometimes and others it is just a panic.

I have found a resolution to me getting these errors. I normally run with pruning set to everything and then I maintain a min block count of unbonding time. When pruning is set to everything, I do not get more than 10 blocks in before the pebbledb issue presents itself in one way or another. With custom pruning set to an interval of 1000, I also still get the problem. But I have been able to avoid the problem with my pruning interval set to 3000.

It appears that smaller intervals for pruning are not being chunked properly for pebbledb, resulting in chunks over 4.0Gb for me.

Panic error:

6:44AM ERR Failed to update lp rewards unclaimed error="0ueden: invalid coins" module=server
6:44AM INF executed block height=8652690 module=state num_invalid_txs=0 num_valid_txs=4
6:44AM ERR Stopping peer for error err="error with peer a97a52a34101fcd7d3186fa5cbcff32b12e6332e: peer did not send us anything" module=p2p peer="Peer{MConn{15.235.204.150:28856} a97a52a34101fcd7d3186fa5cbcff32b12e6332e out}"
6:44AM ERR SendTimeout module=blockchain peer=a97a52a34101fcd7d3186fa5cbcff32b12e6332e reason="peer did not send us anything" timeout=15000
6:44AM ERR Stopping peer for error err="read tcp 10.1.1.69:2694->15.235.204.150:28856: use of closed network connection" module=p2p peer="Peer{MConn{15.235.204.150:28856} a97a52a34101fcd7d3186fa5cbcff32b12e6332e out}"
6:44AM ERR error while stopping peer error="already stopped" module=p2p
6:44AM ERR Stopping peer for error err=EOF module=p2p peer="Peer{MConn{173.249.24.244:20273} 258f523c96efde50d5fe0a9faeea8a3e83be22ca out}"
panic: pebble: batch too large: >= 4.0GB
goroutine 280 [running]:
github.com/cockroachdb/pebble.(*Batch).grow(0xc26800dfc0?, 0xc26800dff8?)
        /validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:1414 +0x12d
github.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyRecord(0xc003268f00, 0x31, 0x0)
        /validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:644 +0x85
github.com/cockroachdb/pebble.(*Batch).DeleteDeferred(...)
        /validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:781
github.com/cockroachdb/pebble.(*Batch).Delete(0xc003268f00, {0xc2658616c0, 0x31, 0x0?}, 0xc2658616c0?)
        /validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:764 +0x2c
github.com/cometbft/cometbft-db.(*pebbleDBBatch).Delete(0xc26585d310?, {0xc2658616c0?, 0xc26585b400?, 0xc26800e048?})
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/pebble.go:277 +0x2c
github.com/cometbft/cometbft-db.prefixDBBatch.Delete({{0xc0027d8470, 0x10, 0x10}, {0x3f921a0, 0xc2418ec7a0}}, {0xc26585b440, 0x21, 0xf39665?})
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/prefixdb_batch.go:35 +0x11c
github.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange.func1({0xc2658715f0, 0x31, 0x31}, {0xc26583bfc0, 0x20, 0x20})
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:545 +0x198
github.com/cosmos/iavl.(*nodeDB).traversePrefix(0xc0007cfd40?, {0xc26258f220?, 0x1?, 0x292a100?}, 0xc26800e320)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:847 +0x154
github.com/cosmos/iavl.(*nodeDB).traverseOrphansVersion(0xc0027d7400, 0x83b86a?, 0xc26800e320)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:808 +0x68
github.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange(0xc0027d7400, 0x83b86a, 0x83ce9f)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:538 +0x425
github.com/cosmos/iavl.(*MutableTree).DeleteVersionsRange(0xc001f04e60, 0x83b86a, 0x83ce9f)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1089 +0x3f
github.com/cosmos/iavl.(*MutableTree).DeleteVersions(0xc001f04e60, {0xc241a66000, 0x4319, 0x4400})
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1077 +0x2aa
github.com/cosmos/cosmos-sdk/store/iavl.(*Store).DeleteVersions(...)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/iavl/store.go:235
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc0012c6e00, 0x98?, {0x0?, 0x0?, 0x0?})
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:641 +0x3d1
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).handlePruning(0xc0012c6e00, 0x840792)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:603 +0x1e7
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0012c6e00)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:462 +0x405
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0003b4b40)
        /validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/baseapp/abci.go:448 +0x1e5
github.com/cometbft/cometbft/abci/client.(*localClient).CommitSync(0xc003f8eae0)
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:267 +0xab
github.com/cometbft/cometbft/proxy.(*appConnConsensus).CommitSync(0xc00187c4b0)
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:116 +0x13c
github.com/cometbft/cometbft/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc002b862f8, 0x6}}, {0xc002b86570, 0xd}, 0x1, 0x840792, {{0xc22b9da7c0, ...}, ...}, ...}, ...)
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:289 +0x252
github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc002b862f8, 0x6}}, {0xc002b86570, 0xd}, 0x1, 0x840792, {{0xc22b9da7c0, ...}, ...}, ...}, ...)
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:241 +0x70e
github.com/cometbft/cometbft/blocksync.(*Reactor).poolRoutine(0xc00118ec40, 0x0)
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:409 +0xb33
created by github.com/cometbft/cometbft/blocksync.(*Reactor).OnStart in goroutine 84
        /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/blocksync/reactor.go:128 +0x6e
elys.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
elys.service: Failed with result 'exit-code'.
elys.service: Consumed 1min 49.783s CPU time.

Consensus failure:

Jul 06 06:47:50 artorias elysd[489609]: 6:47AM ERR CONSENSUS FAILURE!!! err="pebble: batch too large: >= 4.0G
      B" module=consensus stack="goroutine 1573 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debu
      g/stack.go:24 +0x5e\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/validator-hom
      e-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:737 +0x46\npanic({0x2b73a20?, 0xc
      0006a16c0?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.com/cockroachdb/pebble.(*Batch).grow(0x
      d4730a7800?, 0xd4730a7838?)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.g
      o:1414 +0x12d\ngithub.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyRecord(0xc00210f980, 0x31, 0x0)\n\t/v
      alidator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:644 +0x85\ngithub.com/cockroach
      db/pebble.(*Batch).DeleteDeferred(...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/pebble@v1.
      1.0/batch.go:781\ngithub.com/cockroachdb/pebble.(*Batch).Delete(0xc00210f980, {0xd7c49b7c80, 0x31, 0x0?}, 0xd
      7c49b7c80?)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cockroachdb/[email protected]/batch.go:764 +0x2c\ngit
      hub.com/cometbft/cometbft-db.(*pebbleDBBatch).Delete(0xd7c49abda0?, {0xd7c49b7c80?, 0x0?, 0xd4730a7888?})\n\t
      /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/pebble.go:277 +0x2c\ngithub.com/com
      etbft/cometbft-db.prefixDBBatch.Delete({{0xc00661f030, 0x10, 0x10}, {0x3f76260, 0xd4464f4da0}}, {0xd7c49bed80
      , 0x21, 0xf39665?})\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/prefixdb_bat
      ch.go:35 +0x11c\ngithub.com/cosmos/iavl.(*nodeDB).DeleteVersionsRange.func1({0xd7c49cc920, 0x31, 0x31}, {0xd7
      c49a94c0, 0x20, 0x20})\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:545 +0x1
      98\ngithub.com/cosmos/iavl.(*nodeDB).traversePrefix(0xc000a32c00?, {0xd7c22bdbb0?, 0x1?, 0xd4730a7aa0?}, 0xd4
      730a7b60)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:847 +0x154\ngithub.co
      m/cosmos/iavl.(*nodeDB).traverseOrphansVersion(0xc003aba400, 0x81be18?, 0xd4730a7b60)\n\t/validator-home-4/el
      ys/go/pkg/mod/github.com/cosmos/[email protected]/nodedb.go:808 +0x68\ngithub.com/cosmos/iavl.(*nodeDB).DeleteVers
      ionsRange(0xc003aba400, 0x81be18, 0x81c6fa)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]
      .1/nodedb.go:538 +0x425\ngithub.com/cosmos/iavl.(*MutableTree).DeleteVersionsRange(0xc0008a8000, 0x81be18, 0x
      81c6fa)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1089 +0x3f\ngithu
      b.com/cosmos/iavl.(*MutableTree).DeleteVersions(0xc0008a8000, {0xd449ec3000, 0x8e2, 0x900})\n\t/validator-hom
      e-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/mutable_tree.go:1077 +0x2aa\ngithub.com/cosmos/cosmos-sdk/
      store/iavl.(*Store).DeleteVersions(...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.
      47.11/store/iavl/store.go:235\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).PruneStores(0xc0027e1800
      , 0xd8?, {0x0?, 0x0?, 0x0?})\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store
      /rootmulti/store.go:641 +0x3d1\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).handlePruning(0xc0027e1
      800, 0x81d30b)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/sto
      re.go:603 +0x1e7\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0027e1800)\n\t/validator-ho
      me-4/elys/go/pkg/mod/github.com/cosmos/[email protected]/store/rootmulti/store.go:462 +0x405\ngithub.com/co
      smos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc0023990e0)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cosmo
      s/[email protected]/baseapp/abci.go:448 +0x1e5\ngithub.com/cometbft/cometbft/abci/client.(*localClient).Com
      mitSync(0xc002409ec0)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/l
      ocal_client.go:267 +0xab\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).CommitSync(0xc003586cd8)\n\t
      /validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:116 +0x13c\ngithub.c
      om/cometbft/cometbft/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc00f48865a, 0x6}}, {0xc00f488660, 0xd}
      , 0x1, 0x81d30b, {{0xd3e8772300, ...}, ...}, ...}, ...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/comet
      bft/[email protected]/state/execution.go:289 +0x252\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).Apply
      Block(_, {{{0xb, 0x0}, {0xc00f48865a, 0x6}}, {0xc00f488660, 0xd}, 0x1, 0x81d30b, {{0xd3e8772300, ...}, ...},
      ...}, ...)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:241 +
      0x70e\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc005898e08, 0x81d30b)\n\t/validator-h
      ome-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1711 +0xa4c\ngithub.com/cometbf
      t/cometbft/consensus.(*State).tryFinalizeCommit(0xc005898e08, 0x81d30b)\n\t/validator-home-4/elys/go/pkg/mod/
      github.com/cometbft/[email protected]/consensus/state.go:1620 +0x2e8\ngithub.com/cometbft/cometbft/consensus.(
      *State).handleCompleteProposal(0xc005898e08, 0x81d30b)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometb
      ft/[email protected]/consensus/state.go:2016 +0x372\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg
      (0xc005898e08, {{0x3f438c0, 0xd3e804dd10}, {0xcea0fce1b0, 0x28}})\n\t/validator-home-4/elys/go/pkg/mod/github
      .com/cometbft/[email protected]/consensus/state.go:847 +0x178\ngithub.com/cometbft/cometbft/consensus.(*State)
      .receiveRoutine(0xc005898e08, 0x0)\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/
      consensus/state.go:773 +0x3f1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutin
      e 374\n\t/validator-home-4/elys/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:384 +0x10c
      \n"

binary info:

build_tags: netgo,ledger,muslc,osusergo,pebbledb
commit: 07597e3b1386fcc5f89f788713c0e7045f7873a2
cosmos_sdk_version: v0.47.11
go: go version go1.22.5 linux/amd64
name: elys
server_name: elys
version: v0.39.0

uname -a

Linux artorias 5.15.0-1057-intel-iot-realtime #59-Ubuntu SMP PREEMPT_RT Mon Jun 17 21:50:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux