Fast Recovery From App Hash Mismatch

Context

cronosd rollback provide a way to rollback and recover from app-hash mismatch failure, but it takes hours to run on archive node, here is a faster way to do it.

Prerequisite

• tendermint v0.34.22 cli, installed with:

  # rocksdb
  $ go install -tags rocksdb github.com/tendermint/tendermint/cmd/[email protected]
  # leveldb
  $ go install github.com/tendermint/tendermint/cmd/[email protected]
  

• install nix
• python-iavl, could be run with nix on the fly:

  $ # rocksdb
  $ nix run github:crypto-com/python-iavl/c4a8a13#iavl-cli
  $ # leveldb
  $ nix run github:crypto-com/python-iavl/c4a8a13#iavl-cli-leveldb
  

Fast Rollback

1. Disable iavl fastnode (Optional)

   After rollback, fastnode index will be recreated automatically which is slow enough, can disable it first to make it start faster.

   Set iavl-disable-fastnode = true in app.toml

2. Rollback tendermint state

   $ tendermint rollback --home /chain/.cronosd/
   Rolled back state to height $VER and hash [...]
   

3. Fast rollback app state

   replace $VER to the version number shown in the above command.

   $ nix run github:crypto-com/python-iavl/c4a8a13 -- fast-rollback --db /chain/.cronosd/data/application.db --target $VER
   

Side Effects

As far as we understand, the only side effect is waste some disk space by leaving some orphan nodes in the db, should not be a big deal.