Fast Recovery From App Hash Mismatch

Context

cronosd rollback provide a way to rollback and recover from app-hash mismatch failure, but it takes hours to run, here is a faster way to do it.

Prerequisite

• tendermint v0.34.22 cli, installed with:

  # rocksdb
  $ go install -tags rocksdb github.com/tendermint/tendermint/cmd/[email protected]
  # leveldb
  $ go install github.com/tendermint/tendermint/cmd/[email protected]
  

• python-iavl, could be run with nix on the fly:

  $ # rocksdb
  $ nix run github:yihuang/python-iavl/a1aeedf
  $ # leveldb
  $ nix run github:yihuang/python-iavl/a1aeedf#iavl-cli-leveldb
  

Fast Rollback

1. Disable iavl fastnode (Optional)

   After rollback, fastnode index will be recreated automatically which is slow enough, can disable it first to make it start faster.

   Set iavl-disable-fastnode = true in app.toml

2. Rollback tendermint state

   $ tendermint rollback --home /chain/.cronosd/
   Rolled back state to height $VER and hash [...]
   

3. Fast rollback app state

   replace $VER to the version number shown in the above command.

   $ nix run github:yihuang/python-iavl/a1aeedf -- fast-rollback --db /chain/.cronosd/data/application.db --target $VER