tzindex path change leads to broken data

[tmeinlschmidt]

hey,

situation:
we had data in /somedir-tzindex ... I've stopped tzindex (17.0.1), moved to /some-dir/tzindex, changed mount volumes in docker etc (it's /some-dir to /data) and restarted... it ended up with

2023/12/27 22:06:33.175428 INFO MAIN Using config file config.json
2023/12/27 22:06:33.175445 INFO MAIN Blockwatch Data Inc. TzIndex -- v17.0.1-alpine 2ca737b
2023/12/27 22:06:33.175446 INFO MAIN (c) Copyright 2018-2023 Blockwatch Data Inc.
2023/12/27 22:06:33.175447 INFO MAIN Go version go1.21.0
2023/12/27 22:06:33.175449 INFO MAIN Starting on 12 cores
2023/12/27 22:06:33.176233 INFO ETL Registered custom metadata extension.
2023/12/27 22:06:33.176239 INFO MAIN Using bolt database /data/tzindex
2023/12/27 22:06:33.176465 INFO ETL Initializing blockchain crawler in sync mode.
2023/12/27 22:06:33.176716 ERRO ETL op index with unexpected height 0/2379497
2023/12/27 22:06:33.176719 ERRO ETL event index with unexpected height 0/2379497
2023/12/27 22:06:33.176720 ERRO ETL flow index with unexpected height 0/2379497
2023/12/27 22:06:33.176721 ERRO ETL chain index with unexpected height 0/2379497
2023/12/27 22:06:33.176722 ERRO ETL storage index with unexpected height 0/2379497
2023/12/27 22:06:33.176723 ERRO ETL constant index with unexpected height 0/2379497
2023/12/27 22:06:33.176724 ERRO ETL ticket index with unexpected height 0/2379497
2023/12/27 22:06:33.176725 ERRO ETL block index with unexpected height 0/2379497
2023/12/27 22:06:33.176726 ERRO ETL bigmap index with unexpected height 0/2379497
2023/12/27 22:06:33.176726 ERRO ETL metadata index with unexpected height 0/2379497
2023/12/27 22:06:33.176728 ERRO ETL supply index with unexpected height 0/2379497
2023/12/27 22:06:33.176734 INFO ETL Closing account index.
2023/12/27 22:06:33.176819 INFO ETL Closing balance index.
2023/12/27 22:06:33.176885 INFO ETL Closing contract index.
2023/12/27 22:06:33.176943 INFO ETL Closing storage index.
2023/12/27 22:06:33.176992 INFO ETL Closing constant index.
2023/12/27 22:06:33.177044 INFO ETL Closing block index.
2023/12/27 22:06:33.177094 INFO ETL Closing op index.
2023/12/27 22:06:33.177139 INFO ETL Closing event index.
2023/12/27 22:06:33.177188 INFO ETL Closing flow index.
2023/12/27 22:06:33.177234 INFO ETL Closing chain index.
2023/12/27 22:06:33.177362 INFO ETL Closing supply index.
2023/12/27 22:06:33.177416 INFO ETL Closing bigmap index.
2023/12/27 22:06:33.177462 INFO ETL Closing metadata index.
2023/12/27 22:06:33.177509 INFO ETL Closing ticket index.
Error: error initializing crawler: indexer init: Missing database files! Looks like you deleted a database file.

running params:

            "Env": [
                "TZ_RPC_DIAL_TIMEOUT=60s",
                "TZ_RPC_RESPONSE_TIMEOUT=240s",
                "TZ_RPC_CONTINUE_TIMEOUT=240s",
                "TZ_SERVER_READ_TIMEOUT=60s",
                "TZ_SERVER_HEADER_TIMEOUT=120s",
                "TZ_DB_ACCOUNT_CACHE_SIZE=256",
                "TZ_DB_BALANCE_CACHE_SIZE=256",
                "TZ_DB_BLOCK_CACHE_SIZE=16",
                "TZ_DB_CONTRACT_CACHE_SIZE=8",
                "TZ_DB_STORAGE_CACHE_SIZE=32",
                "TZ_DB_OP_CACHE_SIZE=256",
                "TZ_DB_FLOW_CACHE_SIZE=256",
                "TZ_DB_BIGMAP_VALUES_CACHE_SIZE=2048",
                "TZ_DB_BIGMAP_UPDATES_CACHE_SIZE=256",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "BUILD_TARGET=tzindex"
            ],
            "Cmd": [
                "-rpc.url=localhost:8732",
                "-db.path=/data/tzindex",
                "-light",
                "-notls",
                "-server.addr=0.0.0.0",
                "-server.port=8080"
            ],

if I remove all the .db files in that folder and restarted tzindex, it works - and creates all the same files ("just" empty)..

[echa]

Hi, I think the problem is unrelated to changing the path. It looks like your state.db file is corrupt. This can happen in rare circumstances when a container is killed (OOM or otherwise) and the indexer did not have time to close database files during shutdown.

I'm interested: Did you use the TzIndex snapshot feature to create a backup copy. If yes, then this could be a bug in this feature.

[tmeinlschmidt]

hi, nope, no backup. Docker container was stop with 60sec grace time and sigint. I do have filebackup, can I somehow check that state.db for corruption?

[echa]

You could use boltbrowser https://github.com/br0xen/boltbrowser and edit the contents of all keys in bucket tips to the most recent block (you find its height and hash in bucket chaintip). This makes the check on database load happy, but there's no guarantee tables contain the correct state.

I have checked code paths for saving chaintip & indexer tips and found both pieces are not atomic. Moreover in combination with snapshots this will lead to a broken statedb. I have a fix ready soon, but this won't help with your current database unfortunately.