Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate_database_lmdb_to_rocksdb improvements #4647

Merged
merged 12 commits into from
Jul 10, 2024

Conversation

RickiNano
Copy link
Contributor

@RickiNano RickiNano commented Jun 13, 2024

The migrate_database_lmdb_to_rocksdb option is running for a very long time with the current ledger size (almost 200 million blocks). On my local machine it took 65 minutes to complete. Nothing is written on screen during this process and users may think the process has stalled.
This PR adds some progress feedback. One update for each of the 7 tables that are migrated.
It also adds a simple disk space check to warn users if they might not have enough space to complete the migration.
The current converted RocksDb database is 73 GB, and the warning is given if the system has less than 75GB available.
The warning is given based on the size of the LMDB database that is being migrated. The final RocksDb size is approximately 65% of the LMDB space.

@pwojcikdev
Copy link
Contributor

This should be using node logger not cout, otherwise logs aren't saved to disk. There are some places in the existing code that are using cout too, but those are leftovers. Also, printing progress every x converted entries would be nice.

@RickiNano
Copy link
Contributor Author

I have been unable to get node logger implemented. I don't think it's important since the migration is a one time process.
I've updated the code with more granular updates to indicate that progress is being made.
Here is the output of a full production lmdb to rocksDb migration:

image

@pwojcikdev
Copy link
Contributor

I have been unable to get node logger implemented. I don't think it's important since the migration is a one time process.

And that's exactly the reason why it should be logged well, in line with production ready code. In case something goes wrong there should be a persistent record.

@RickiNano
Copy link
Contributor Author

@pwojcikdev
I got the Nano logger implemented now.
Instead of outputting dots for each step it is now giving actual numbers to the log.
image

gr0vity-dev pushed a commit to gr0vity-dev/nano-node that referenced this pull request Jul 1, 2024
gr0vity-dev pushed a commit to gr0vity-dev/nano-node that referenced this pull request Jul 2, 2024
gr0vity-dev pushed a commit to gr0vity-dev/nano-node that referenced this pull request Jul 2, 2024
gr0vity-dev pushed a commit to gr0vity-dev/nano-node that referenced this pull request Jul 2, 2024
gr0vity-dev pushed a commit to gr0vity-dev/nano-node that referenced this pull request Jul 2, 2024
@qwahzi qwahzi added this to the V27 milestone Jul 2, 2024
nano/nano_node/entry.cpp Outdated Show resolved Hide resolved
nano/secure/ledger.cpp Outdated Show resolved Hide resolved
gr0vity-dev pushed a commit to gr0vity-dev/nano-node that referenced this pull request Jul 7, 2024
@clemahieu clemahieu merged commit 809becf into nanocurrency:develop Jul 10, 2024
24 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Merged / V27.0
Development

Successfully merging this pull request may close these issues.

4 participants