Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bonsai archive feature #7475

Open
wants to merge 62 commits into
base: main
Choose a base branch
from
Open

Bonsai archive feature #7475

Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
a5315af
rebase of bonsai archive
jframe Mar 6, 2024
ec8a7ff
Change BonsaiReferenceTestWorldStateStorage getContextSafeCopy() to u…
jframe Mar 8, 2024
b2d9196
Force archive mode for testing
jframe Mar 8, 2024
fd300b0
Rebase on main
matthew1001 Jul 17, 2024
b10b6ae
Create new DiffBasedWorldStateConfig when copying bonsai archive worl…
matthew1001 Jul 19, 2024
6e786f3
Add BONSAI_ARCHIVE storage format
matthew1001 Jul 31, 2024
32771d3
New ACCOUNT_FREEZER_STATE DB segment. New BonsaiArchiveFreezer to lis…
matthew1001 Aug 16, 2024
b1f448d
Add freezer segment for account storage
matthew1001 Aug 21, 2024
06ba76e
Find frozen storage slots correctly, refactor the DB segment names
matthew1001 Sep 4, 2024
1d930f5
Check for blocks to freeze state for on startup. Store the most recen…
matthew1001 Sep 5, 2024
605d590
Make sure genesis world state is created in archive mode
matthew1001 Sep 20, 2024
c7c2ee5
Fix incorrect logs
matthew1001 Sep 23, 2024
990bd87
Ensure deleted storage is returned from live DB segment, not old stor…
matthew1001 Sep 26, 2024
8518428
Adding tests
matthew1001 Sep 26, 2024
a1cb77d
Refactor DB TX with retries
matthew1001 Sep 30, 2024
660d449
Honour code hash or account hash config for storing code. Add tests f…
matthew1001 Sep 30, 2024
4063128
Refactor code to move one block's worth of state/storage changes at a…
matthew1001 Oct 1, 2024
829a98e
Refactoring, fix issues, ensure only 1 batch of blocks is archived at…
matthew1001 Oct 3, 2024
faa8da1
Improve archive progress logs
matthew1001 Oct 4, 2024
0543017
Add archive mode tests to bonsai key/value tests
matthew1001 Oct 4, 2024
bf3d171
Implement snap-serving from a Bonsai Archive node and add tests
matthew1001 Oct 4, 2024
d2e5a24
Parameterize snap-server tests, fix bug in storage slot ranges
matthew1001 Oct 7, 2024
ee7c2fc
Remove todo comments
matthew1001 Oct 7, 2024
38cfdd9
Make BONSAI_ARCHIVE experimental for the first release. Add metrics. …
matthew1001 Oct 7, 2024
35d60dd
Add changelog entry
matthew1001 Oct 8, 2024
b8facef
Don't store every block that is waiting to be archived
matthew1001 Oct 8, 2024
75e57cc
Remove pre-startup catchup. Start on first block import instead
matthew1001 Oct 8, 2024
a6552df
Multiple deletes of the same address in a single block causes Bonsai …
matthew1001 Oct 8, 2024
329da00
Metric should be a gauge not a counter
matthew1001 Oct 9, 2024
e6364b2
Refactor archive tests after code refactor
matthew1001 Oct 9, 2024
8824b21
Merge branch 'main' into multi-version-flat-db-rebase
matthew1001 Oct 9, 2024
46a33ef
Remove commented out sections from tests
matthew1001 Oct 9, 2024
4051eb0
Update the BFT soak test to include a Bonsai archive node
matthew1001 Oct 17, 2024
c28a383
Add archive-specific checks to the BFT soak test
matthew1001 Oct 22, 2024
c1f239f
Merge with main
matthew1001 Nov 4, 2024
c226aa6
Update plugin API hash
matthew1001 Nov 4, 2024
a6123bb
Add javadoc
matthew1001 Nov 5, 2024
c4a9516
Merge fix
matthew1001 Nov 5, 2024
81a0907
Unit test fix
matthew1001 Nov 5, 2024
0036e61
Merge branch 'main' into multi-version-flat-db-rebase
matthew1001 Nov 5, 2024
0b0bed4
More UT fixing
matthew1001 Nov 5, 2024
614e5c2
Tidy up
matthew1001 Nov 6, 2024
5392ef4
Merge branch 'main' into multi-version-flat-db-rebase
matthew1001 Nov 6, 2024
763e810
Update ethereum/core/src/main/java/org/hyperledger/besu/ethereum/trie…
matthew1001 Jan 9, 2025
8caf063
Update plugins/rocksdb/src/main/java/org/hyperledger/besu/plugin/serv…
matthew1001 Jan 9, 2025
4065698
Update plugins/rocksdb/src/main/java/org/hyperledger/besu/plugin/serv…
matthew1001 Jan 9, 2025
602e18e
Remove debug file
matthew1001 Jan 9, 2025
f298255
Various refactoring to manage archive flat DB access more cleanly
matthew1001 Jan 14, 2025
f0a4ce1
Merge branch 'main' into multi-version-flat-db-rebase
matthew1001 Jan 14, 2025
88bd33e
Merge fix
matthew1001 Jan 14, 2025
2537ce2
Merge tests
matthew1001 Jan 14, 2025
f2954f5
Use alternative clearStorage()
matthew1001 Jan 14, 2025
c002f4b
Merge branch 'main' into multi-version-flat-db-rebase
matthew1001 Jan 14, 2025
b65fdaa
Fix metric name
matthew1001 Jan 14, 2025
b5d3c1a
Don't override rollMutableStateToBlockHash, create archive-specific f…
matthew1001 Jan 14, 2025
3873f8e
Update world state with block number being persisted
matthew1001 Jan 16, 2025
63db389
Merge branch 'main' into multi-version-flat-db-rebase
matthew1001 Jan 16, 2025
25c5bc6
Update changelog
matthew1001 Jan 16, 2025
633fad4
Move context management out of flat DB strategy
matthew1001 Jan 20, 2025
3d8d8c7
Add more test cases, tidy up code
matthew1001 Jan 21, 2025
24fa310
Merge branch 'main' into multi-version-flat-db-rebase
matthew1001 Jan 21, 2025
8f98f28
Update plugin hash
matthew1001 Jan 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 18 additions & 9 deletions ...va/org/hyperledger/besu/ethereum/trie/diffbased/common/worldview/DiffBasedWorldState.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import org.hyperledger.besu.datatypes.StorageSlotKey;
import org.hyperledger.besu.ethereum.core.BlockHeader;
import org.hyperledger.besu.ethereum.core.MutableWorldState;
import org.hyperledger.besu.ethereum.trie.diffbased.bonsai.storage.flat.BonsaiArchiveFlatDbStrategy;
import org.hyperledger.besu.ethereum.trie.diffbased.common.StorageSubscriber;
import org.hyperledger.besu.ethereum.trie.diffbased.common.cache.DiffBasedCachedWorldStorageManager;
import org.hyperledger.besu.ethereum.trie.diffbased.common.storage.DiffBasedLayeredWorldStateKeyValueStorage;
Expand Down Expand Up @@ -165,8 +166,23 @@ public void persist(final BlockHeader blockHeader) {

boolean success = false;

final DiffBasedWorldStateKeyValueStorage.Updater stateUpdater =
worldStateKeyValueStorage.updater();
// Bonsai archive uses context set in the world state storage for the specified block.
// Update the block context before putting entries to storage via calculateRootHash()
// TODO - rename calculateRootHash() to be clearer that it updates state, it doesn't just
// calculate a hash
if (worldStateKeyValueStorage.getFlatDbStrategy() instanceof BonsaiArchiveFlatDbStrategy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to put it all the time, it doesn't really have a huge impact and avoids an additional condition

&& blockHeader != null) {
DiffBasedWorldStateKeyValueStorage.Updater stateUpdater = worldStateKeyValueStorage.updater();
stateUpdater
.getWorldStateTransaction()
.put(
TRIE_BRANCH_STORAGE,
WORLD_BLOCK_NUMBER_KEY,
Long.toHexString(blockHeader.getNumber()).getBytes(StandardCharsets.UTF_8));
stateUpdater.commit();
}

DiffBasedWorldStateKeyValueStorage.Updater stateUpdater = worldStateKeyValueStorage.updater();
Runnable saveTrieLog = () -> {};

try {
Expand Down Expand Up @@ -207,13 +223,6 @@ public void persist(final BlockHeader blockHeader) {
.getWorldStateTransaction()
.put(TRIE_BRANCH_STORAGE, WORLD_BLOCK_HASH_KEY, blockHeader.getHash().toArrayUnsafe());
worldStateBlockHash = blockHeader.getHash();

stateUpdater
.getWorldStateTransaction()
.put(
TRIE_BRANCH_STORAGE,
WORLD_BLOCK_NUMBER_KEY,
Long.toHexString(blockHeader.getNumber()).getBytes(StandardCharsets.UTF_8));
} else {
stateUpdater.getWorldStateTransaction().remove(TRIE_BRANCH_STORAGE, WORLD_BLOCK_HASH_KEY);
worldStateBlockHash = null;
Expand Down
Loading