feat: cache smt every 1000 blocks #6732
Conversation
SWvheerden
changed the title
Test Results (CI) 3 files 129 suites 36m 33s ⏱️ Results for commit 2e64414. ♻️ This comment has been updated with latest results. |
| @@ -1341,6 +1384,10 @@ impl LMDBDatabase { | |||
| header.height, | |||
| &BlockAccumulatedData::new(kernel_mmr.get_pruned_hash_set()?, total_kernel_sum), | |||
| )?; | |||
| allow_smt_change.store(false, Ordering::SeqCst); | |||
There was a problem hiding this comment.
it does not,
This is to ensure it only ever runs once on smt per transction
| allow_smt_change: Arc<AtomicBool>, | ||
| ) -> Result<(), ChainStorageError> { | ||
| let can_we_change_smt = allow_smt_change.load(Ordering::SeqCst); |
There was a problem hiding this comment.
There is a lot happening in this function outside of whether we can change the SMT or not, and I don't think it matters for most of the function, as we don't end up updating the SMT until the end.
Is this something that we could not do in this function, and somehow test at the caller after this returns, to see fi we also need to call insert_smt ?
There was a problem hiding this comment.
This is a weird design detail, and I think the least worse one there is.
Main issue we have is we have the DB which is the source of truth, now we have the memory cache of the SMT because loading it takes too long. Now we want to create a File copy of the cache as calculating it takes too long.
So the DB needs to stay the source of truth and always needs to be correct and in sync.
But all other patterns require multiple rwlock's which might lead to an inconsistent DB state, and or requires manual dev implementation to dave the SMT
Co-authored-by: Brian Pearce <[email protected]>
Description
Caches the smt every 1000 blocks by default or half the prune interval.
Motivation and Context
Loading the smt can take almost 2 mins on nextnet atm. This brings down he loading to 3 secs.
How Has This Been Tested?
Manual tests on nextnet