feat: forester batched ops #1357
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added bloomfilter batch e2e test failing because filter change doesn't work yet debugging batched Merkle tree e2e test works with output queue fix: e2e test work with input and output queues chore: rename wipe bloomfilter -> wipe_batch refactored input queue into mt account stash test: rnd account init works chore: replace is_inserted with state chore: moved offset to bounded vec, added bounded vec init and deserialize fixed bounded vec imports refactor: partial batch zkp updates, flexible sized root history array with seq implemented batch prove and verification in rust tests untested stash: batched updates work with simulate transactions test add comment stash before account *mut onchain account init works test: account compression program batch updates work feat: add flexbile nullifier to circuit feat: implement nullifier derivation from tx hash test: batch test: bounded vec cleanup cleanup cleanup feat: add flexible nullifier to update circuit fix: e2e test chore: add full asserts to simulate tx test test: root overwrite test cleanup
…te prove_batch_append test Change CI runner to buildjet-8vcpu-ubuntu-2204 for rust tests run rust.yml on buildjet-16vcpu-ubuntu-2204 debug rust.yml workflow Add step to download proving keys for big batches Remove redundant setup steps and update test configurations Update ProverConfig to use test-specific circuits cleaning up after rebase to main
Issue: the batch append zkp public input LeavesHashchainHash is computed dynamically from the values array during the batched update. If the values array changes in between proof generation and transaction execution the batched append update fails. -> Users can make batched append update transactions fail. Solution: compute LeavesHashchainHash incrementally when the output value is inserted into the value array (the same way we compute it with nullifiers for input queue insertions). Changes: 1. switch to updated circuits 2. batch.rs - add start_index field to derive the leaf_index of values from a value array - introduce store_and_hash_value to store an output queue value and add it to the LeavesHashchainHash 3. batched_queue.rs - init_queue initialize start_index during account initialization - use store_and_hash_value instead of store_value when inserting into output queues - update batch.start_index when wiping a batch 4. batched_merkle_tree.rs - remove subtree field (it is not used in new batch append circuit) - initialize root_history with the zero_value_root - add hashchain store for leaves_hashchain in ZeroCopyBatchedMerkleTreeAccount - add leaf_index to nullifier hash H(value, leaf_index, tx_hash) Co-authored-by: ananas-block <[email protected]>
…Accounts, Bundle} from forester-utils
| @@ -35,6 +37,17 @@ fn process_state_account(account: &Account, pubkey: Pubkey) -> Result<TreeAccoun | |||
| )) | |||
| } | |||
|
|
|||
| fn process_batch_state_account(account: &mut Account, pubkey: Pubkey) -> Result<TreeAccounts> { | |||
| check_discriminator::<BatchedMerkleTreeAccount>(&account.data)?; | |||
| let tree_account = ZeroCopyBatchedMerkleTreeAccount::from_bytes_mut(&mut account.data)?; | |||
There was a problem hiding this comment.
from_bytes_mut checks the discriminator
forester/tests/batched_op_test.rs
Outdated
Comment on lines
178
to
188
| if i == merkle_tree.get_account().queue.batch_size / 2 { | ||
| instruction_data = Some( | ||
| create_append_batch_ix_data( | ||
| &mut e2e_env.rpc, | ||
| &mut e2e_env.indexer.state_merkle_trees[0], | ||
| state_merkle_tree_pubkey, | ||
| output_queue_pubkey, | ||
| ) | ||
| .await, | ||
| ); | ||
| } |
There was a problem hiding this comment.
best remove this it was just to test concurrency in the other test
| .ok_or(AccountCompressionErrorCode::InclusionProofByIndexFailed)?; | ||
|
|
||
| if element == value { | ||
| *element = [0; 32]; |
There was a problem hiding this comment.
Suggested change
| *element = [0; 32]; |
We need to remove this so that we can use the value from output_queue.value_vec[full_batch_index] directly instead from the indexer.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.