Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[saya] Scheduler #1917

Merged
merged 64 commits into from
Jun 11, 2024
Merged
Changes from 1 commit
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
f532a6a
proof scheduler
matzayonc May 3, 2024
6fa7fd6
cleared leftovers
matzayonc May 3, 2024
f5d6b06
feature fix
matzayonc May 3, 2024
25644cf
Deserialization and tests fix
chudkowsky May 3, 2024
19f7a0d
uncommented apply diffs logic
matzayonc May 6, 2024
b6fb8b5
genesis block fix
matzayonc May 6, 2024
1f0b356
fetching last block
matzayonc May 6, 2024
99703ef
moved fetching to separate function
matzayonc May 6, 2024
6fe5cc5
process_block no async
chudkowsky May 6, 2024
379abd3
Adding traces and infos for async functions
chudkowsky May 6, 2024
5faf4b5
made dojo core work with merger program
matzayonc May 7, 2024
98aef17
minor logs adjustments in saya
matzayonc May 7, 2024
1ab8a46
final world contract
matzayonc May 8, 2024
686bb80
preparation for http prover
matzayonc May 17, 2024
d3fad19
proving example data by
matzayonc May 20, 2024
8d214c7
serializing arguments
matzayonc May 20, 2024
c75a99b
debug print
chudkowsky May 20, 2024
205a97b
only one program in verifier
matzayonc May 21, 2024
fc224ce
preparing args for prover
matzayonc May 21, 2024
8a00505
batch size arg
matzayonc May 21, 2024
020e0bb
moved extraction of data from proof to separate module
matzayonc May 21, 2024
038004b
minor cleanup
matzayonc May 21, 2024
e00050c
updated bin/scheduler
chudkowsky May 21, 2024
dac6cd0
laziness fix and update prover sdk
matzayonc May 22, 2024
6588381
half done
matzayonc May 23, 2024
4353302
cairo 1 behind flag
matzayonc May 24, 2024
c247b09
added cairo 0 differ and merger
matzayonc May 24, 2024
7aa553a
support for 2 programs in `upgrade_state`
matzayonc May 24, 2024
e5f7636
end 2 end saya readme
matzayonc May 24, 2024
e568ef4
data extraction from proof
matzayonc May 24, 2024
e0ef120
saya working with cairo 0
matzayonc May 27, 2024
d697eac
updated prover and readme
matzayonc May 27, 2024
9d89996
added a clause about keygen
matzayonc May 27, 2024
de4cf5a
updated port in readme
matzayonc May 27, 2024
aebc146
clippy and formatting
matzayonc May 27, 2024
23f2136
ignored heavy tests
matzayonc May 27, 2024
c6e405f
moved readme to bin
matzayonc May 28, 2024
f282f0e
updated stone prover image
matzayonc May 28, 2024
0ad62e7
made tests pass
matzayonc May 28, 2024
5fba11c
removed unneeded files and fixed compilation
matzayonc May 28, 2024
8f8f0ed
formatting after rebase
matzayonc May 28, 2024
a49a3e1
better fetch parallelization
matzayonc May 28, 2024
bf2e4db
fixed valid ignoring of not l1_handler
matzayonc May 28, 2024
e65768f
fixed most tests and made compiled programs one line
matzayonc May 28, 2024
cf5c18f
cairo import formatting
matzayonc May 28, 2024
e04cbc6
using recursive layout
matzayonc May 30, 2024
5af850e
changed docker images to latest
matzayonc May 30, 2024
66a9791
Update examples/spawn-and-move/Scarb.toml
matzayonc May 30, 2024
506edaf
most of pr comments
matzayonc May 31, 2024
a947dea
keeping track of nonce to avoid sleep
matzayonc May 31, 2024
c332040
rest of PR comments
matzayonc May 31, 2024
246aaf3
made changes to fix the tests
matzayonc May 31, 2024
cc44a15
formatting
matzayonc May 31, 2024
1ecf70f
fixed all the tests
matzayonc May 31, 2024
6d09880
final formatting
matzayonc May 31, 2024
fe83ffa
rebuilt artifacts
matzayonc May 31, 2024
f6d3d32
fix: typo in README
glihm Jun 3, 2024
6266e2a
fixed duplicated proving state update
matzayonc Jun 4, 2024
8af108e
fix: remove cargo warning for dual license source
glihm Jun 10, 2024
338023e
fix: reword some commands + fix options for saya
glihm Jun 10, 2024
a57d30b
Merge remote-tracking branch 'dojo/main' into saya/scheduler
glihm Jun 10, 2024
7b7433c
fix: update README
glihm Jun 11, 2024
3fd5f5b
fix: ensure model get fails if no key is provided
glihm Jun 11, 2024
4777215
fix: use --bin instead of -p
glihm Jun 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
uncommented apply diffs logic
matzayonc committed May 30, 2024
commit 19f7a0ddbde5cf0f93f6a135b2140c7c68eb4ecb
2 changes: 1 addition & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

123 changes: 66 additions & 57 deletions crates/saya/core/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
//! Saya core library.

use std::sync::Arc;
use std::time::Duration;

use cairo_proof_parser::output::{extract_output, ExtractOutputResult};
use cairo_proof_parser::parse;
use cairo_proof_parser::program::{extract_program, ExtractProgramResult};
use futures::future::{self, join};
use katana_primitives::block::{BlockNumber, FinalityStatus, SealedBlock, SealedBlockWithStatus};
use katana_primitives::transaction::Tx;
@@ -10,13 +14,16 @@ use prover::ProverIdentifier;
use saya_provider::rpc::JsonRpcProvider;
use saya_provider::Provider as SayaProvider;
use serde::{Deserialize, Serialize};
use tracing::{error, trace};
use starknet_crypto::poseidon_hash_many;
use tokio::time::sleep;
use tracing::{error, info, trace};
use url::Url;

use crate::blockchain::Blockchain;
use crate::data_availability::{DataAvailabilityClient, DataAvailabilityConfig};
use crate::error::SayaResult;
use crate::prover::{extract_messages, ProgramInput, Scheduler};
use crate::verifier::VerifierIdentifier;

pub mod blockchain;
pub mod data_availability;
@@ -96,6 +103,11 @@ impl Saya {
join(self.provider.fetch_block(0), self.provider.fetch_block(block - 1)).await;
let genesis_state_hash = genesis_block?.header.header.state_root;
let mut previous_block_state_root = block_before_the_first?.header.header.state_root;

// The structure responsible for proving.
let mut prove_scheduler =
Scheduler::new(4, self.config.world_address, ProverIdentifier::Stone);

loop {
let latest_block = match self.provider.block_number().await {
Ok(block_number) => block_number,
@@ -131,26 +143,21 @@ impl Saya {
.map(|(b, s)| (b, s, genesis_state_hash))
.collect::<Vec<_>>();

let mut processed = Vec::with_capacity(params.len());
// Updating the local state sequentially, as there is only one instance of
// `self.blockchain` This part does no actual proving, so should not be a
// problem
for p in params.clone() {
let prover_input = self.process_block(block, p).await?;
if let Some(input) = prover_input {
processed.push(input);
self.process_block(&mut prove_scheduler, block, p).await?;

if prove_scheduler.is_full() {
// ProverIdentifier::Http(self.config.prover_url.clone()),
self.process_proven(prove_scheduler).await?; // TODO: spawn it, but store a handle
prove_scheduler =
Scheduler::new(4, self.config.world_address, ProverIdentifier::Stone);
}

block += 1;
}

// Prove each of the leaf nodes of the recursion tree and merge them into one
let proof = Scheduler::prove_recursively(
processed,
self.config.world_address,
ProverIdentifier::Stone,
)
.await?;
println!("Proof: {:?}", proof.0);
}
}

@@ -164,10 +171,7 @@ impl Saya {
///
/// 3. Computes facts for this state transition. We may optimistically register the facts.
///
/// 4. Computes the proof from the trace with a prover.
///
/// 5. Registers the facts + the send the proof to verifier. Not all provers require this step
/// (a.k.a. SHARP).
/// 4. Starts computing the proof from the trace with a prover.
///
/// # Arguments
///
@@ -176,9 +180,10 @@ impl Saya {
/// genesis block.
async fn process_block(
&mut self,
prove_scheduler: &mut Scheduler,
block_number: BlockNumber,
blocks: (SealedBlock, FieldElement, FieldElement),
) -> SayaResult<Option<ProgramInput>> {
) -> SayaResult<()> {
trace!(target: LOG_TARGET, block_number = %block_number, "Processing block.");

let (block, prev_state_root, _genesis_state_hash) = blocks;
@@ -197,14 +202,14 @@ impl Saya {
self.blockchain.update_state_with_block(block.clone(), state_updates)?;

if block_number == 0 {
return Ok(None);
return Ok(());
}

let exec_infos = self.provider.fetch_transactions_executions(block_number).await?;

if exec_infos.is_empty() {
trace!(target: LOG_TARGET, block_number, "Skipping empty block.");
return Ok(None);
return Ok(());
}

let transactions = block
@@ -233,42 +238,46 @@ impl Saya {
};
state_diff_prover_input.fill_da(self.config.world_address);

// let world_da = new_program_input.da_as_calldata(self.config.world_address);
// let world_da_printable: Vec<String> = world_da.iter().map(|x| x.to_string()).collect();
// trace!(target: LOG_TARGET, world_da = ?world_da_printable, "World DA computed.");

trace!(target: LOG_TARGET, "Proving block {block_number}.");
// let proof = prover::prove(
// new_program_input.serialize(self.config.world_address)?,
// ProverIdentifier::Http(self.config.prover_url.clone()),
// )
// .await?;
// info!(target: LOG_TARGET, block_number, "Block proven.");

// let serialized_proof: Vec<FieldElement> = parse(&proof)?.into();

// trace!(target: LOG_TARGET, block_number, "Verifying block.");
// let transaction_hash = verifier::verify(
// VerifierIdentifier::HerodotusStarknetSepolia(self.config.fact_registry_address),
// serialized_proof,
// )
// .await?;
// info!(target: LOG_TARGET, block_number, transaction_hash, "Block verified.");

// let ExtractProgramResult { program: _, program_hash } = extract_program(&proof)?;
// let ExtractOutputResult { program_output, program_output_hash } = extract_output(&proof)?;
// let expected_fact = poseidon_hash_many(&[program_hash, program_output_hash]).to_string();
// info!(target: LOG_TARGET, expected_fact, "Expected fact.");

// sleep(Duration::from_millis(5000)).await;

// trace!(target: LOG_TARGET, block_number, "Applying diffs.");
// let transaction_hash =
// dojo_os::starknet_apply_diffs(self.config.world_address, world_da, program_output)
// .await?;
// info!(target: LOG_TARGET, block_number, transaction_hash, "Diffs applied.");

Ok(Some(state_diff_prover_input))
prove_scheduler.push_diff(state_diff_prover_input).unwrap();
glihm marked this conversation as resolved.
Show resolved Hide resolved

info!(target: LOG_TARGET, block_number, "Block processed.");

Ok(())
}

/// Registers the facts + the send the proof to verifier. Not all provers require this step
/// (a.k.a. SHARP).
async fn process_proven(&self, prove_scheduler: Scheduler) -> SayaResult<()> {
// Prove each of the leaf nodes of the recursion tree and merge them into one
let (proof, state_diff) = prove_scheduler.proved().await.unwrap();

let serialized_proof: Vec<FieldElement> = parse(&proof)?.into();
let world_da = state_diff.world_da.unwrap();

let block_number = 999; // TODO: find a way to log it properly

trace!(target: LOG_TARGET, block_number, "Verifying block.");
let transaction_hash = verifier::verify(
VerifierIdentifier::HerodotusStarknetSepolia(self.config.fact_registry_address),
serialized_proof,
)
.await?;
info!(target: LOG_TARGET, block_number, transaction_hash, "Block verified.");

let ExtractProgramResult { program: _, program_hash } = extract_program(&proof)?;
let ExtractOutputResult { program_output, program_output_hash } = extract_output(&proof)?;
let expected_fact = poseidon_hash_many(&[program_hash, program_output_hash]).to_string();
info!(target: LOG_TARGET, expected_fact, "Expected fact.");

sleep(Duration::from_millis(5000)).await;
glihm marked this conversation as resolved.
Show resolved Hide resolved

trace!(target: LOG_TARGET, block_number, "Applying diffs.");
let transaction_hash =
dojo_os::starknet_apply_diffs(self.config.world_address, world_da, program_output)
.await?;
info!(target: LOG_TARGET, block_number, transaction_hash, "Diffs applied.");

Ok(())
}
}

4 changes: 2 additions & 2 deletions examples/spawn-and-move/Scarb.toml
Original file line number Diff line number Diff line change
@@ -7,7 +7,7 @@ version = "0.6.0"
edition = "2023_10"

[cairo]
sierra-replace-ids = true
sierra-replace-ids = trues

[dependencies]
dojo = { path = "../../crates/dojo-core" }
@@ -22,7 +22,7 @@ name = "example"
[tool.dojo.env]
rpc_url = "http://localhost:5050/"

# Default account for katana with seed = 0
# Default account for katana with seed = 0s
matzayonc marked this conversation as resolved.
Show resolved Hide resolved
account_address = "0x6162896d1d7ab204c7ccac6dd5f8e9e7c25ecd5ae4fcb4ad32e57786bb46e03"
private_key = "0x1800000000300000180000000000030000000000003006001800006600"
world_address = "0x2e31cfde9f9990c7fe44b25043e3c6958a849c0a66ab535686d2b710e97f309"