Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(torii): limit number of blocks processed in one go #2505

Merged
merged 1 commit into from
Oct 9, 2024
Merged

feat(torii): limit number of blocks processed in one go #2505

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions bin/torii/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,10 @@
#[arg(long, default_value = "1024")]
events_chunk_size: u64,

/// Number of blocks to process before commiting to DB
#[arg(long, default_value = "10240")]
blocks_chunk_size: u64,

Check warning on line 119 in bin/torii/src/main.rs

View check run for this annotation

Codecov / codecov/patch

bin/torii/src/main.rs#L119 

Added line #L119 was not covered by tests

/// Enable indexing pending blocks
#[arg(long, action = ArgAction::Set, default_value_t = true)]
index_pending: bool,
Expand Down Expand Up @@ -239,6 +243,7 @@
EngineConfig {
max_concurrent_tasks: args.max_concurrent_tasks,
start_block: 0,
blocks_chunk_size: args.blocks_chunk_size,

Check warning on line 246 in bin/torii/src/main.rs

View check run for this annotation

Codecov / codecov/patch

bin/torii/src/main.rs#L246 

Added line #L246 was not covered by tests
events_chunk_size: args.events_chunk_size,
index_pending: args.index_pending,
polling_interval: Duration::from_millis(args.polling_interval),
Expand Down
11 changes: 9 additions & 2 deletions crates/torii/core/src/engine.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@
pub struct EngineConfig {
pub polling_interval: Duration,
pub start_block: u64,
pub blocks_chunk_size: u64,
pub events_chunk_size: u64,
pub index_pending: bool,
pub max_concurrent_tasks: usize,
Expand All @@ -142,6 +143,7 @@
Self {
polling_interval: Duration::from_millis(500),
start_block: 0,
blocks_chunk_size: 10240,
events_chunk_size: 1024,
index_pending: true,
max_concurrent_tasks: 100,
Expand Down Expand Up @@ -286,15 +288,20 @@
}
}

// TODO: since we now process blocks in chunks we can parallelize the fetching of data
pub async fn fetch_data(&mut self, cursors: &Cursors) -> Result<FetchDataResult> {
let latest_block_number = self.provider.block_hash_and_number().await?.block_number;

let from = cursors.head.unwrap_or(0);
let total_remaining_blocks = latest_block_number - from;
let blocks_to_process = total_remaining_blocks.min(self.config.blocks_chunk_size);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Possible underflow risk when calculating total_remaining_blocks.

If from is greater than latest_block_number, subtracting may cause an underflow panic. Consider adding a check to handle this scenario gracefully.

Apply this change to prevent potential underflow:

 let total_remaining_blocks = latest_block_number - from;
+if from > latest_block_number {
+    return Ok(FetchDataResult::None);
+}

Committable suggestion was skipped due to low confidence.

let to = from + blocks_to_process;

Check warning on line 298 in crates/torii/core/src/engine.rs

View check run for this annotation

Codecov / codecov/patch

crates/torii/core/src/engine.rs#L296-L298 

Added lines #L296 - L298 were not covered by tests

let instant = Instant::now();
let result = if from < latest_block_number {
let from = if from == 0 { from } else { from + 1 };
let data = self.fetch_range(from, latest_block_number, &cursors.cursor_map).await?;
debug!(target: LOG_TARGET, duration = ?instant.elapsed(), from = %from, to = %latest_block_number, "Fetched data for range.");
let data = self.fetch_range(from, to, &cursors.cursor_map).await?;
debug!(target: LOG_TARGET, duration = ?instant.elapsed(), from = %from, to = %to, "Fetched data for range.");

Check warning on line 304 in crates/torii/core/src/engine.rs

View check run for this annotation

Codecov / codecov/patch

crates/torii/core/src/engine.rs#L303-L304 

Added lines #L303 - L304 were not covered by tests
FetchDataResult::Range(data)
} else if self.config.index_pending {
let data =
Expand Down
Loading