-
Notifications
You must be signed in to change notification settings - Fork 11.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[indexer-alt] expose mapping from cp to tx or epoch interval in indexer-alt-framework for pruners if needed #20605
base: main
Are you sure you want to change the base?
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Skipped Deployments
|
crates/sui-indexer-alt-framework/migrations/2024-12-11-234958_cp_mapping/up.sql
Outdated
Show resolved
Hide resolved
84e160e
to
5f740dd
Compare
f961d13
to
183e9d8
Compare
160d0a8
to
6cabf8a
Compare
crates/sui-indexer-alt-framework/src/handlers/cp_sequence_numbers.rs
Outdated
Show resolved
Hide resolved
crates/sui-indexer-alt-framework/src/handlers/cp_sequence_numbers.rs
Outdated
Show resolved
Hide resolved
crates/sui-indexer-alt-framework/src/handlers/cp_sequence_numbers.rs
Outdated
Show resolved
Hide resolved
let [first, last] = results.as_slice() else { | ||
warn!("No checkpoint mapping found for checkpoint {from_cp} and {to_cp}. Found {} mapping(s) instead of 2", results.len()); | ||
return Err(RangeError::NoCheckpointMapping(from_cp, to_cp)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
from_cp == to_cp
is a valid case where this slice pattern would not match.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the simplest way to structure this code is to query using eq_any
and then scan the results to find a match for each of from_cp
and to_cp
, something like the following, in the case of tx_interval
:
let results: Vec<(i64, i64)> = cp_sequence_numbers::table
.select((cp_sequence_numbers::cp_sequence_number, cp_sequence_numbers::tx_lo))
.filter(cp_sequence_numbers::cp_sequence_number.eq_any([from_cp as i64, to_cp as i64])
.load(conn)
.await?;
let Some(from_tx) = results.iter().find_map(|(cp, tx)| (cp == from_cp).then(tx)) else {
bail!("Could not find tx sequence number for checkpoint {from_cp}");
};
let Some(to_tx) = results.iter().find_map(|(cp, tx)| (cp == to_cp).then(tx)) else {
bail!("Could not find tx sequence number for checkpoint {to_cp}");
};
Ok((from_tx..to_tx))
This also means you're left with only one error case, which further justifies simplifying the error story!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Btw, in what scenario would from_cp == to_cp
be a valid case?
watermark.next_chunk
returns to
as (self.pruner_hi + size).min(self.reader_lo)
. pruner_hi
and reader_lo
should not be the same value, and size
wouldn't make sense as a non-positive integer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i suppose from == to
is possible when we update pruner_hi
on the pruning side before the committing side gets to update reader_lo
In that case though, it seems to be confusing to allow the pruning code to proceed since we expect [from_cp, to_cp)
4a54772
to
1b7b826
Compare
af5edf6
to
92c75e5
Compare
…o use the checkpoint, tx, or epoch_interval, all of which have an exclusive hi address first set of comments. trait fn prune still accepts from and to checkpoints, up to handler prune to decide what it needs. doc comments for PrunableRange regarding what it expects on instantiation extend PrunableRange::get_range to return an error if from_cp is not < to_cp. Output warning if it tries to fetch from cp_mapping and can't. and remove warning in framework/lib.rs around cp_mapping split out models from handlers simplify error story expose intervals as functions instead of in a struct anyhow error when from_cp not < to_cp
92c75e5
to
87504e7
Compare
Description
With the
cp_sequence_numbers
table, pruner tasks that need information beyond the pruner checkpoint range can depend on the table for the corresponding interval. A pruner task can now calltx_interval
orepoch_interval
, and will throw an anyhow error if the mapping for the checkpoint cannot be retrieved.Test plan
How did you test the new or updated feature?
Release notes
Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.
For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.