Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[indexer-alt] expose mapping from cp to tx or epoch interval in indexer-alt-framework for pruners if needed #20605

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

wlmyng
Copy link
Contributor

@wlmyng wlmyng commented Dec 12, 2024

Description

With the cp_sequence_numbers table, pruner tasks that need information beyond the pruner checkpoint range can depend on the table for the corresponding interval. A pruner task can now call tx_interval or epoch_interval, and will throw an anyhow error if the mapping for the checkpoint cannot be retrieved.

Test plan

How did you test the new or updated feature?


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • Indexer:
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • REST API:

Copy link

vercel bot commented Dec 12, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
sui-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 26, 2024 5:13pm
3 Skipped Deployments
Name Status Preview Comments Updated (UTC)
multisig-toolkit ⬜️ Ignored (Inspect) Visit Preview Dec 26, 2024 5:13pm
sui-kiosk ⬜️ Ignored (Inspect) Visit Preview Dec 26, 2024 5:13pm
sui-typescript-docs ⬜️ Ignored (Inspect) Visit Preview Dec 26, 2024 5:13pm

@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 12, 2024 03:20 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 12, 2024 03:34 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 12, 2024 04:53 — with GitHub Actions Inactive
@wlmyng wlmyng changed the title [indexer-alt] cp_mapping table to support pruning [indexer-alt] add cp_mapping table to support pruning and update Handler::prune Dec 12, 2024
@wlmyng wlmyng marked this pull request as ready for review December 12, 2024 04:56
@wlmyng wlmyng requested a review from ronny-mysten as a code owner December 12, 2024 04:56
@wlmyng wlmyng requested review from amnn and removed request for ronny-mysten December 12, 2024 04:56
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 12, 2024 04:56 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 12, 2024 21:16 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 12, 2024 21:33 — with GitHub Actions Inactive
@wlmyng wlmyng force-pushed the indexer-alt-cp-mapping-for-pruning branch from 84e160e to 5f740dd Compare December 13, 2024 21:33
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 13, 2024 21:33 — with GitHub Actions Inactive
@wlmyng wlmyng changed the base branch from main to indexer-alt-framework-cp-mapping December 13, 2024 21:33
@wlmyng wlmyng changed the title [indexer-alt] add cp_mapping table to support pruning and update Handler::prune [indexer-alt] update indexer-alt-framework code's pruner to require cp_mapping Dec 13, 2024
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 13, 2024 21:53 — with GitHub Actions Inactive
@wlmyng wlmyng requested a review from emmazzz December 13, 2024 21:56
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 16, 2024 18:53 — with GitHub Actions Inactive
@wlmyng wlmyng requested review from lxfind and amnn December 16, 2024 18:53
@wlmyng wlmyng force-pushed the indexer-alt-framework-cp-mapping branch 2 times, most recently from f961d13 to 183e9d8 Compare December 19, 2024 18:06
@wlmyng wlmyng force-pushed the indexer-alt-cp-mapping-for-pruning branch from 160d0a8 to 6cabf8a Compare December 19, 2024 18:21
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 19, 2024 18:21 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 19, 2024 18:22 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 19, 2024 18:23 — with GitHub Actions Inactive
Comment on lines 67 to 69
let [first, last] = results.as_slice() else {
warn!("No checkpoint mapping found for checkpoint {from_cp} and {to_cp}. Found {} mapping(s) instead of 2", results.len());
return Err(RangeError::NoCheckpointMapping(from_cp, to_cp));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from_cp == to_cp is a valid case where this slice pattern would not match.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the simplest way to structure this code is to query using eq_any and then scan the results to find a match for each of from_cp and to_cp, something like the following, in the case of tx_interval:

let results: Vec<(i64, i64)> = cp_sequence_numbers::table
    .select((cp_sequence_numbers::cp_sequence_number, cp_sequence_numbers::tx_lo))
    .filter(cp_sequence_numbers::cp_sequence_number.eq_any([from_cp as i64, to_cp as i64])
    .load(conn)
    .await?;

let Some(from_tx) = results.iter().find_map(|(cp, tx)| (cp == from_cp).then(tx)) else {
    bail!("Could not find tx sequence number for checkpoint {from_cp}");
};

let Some(to_tx) = results.iter().find_map(|(cp, tx)| (cp == to_cp).then(tx)) else {
    bail!("Could not find tx sequence number for checkpoint {to_cp}"); 
};

Ok((from_tx..to_tx))

This also means you're left with only one error case, which further justifies simplifying the error story!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, in what scenario would from_cp == to_cp be a valid case?

watermark.next_chunk returns to as (self.pruner_hi + size).min(self.reader_lo). pruner_hi and reader_lo should not be the same value, and size wouldn't make sense as a non-positive integer

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i suppose from == to is possible when we update pruner_hi on the pruning side before the committing side gets to update reader_lo

In that case though, it seems to be confusing to allow the pruning code to proceed since we expect [from_cp, to_cp)

Base automatically changed from indexer-alt-framework-cp-mapping to main December 23, 2024 21:04
@wlmyng wlmyng force-pushed the indexer-alt-cp-mapping-for-pruning branch from 4a54772 to 1b7b826 Compare December 23, 2024 21:30
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 23, 2024 21:30 — with GitHub Actions Inactive
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 23, 2024 22:37 — with GitHub Actions Inactive
@wlmyng wlmyng changed the title [indexer-alt] update indexer-alt-framework code's pruner to require cp_mapping [indexer-alt] expose mapping from cp to tx or epoch interval in indexer-alt-framework for pruners if needed Dec 23, 2024
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 23, 2024 22:51 — with GitHub Actions Inactive
@wlmyng wlmyng requested review from amnn and gegaowp December 23, 2024 22:56
@wlmyng wlmyng force-pushed the indexer-alt-cp-mapping-for-pruning branch from af5edf6 to 92c75e5 Compare December 26, 2024 16:59
@wlmyng wlmyng temporarily deployed to sui-typescript-aws-kms-test-env December 26, 2024 16:59 — with GitHub Actions Inactive
…o use the checkpoint, tx, or epoch_interval, all of which have an exclusive hi

address first set of comments. trait fn prune still accepts from and to checkpoints, up to handler prune to decide what it needs. doc comments for PrunableRange regarding what it expects on instantiation

extend PrunableRange::get_range to return an error if from_cp is not < to_cp. Output warning if it tries to fetch from cp_mapping and can't. and remove warning in framework/lib.rs around cp_mapping

split out models from handlers

simplify error story

expose intervals as functions instead of in a struct

anyhow error when from_cp not < to_cp
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants