Skip to content

Commit

Permalink
fix(indexer-alt): overly chatty loud watermark updates
Browse files Browse the repository at this point in the history
## Description

The loud watermark update logic always bumped the previous update
watermark by the same interval. When the indexer was running well ahead
of the loud watermark update rate, this would cause many updates to be
issued successively.

This change makes it so that the next update is always
`LOUD_WATERMARK_UPDATE_INTERVAL` away from the last loud update.

## Test plan

Run the indexer. Previously updates -- particularly from summary tables
that gathered changes up and write them out in big batches -- would come
in bursts, and this behaviour is no longer apparent after the change:

```
sui$ cargo run -p sui-indexer --release --                                       \
  --database-url "postgres://postgres:postgrespw@localhost:5432/sui_indexer_alt" \
  indexer --remote-store-url https://checkpoints.mainnet.sui.io                  \
  --last-checkpoint 1000000 --consistent-range 3600
```
  • Loading branch information
amnn committed Nov 13, 2024
1 parent 18e6aee commit d0c7294
Show file tree
Hide file tree
Showing 2 changed files with 4 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,8 @@ pub(super) fn commit_watermark<H: Handler + 'static>(
}

if watermark.checkpoint_hi_inclusive > next_loud_watermark_update {
next_loud_watermark_update += LOUD_WATERMARK_UPDATE_INTERVAL;
next_loud_watermark_update = watermark.checkpoint_hi_inclusive + LOUD_WATERMARK_UPDATE_INTERVAL;

info!(
pipeline = H::NAME,
epoch = watermark.epoch_hi_inclusive,
Expand Down
3 changes: 2 additions & 1 deletion crates/sui-indexer-alt/src/pipeline/sequential/committer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,8 @@ pub(super) fn committer<H: Handler + 'static>(
.set(watermark.timestamp_ms_hi_inclusive);

if watermark.checkpoint_hi_inclusive > next_loud_watermark_update {
next_loud_watermark_update += LOUD_WATERMARK_UPDATE_INTERVAL;
next_loud_watermark_update = watermark.checkpoint_hi_inclusive + LOUD_WATERMARK_UPDATE_INTERVAL;

info!(
pipeline = H::NAME,
epoch = watermark.epoch_hi_inclusive,
Expand Down

0 comments on commit d0c7294

Please sign in to comment.