-
Notifications
You must be signed in to change notification settings - Fork 11.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[indexer] align watermarks table schema in live indexer to alt indexer (
#19908) ## Description Since watermarks table isn't being written to yet, modify the db schema to match alt-indexer. The changes are to rename entity -> pipeline, tx_hi_inclusive -> tx_hi, and pruner_hi_inclusive -> pruner_hi and make it a non-null column. This works out nicely for graphql, since the transactions query implementations expect a half-open interval. Also simplifies pruner logic, since it can write the `reader_lo` as `pruner_hi` after delay, and table pruners will delete between `[table_data, pruner_hi)`. ## Test plan How did you test the new or updated feature? --- ## Release notes Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required. For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates. - [ ] Protocol: - [ ] Nodes (Validators and Full nodes): - [ ] Indexer: - [ ] JSON-RPC: - [ ] GraphQL: - [ ] CLI: - [ ] Rust SDK: - [ ] REST API:
- Loading branch information
Showing
7 changed files
with
68 additions
and
65 deletions.
There are no files selected for viewing
51 changes: 28 additions & 23 deletions
51
crates/sui-indexer/migrations/pg/2024-09-12-213234_watermarks/up.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,29 +1,34 @@ | ||
CREATE TABLE watermarks | ||
CREATE TABLE IF NOT EXISTS watermarks | ||
( | ||
-- The table governed by this watermark, i.e `epochs`, `checkpoints`, `transactions`. | ||
entity TEXT NOT NULL, | ||
-- Inclusive upper epoch bound for this entity's data. Committer updates this field. Pruner uses | ||
-- this to determine if pruning is necessary based on the retention policy. | ||
-- The pipeline governed by this watermark, i.e `epochs`, `checkpoints`, | ||
-- `transactions`. | ||
pipeline TEXT PRIMARY KEY, | ||
-- Inclusive upper epoch bound for this entity's data. Committer updates | ||
-- this field. Pruner uses this to determine if pruning is necessary based | ||
-- on the retention policy. | ||
epoch_hi_inclusive BIGINT NOT NULL, | ||
-- Inclusive lower epoch bound for this entity's data. Pruner updates this field when the epoch range exceeds the retention policy. | ||
epoch_lo BIGINT NOT NULL, | ||
-- Inclusive upper checkpoint bound for this entity's data. Committer updates this field. All | ||
-- data of this entity in the checkpoint must be persisted before advancing this watermark. The | ||
-- committer refers to this on disaster recovery to resume writing. | ||
-- Inclusive upper checkpoint bound for this entity's data. Committer | ||
-- updates this field. All data of this entity in the checkpoint must be | ||
-- persisted before advancing this watermark. The committer refers to this | ||
-- on disaster recovery to resume writing. | ||
checkpoint_hi_inclusive BIGINT NOT NULL, | ||
-- Inclusive upper transaction sequence number bound for this entity's data. Committer updates | ||
-- this field. | ||
tx_hi_inclusive BIGINT NOT NULL, | ||
-- Inclusive low watermark that the pruner advances. Corresponds to the epoch id, checkpoint | ||
-- sequence number, or tx sequence number depending on the entity. Data before this watermark is | ||
-- considered pruned by a reader. The underlying data may still exist in the db instance. | ||
-- Exclusive upper transaction sequence number bound for this entity's | ||
-- data. Committer updates this field. | ||
tx_hi BIGINT NOT NULL, | ||
-- Inclusive lower epoch bound for this entity's data. Pruner updates this | ||
-- field when the epoch range exceeds the retention policy. | ||
epoch_lo BIGINT NOT NULL, | ||
-- Inclusive low watermark that the pruner advances. Corresponds to the | ||
-- epoch id, checkpoint sequence number, or tx sequence number depending on | ||
-- the entity. Data before this watermark is considered pruned by a reader. | ||
-- The underlying data may still exist in the db instance. | ||
reader_lo BIGINT NOT NULL, | ||
-- Updated using the database's current timestamp when the pruner sees that some data needs to | ||
-- be dropped. The pruner uses this column to determine whether to prune or wait long enough | ||
-- that all in-flight reads complete or timeout before it acts on an updated watermark. | ||
-- Updated using the database's current timestamp when the pruner sees that | ||
-- some data needs to be dropped. The pruner uses this column to determine | ||
-- whether to prune or wait long enough that all in-flight reads complete | ||
-- or timeout before it acts on an updated watermark. | ||
timestamp_ms BIGINT NOT NULL, | ||
-- Column used by the pruner to track its true progress. Data at and below this watermark can | ||
-- be immediately pruned. | ||
pruner_hi_inclusive BIGINT, | ||
PRIMARY KEY (entity) | ||
-- Column used by the pruner to track its true progress. Data below this | ||
-- watermark can be immediately pruned. | ||
pruner_hi BIGINT NOT NULL | ||
); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters