WIP: copy indexer deposits data to scraper db #506

melisaguevara · 2024-12-31T20:00:20Z

Configuration steps to use both databases from the indexer can be found here: https://linear.app/uma/issue/ACX-3496/copy-indexer-deposits-data-to-scraper-db

linear · 2024-12-31T20:00:24Z

ACX-3496 Copy Indexer deposits data to Scraper DB

melisaguevara · 2024-12-31T20:04:20Z

docker-compose.yml

+networks:
+  test-network:
+    external: true
+


This network can be renamed.
I still have to check that both the indexer and scraper work fine using this network.
Indexer draft PR configuring the network: across-protocol/indexer#142

melisaguevara · 2024-12-31T20:04:47Z

docker-compose.yml

@@ -27,21 +33,26 @@ services:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
+    command: ["postgres", "-c", "log_statement=all"]


This is just to debug the auth issue I am having. Can be deleted after

melisaguevara · 2024-12-31T20:05:09Z

docker-compose.yml

    ports:
-      - 5432:5432
+      - 5433:5432


This is needed to avoid port collisions between indexer and scraper

melisaguevara · 2024-12-31T20:08:45Z

src/modules/rewards/services/opRebateServiceV2.ts

This is mainly a copy of the precious service but adapted to the new deposits and rewards table structures.

melisaguevara · 2024-12-31T20:09:19Z

src/modules/rewards/services/opRebateServiceV2.ts

+  public async getEarnedRewards(userAddress: string) {
+    userAddress = assertValidAddress(userAddress);
+
+    const baseQuery = this.buildBaseQuery(this.rewardRepository.createQueryBuilder("r"), userAddress);
+    const { opRewards } = await baseQuery
+      .select("SUM(CAST(r.amount as DECIMAL))", "opRewards")
+      .where("r.isClaimed = :isClaimed", { isClaimed: true })
+      .getRawOne<{ opRewards: string }>();
+
+    return opRewards;
+  }
+
+  public async getOpRebatesSummary(userAddress: string) {
+    userAddress = assertValidAddress(userAddress);
+
+    const baseQuery = this.buildBaseQuery(this.rewardRepository.createQueryBuilder("r"), userAddress);
+    baseQuery.andWhere("r.isClaimed = :isClaimed", { isClaimed: false });
+    const [{ depositsCount }, { unclaimedRewards }, { volumeUsd }] = await Promise.all([
+      baseQuery.select("COUNT(DISTINCT r.depositPrimaryKey)", "depositsCount").getRawOne<{
+        depositsCount: string;
+      }>(),
+      baseQuery.select("SUM(CAST(r.amount as DECIMAL))", "unclaimedRewards").getRawOne<{
+        unclaimedRewards: number;
+      }>(),
+      baseQuery
+        .leftJoinAndSelect("r.deposit", "d")
+        .leftJoinAndSelect("d.token", "t")
+        .leftJoinAndSelect("d.price", "p")
+        .select(`COALESCE(SUM(d.amount / power(10, t.decimals) * p.usd), 0)`, "volumeUsd")
+        .getRawOne<{
+          volumeUsd: number;
+        }>(),
+      // TODO: add claimable rewards
+    ]);
+
+    return {
+      depositsCount: parseInt(depositsCount),
+      unclaimedRewards,
+      volumeUsd,
+      claimableRewards: "0",
+    };
+  }


I haven't tested this functions still work as expected.

melisaguevara · 2024-12-31T20:10:40Z

src/modules/rewards/services/opRebateServiceV2.ts

+  // public async getOpRebateRewards(query: GetRewardsQuery) {
+  //   const limit = parseInt(query.limit ?? "10");
+  //   const offset = parseInt(query.offset ?? "0");
+  //   const userAddress = assertValidAddress(query.userAddress);
+
+  //   const baseQuery = this.buildBaseQuery(this.rewardRepository.createQueryBuilder("r"), userAddress);
+
+  //   const rewardsQuery = baseQuery.orderBy("r.depositDate", "DESC").limit(limit).offset(offset);
+  //   const [rewards, total] = await rewardsQuery.getManyAndCount();
+
+  //   // JOIN instead of query deposits separately 
+  //   const depositPrimaryKeys = rewards.map((reward) => reward.depositPrimaryKey);
+  //   const deposits = await this.depositRepository.find({
+  //     where: { id: In(depositPrimaryKeys) },
+  //   });
+
+  //   return {
+  //     rewards: rewards.map((reward) => ({
+  //       ...reward,
+  //       deposit: deposits.find((deposit) => deposit.id === reward.depositPrimaryKey),
+  //     })),
+  //     pagination: {
+  //       limit,
+  //       offset,
+  //       total,
+  //     },
+  //   };
+  // }


This could be rewritten to use JOIN instead of separate queries now that the FK between rewards and deposits is a composite key (depositId, originChainId).

melisaguevara · 2024-12-31T20:11:30Z

src/modules/rewards/model/OpRewardV2.entity.ts

+  @OneToOne(() => RewardedDeposit)
+  @JoinColumn([
+    { name: "depositId", referencedColumnName: "depositId", foreignKeyConstraintName: "FK_op_reward_deposit" },
+    { name: "originChainId", referencedColumnName: "originChainId", foreignKeyConstraintName: "FK_op_reward_deposit" },
+  ])
+  deposit: RewardedDeposit;


This relation used to be ManyToOne, but I think it's ok to set it to OneToOne.

melisaguevara · 2024-12-31T20:12:14Z

src/modules/rewards/services/opRebateServiceV2.ts

+    // We use the `from` address of the deposit transaction as the reward receiver
+    // to also take into accounts deposits routed through the SpokePoolVerifier contract.
+    const provider = this.ethProvidersService.getProvider(deposit.originChainId);
+    const depositTransaction = await provider.getTransaction(deposit.depositTxHash); // This could also be part of the indexer data


Maybe we could fetch this in the indexer as we are already retrieving the transaction.

melisaguevara · 2024-12-31T20:13:49Z

src/modules/rewards/services/opRebateServiceV2.ts

+    const inputToken = await this.ethProvidersService.getCachedToken(deposit.originChainId, deposit.inputToken);
+    const outputToken = await this.ethProvidersService.getCachedToken(deposit.originChainId, deposit.outputToken);


As we no longer have the token relation, we have to get token details using the addresses.
This is erroring locally for me with some chains though. It fails with:
Error: call revert exception [ See: https://links.ethers.org/v5-errors-CALL_EXCEPTION ] (method="symbol()", data="0x", errorArgs=null, errorName=null, errorSignature=null, reason=null, code=CALL_EXCEPTION, version=abi/5.7.0)

melisaguevara · 2024-12-31T20:14:26Z

src/modules/rewards/services/opRebateServiceV2.ts

+    const inputToken = await this.ethProvidersService.getCachedToken(deposit.originChainId, deposit.inputToken);
+    const outputToken = await this.ethProvidersService.getCachedToken(deposit.originChainId, deposit.outputToken);
+
+    const inputTokenPrice = await this.getInputTokenPrice(deposit, inputToken, outputToken); // This could also be part of the indexer data


This could also be part of the indexer data, but for now I'm fetching it here too.
This is needed to calculate the MAX reward amount for each deposit.

melisaguevara · 2024-12-31T20:15:14Z

src/modules/scraper/service/CopyIndexerDepositsCron.ts

+          WHERE
+            indexer_rhi."status" = 'filled'
+            AND indexer_deposit_events.finalised is true
+            AND indexer_fills.finalised is true
+            AND indexer_rhi."destinationChainId" in (${eligibleDestinationChains})
+            LIMIT 5000


We could also add a condition here to fetch only deposits starting from 2025.

melisaguevara · 2024-12-31T20:15:47Z

src/modules/scraper/service/CopyIndexerDepositsCron.ts

+    performance.mark("query-start");
+    const newFilledDeposits = await this.dataSource.query(newFilledDepositsQuery) as RewardedDeposit[];
+    performance.mark("query-end");
+    performance.measure("query", "query-start", "query-end");


performance marks can be deleted before merging

melisaguevara added 2 commits December 30, 2024 17:54

draft: cronjob for copying indexer deposits into scraper

31d8c71

v2 op rewards proccesor

aef7d52

melisaguevara commented Dec 31, 2024

View reviewed changes

docker-compose.yml

ports:

- 5432:5432

- 5433:5432

Copy link

Author

melisaguevara Dec 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is needed to avoid port collisions between indexer and scraper

melisaguevara commented Dec 31, 2024

View reviewed changes

use all deposits when inserting

d1e5469

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: copy indexer deposits data to scraper db #506

WIP: copy indexer deposits data to scraper db #506

melisaguevara commented Dec 31, 2024 •

edited

Loading

linear bot commented Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024 •

edited

Loading

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

melisaguevara Dec 31, 2024

		const inputToken = await this.ethProvidersService.getCachedToken(deposit.originChainId, deposit.inputToken);
		const outputToken = await this.ethProvidersService.getCachedToken(deposit.originChainId, deposit.outputToken);

WIP: copy indexer deposits data to scraper db #506

Are you sure you want to change the base?

WIP: copy indexer deposits data to scraper db #506

Conversation

melisaguevara commented Dec 31, 2024 • edited Loading

linear bot commented Dec 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

melisaguevara Dec 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

melisaguevara commented Dec 31, 2024 •

edited

Loading

melisaguevara Dec 31, 2024 •

edited

Loading