feat(indexer): add coingecko price collection from tokens against usd #138

daywiss · 2024-12-18T19:51:00Z

motivation

we want to start collecting price data in the indexer

changes

This dynamically looks up prices based on fill events, passing them to a worker to then fetch and/or cache the price at the deposit time

linear · 2024-12-18T19:51:04Z

ACX-3505 Identify tokens to start indexing prices

amateima · 2024-12-19T09:14:52Z

...ges/indexer-database/packages/indexer-database/src/migrations/1733941169940-HistoricPrice.ts

+import { MigrationInterface, QueryRunner } from "typeorm";
+
+export class HistoricPrice1733941169940 implements MigrationInterface {
+    name = 'HistoricPrice1733941169940'
+
+    public async up(queryRunner: QueryRunner): Promise<void> {
+        await queryRunner.query(`CREATE TABLE "historic_market_price" ("id" SERIAL NOT NULL, "baseCurrency" character varying NOT NULL, "quoteCurrency" character varying NOT NULL DEFAULT 'usd', "date" date NOT NULL, "price" numeric NOT NULL, "createdAt" TIMESTAMP NOT NULL DEFAULT now(), CONSTRAINT "UK_historic_price_baseCurrency_quoteCurrency_date" UNIQUE ("baseCurrency", "quoteCurrency", "date"), CONSTRAINT "PK_b0a22436b47e742187aa7408561" PRIMARY KEY ("id"))`);
+    }
+
+    public async down(queryRunner: QueryRunner): Promise<void> {
+        await queryRunner.query(`DROP TABLE "historic_market_price"`);
+    }
+
+}


Not sure if these 2 migration files are different, but I guess one of them is not needed. The second migration will fail because the table will already be created by the first one

yea this was a mistake. wil fix

amateima · 2024-12-19T09:17:47Z

packages/indexer-database/src/entities/HistoricPrice.ts

+  @Column()
+  date: string;


This should definitely be a typed Date property and the DB column should be date. This way we can take advantage of special queries/functionalities that SQL provides for such column type.

i think the complexity is that we only care about 1 price per day, which means deterministically calculating the "day" based on a js Date or timestamp becomes more involved. Ill add the logic and u can see if thats what you want

amateima · 2024-12-19T09:33:09Z

packages/indexer/src/services/priceProcessor.ts

+          return;
+        }
+        await historicPriceRepository.insert({
+          date: dbFormattedDate,


I propose to not leak here the weird way that CoinGecko handles date format. Here we should pass a simple date instance as value

amateima · 2024-12-19T09:37:20Z

packages/indexer/src/services/priceProcessor.ts

+        }
+        await historicPriceRepository.insert({
+          date: dbFormattedDate,
+          baseCurrency: symbol,


In the past I experienced inconsistencies with the symbols returned by CoinGecko API, for example different letter capitalisation for the same token, so I propose to do a toLowerCase() on the symbol

this is currently an enum, so if anything is wrong we can fix it in the enum

amateima · 2024-12-19T10:11:52Z

packages/indexer/src/parseEnv.ts

@@ -193,6 +196,9 @@ export function envToConfig(env: Env): Config {
    enabledWebhookRequestWorkers: true,
    clients: parseWebhookClientsFromString(env.WEBHOOK_CLIENTS ?? "[]"),
  };
+  const coingeckoSymbols = parseArray(env.COINGECKO_SYMBOLS).map((symbol) =>


We should document this ENV value in the README and specify the format of it (list, object, etc)

amateima · 2024-12-30T12:48:49Z

packages/error-handling/src/utils/assert.ts

  try {
-    return assertModule.ok(value, message);
+    return assertModule.ok(value !== null && value !== undefined, message);


What was the edge case that was not handled gracefully by the current implementation?

i probably dont need to do this

amateima · 2024-12-30T13:05:31Z