OP Refactor Tokens & Housekeeping (#2207)

* fix perp contract

* update token generations & move files

* mod aave sources

* update erc20 schema

* mod atoken sources

* fix indent

* aave sources alignment

* update aave schema again

* granary sources update

* more schema updates

* update ovm_optimism schema

* mod aave schema

* whoops - fix commas

* fix commas

* take - out of descriptions

* fixes

* expose OP erc20s as a separate detailed list

* map op back to erc20 tokens

* add standard to nft for op

* update dbt project

* add sided

* Update contracts_optimism_contract_creator_address_list.sql

* small

* typo

* Update tokens_optimism_schema.yml

* adds

* add dependency on snapshot

* Fix dbt_project.yml

* Update tokens_optimism_nft.sql

* remove generated tokens for now

* initialize without circular dependencies

* update prices

* update btc.b

* forget snapshot

* remove deps

* aave chain insert

* fix aave for loop

* fix atokens select

* simplify

* alias

* fix gran

* fix gran

note - granary initialized events are missing

* comment out descriptor

* erc20 schema

* fix

* Update tokens_optimism_erc20.sql

* price dupe

* update lists

* nft list

* fix unique nft

* add attestation override

* add mm swaps

* Update tokens_optimism_nft.sql

* add gysr

* add nft

* add erc20

* quix <> rh contract

* stick

* add type on atokens

* add woofi

* adds

* Reformat aave_v3_tokens for compatibility

* Fix typo

* Add uniqueness test

* Remove schema in config block

* Reformat granary_tokens and compatibility for Trino

* add token to prices

* adds

* add creator and update nft name

* revert OaycNFT renaming

* Apply CAST() to number_of_items and amount_raw

* fix tofu_bnb schema

* Add CAST() to fields

* fix fee convention

* Remove hardcoded fields from nft_trades

* convert to hours (#2399)

* [Easy] [Label] Implement trader platforms label (#2394)

* Implemented trader platforms label

* Fixed errors in spell

* Added forgotten group by

* Fixed columns selection

* Added ordering and keeping of order for label

* Fixed error in brackets

* Added missing block_time column

* Added time to keep order

* [easy] new tokens (#2414)

* Adding Aave V3 assets to tokens_avalanche_c_erc20.sql and tokens_polygon_erc20.sql (#2410)

* add AAVE, EURS to arbitrum_erc20.sql

* adding remaining aave v3 assets to erc20

* [CoW Protocol] Add Referral Mapping (#2377)

* add referral table

* remove duplicate anchor

* remove unused subquery

* Add Pancakeswap NFT Events (#2255)

* add seed in yml file

* Add Pancakeswap NFT V1 to nft trades

* delete tofu

* Adjust

* update

* update LOGIC

* initial commit

* update

* Update models/pancakeswap/bnb/pancakeswap_bnb_nft_market_v1_event.sql

Co-authored-by: 0xRob <[email protected]>

* Update models/pancakeswap/bnb/pancakeswap_bnb_nft_market_v1_event.sql

Co-authored-by: 0xRob <[email protected]>

* add test

* add

* Update models/pancakeswap/bnb/pancakeswap_bnb_nft_market_v1_event.sql

Co-authored-by: 0xRob <[email protected]>

* delete liquidifty

* add test file

* .

* update

* fix unique ID

* update sample file

* change pancakeswap seed schema

* temporary comment trade section

* reinstate test and correct columns

* change sample file

* change seed file

* add pancakeswap to nft trades

* add royalty info

* fix currency_symbol

* guard fee logic

* cast null as string

* remove double column

Co-authored-by: 0xRob <[email protected]>
Co-authored-by: 0xRob <[email protected]>

* Migrate Balancer `vebal_slopes` abstraction (#2179)

* Add balancer_vebal_slopes to schema yml file

* Add veBAL related sources to sources yml file

* Add balancer_vebal_slopes spell

* Reorder sources

* Change casting syntax

* Rename datetime field from schema .yml file

* Add incremental logic to vebal_slopes spell

* Update models/balancer/ethereum/balancer_vebal_slopes.sql

Co-authored-by: Huang Geyang <[email protected]>

* Update models/balancer/ethereum/balancer_vebal_slopes.sql

Co-authored-by: Huang Geyang <[email protected]>

* Update models/balancer/ethereum/balancer_ethereum_schema.yml

Co-authored-by: Huang Geyang <[email protected]>

* Update models/balancer/ethereum/balancer_vebal_slopes.sql

Co-authored-by: Huang Geyang <[email protected]>

* Replace block_time unique key for block_number

* Update models/balancer/ethereum/balancer_ethereum_schema.yml

Co-authored-by: Huang Geyang <[email protected]>

* Update models/balancer/ethereum/balancer_vebal_slopes.sql

Co-authored-by: Huang Geyang <[email protected]>

* Replace block_date for block_number in uniqueness test

* Rewrite group by in full column names

* Fix unmatched columns in schema.yml

* add descriptions to vebal_slopes columns

Co-authored-by: Huang Geyang <[email protected]>
Co-authored-by: Fábio Mendes <[email protected]>

* Update cow_protocol_ethereum_referrals.sql (#2423)

* Fix Source Chain ID/Name Hop Optimism (#2313)

* Update hop_protocol_flows.sql

* Revert "Update hop_protocol_flows.sql"

This reverts commit be30148.

* Update hop_protocol_optimism_flows.sql

* Add BSC Dodo trades to dex.trades (#2157)

* dodo-bnb-trades-real

* add seed tests

* seed value typo

* remove depracated test

* Modify dodo_bnb to aligh with dodo_eth in project.yml

* combine cross-chain bnb & ethereum trades

* correct ethereum dodo trades

* downgrade 'dbt_utils'

* Change back dbt_utils version to 0.9.2

* final tweaks

* force failure

* Remove extra line

* correct seed file

* restore deleted prices

* minor formatting

* cleanup dbt_project with latest standard

* add freshness check

Co-authored-by: Huang Geyang <[email protected]>
Co-authored-by: jeff-dude <[email protected]>

* script to find sources missing checks (#2401)

* script to find sources missing checks + new freshness checks

* T-22136: Missing ERC20 tokens (#2425)

* add A LOT of missing tokens

* sort + indentation

* escape backtick

* add missing comma

* remove last comma

* remove duplicate

* add trim to symbol

* remove trim, see if test pass

* fixed trim

Co-authored-by: jeff-dude <[email protected]>

* Update pull_request_template.md (#2429)

* Update README.md (#2411)

* T-23065 - optimism transfers: update value filter for string datatype (#2426)

* update value filter for string datatype

* directory cleanup

* Seaport ethereum patch (#2419)

* unifying seaport source

* keep unique_trade_id for opensea.trades for nft.trades compatability

* merge schema file

* cast amount to decimal(38,0)

* add test seeds

* update schema.yml

* Fix: minor typos (#2424)

* Fix: minor typos

Fix: minor typos

* Fix: minor typo

Fix: minor typo

* Fix: minor typo

Fix: minor typo

* Fix: minor typo

Fix: minor typo

* [Quick] Updates to metadata files (#2421)

* add new metadata

* fix duplicate error

Co-authored-by: cat <[email protected]>

* Added alphasharks.io as an aggregator (#2417)

* Fix aave liquidation call (#2432)

* FIX: cast to varchar in opensea_v3 trades (#2433)

* cast to varchar in concat in view

* define varchar length'

* use batch api calls for price checker (#2431)

* use batch api calls for price checker

* Update dodo_ethereum_trades & add dodo_bnb_trades & split them to aggregator and pools (#2169)

* update dodo_ethereum_trades && add dodo_bnb_trades

* split dodo_trades to pools & aggregator on Ethereum and BNBChain

* label different dodoV2 pools

* fix schema & sql

* add dodo_proxies on Ethereum

* Modify dodo_bnb_trades to incremental view

* Apply check_dex_seed and modify dodo_eth_trades to incremental view

* Add DODO bnb trade seeds

* Apply check_dex_seed to DODO bnb trades model

* update dex_trade_seeds

* fix

* Add DODO bnb in project.yml

* remove gitkeep in dodo

* Update models/dodo/bnb/dodo_pools_bnb_trades.sql

* Update check_dex_seed in dodo_ethereum_schema.yml

* fix dodo_ethereum_schema

* Remove schema in the config blocks

* Add dodo_trades files by scoffie

* add dppOracle & dppAdvanced to dodo_bnb

* fix

* update dbt_project on dodo

* fix dodo_bnb_sources.yml

* move cast to base model

* update dex.trades seeds on DODO_BNBChain

* final formatting & cleanup expose spell macro

Co-authored-by: Huang Geyang <[email protected]>
Co-authored-by: jeff-dude <[email protected]>

* [CoW] Fix Fee USD Calculation (#2422)

* use most natural form of fee computation when possible

* add comment about imprecision

* Fix failing tests (#2396)

* trigger test

* enable test in CI

* defer in test

* fix aragon

* fix native token test

* fix native mints unique id

* fix nft trader labels

* union all in nft labels

* union all for performance

* trigger wash_trades full refresh

* fix cow protocol distinct labels

* revert null wallet_address fix

Co-authored-by: soispoke <[email protected]>

* Add ETH Transfers to L1 (#2440)

* add eth transfers and small type fixes in op eth transfers

* comment update

* update schema

* update transfer id

* add tx to and from

* add tx_to and tx_from

* nit remove spaces

Co-authored-by: soispoke <[email protected]>

* Token checker tweaks (#2437)

* allow for case insensitive token symbol matching

* adapt token checker error/warns

* check for warning in test instead of error

* Allow for local compile of incremental models.  (#2403)

* base is_incremental

* add force-incremental variable

* clean up dbt_project values

* add version schema name

* rename ovm directory

* naming standards

* keep optimism erc20 view name

Co-authored-by: Huang Geyang <[email protected]>
Co-authored-by: 0xRob <[email protected]>
Co-authored-by: Belén Pruvost <[email protected]>
Co-authored-by: Meghan Heintz <[email protected]>
Co-authored-by: Gent Rexha <[email protected]>
Co-authored-by: josojo <[email protected]>
Co-authored-by: mc <[email protected]>
Co-authored-by: Benjamin Smith <[email protected]>
Co-authored-by: Harry <[email protected]>
Co-authored-by: 0xRob <[email protected]>
Co-authored-by: Victor Stefenon <[email protected]>
Co-authored-by: Fábio Mendes <[email protected]>
Co-authored-by: soispoke <[email protected]>
Co-authored-by: Levis Kithinji <[email protected]>
Co-authored-by: jeff-dude <[email protected]>
Co-authored-by: jeff-dude <[email protected]>
Co-authored-by: sohwak <[email protected]>
Co-authored-by: omahs <[email protected]>
Co-authored-by: catherine-2 <[email protected]>
Co-authored-by: cat <[email protected]>
Co-authored-by: Garv Maggu <[email protected]>
Co-authored-by: Dias <[email protected]>
Co-authored-by: owen05 <[email protected]>

dbt_project.yml

@@ -566,6 +566,11 @@ models:
     bridge:
       +schema: bridge
 
+    the_granary:
+      +schema: the_granary
+      optimism:
+        +schema: the_granary_optimism
+
     tessera:
       +schema: tessera
       ethereum:
@@ -884,7 +889,6 @@ seeds:
             swap_flag: boolean
             matcha_limit_order_flag: boolean
             volume_usd: double
-
     gmx:
       avalanche_c:
         gmx_glp_aum_seed:

models/aave/aave_schema.yml

@@ -0,0 +1,50 @@
+version: 2
+
+models:
+  - name: aave_v3_tokens
+
+    meta:
+      blockchain: optimism, ethereum, polygon, avalanche_c, arbitrum
+      sector: lending
+      project: aave
+      contributors: msilb7
+
+    config:
+      tags: ['cross-chain', 'aave', 'aave_v3','tokens','lending', 'borrowing']
+    description: "Aave aToken list and mappings to underlying tokens"
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - blockchain
+            - atoken_address
+    columns:
+    - &blockchain
+      name: blockchain
+      description: "Blockchain for the token"
+    - &atoken_address
+      name: atoken_address
+      description: "Address for the atoken"
+    - &underlying_address
+      name: underlying_address
+      description: "Underlying ERC20 token that that the aToken represents"
+    - &atoken_decimals
+      name: atoken_decimals
+      description: "Decimals for the atoken"
+    - &side
+      name: side
+      description: "Supply side or Borrow side for the atoken"
+    - &atoken_rate_type
+      name: arate_type
+      description: "Variable or Stable rate for the atoken"
+    - &atoken_symbol
+      name: atoken_symbol
+      description: "Symbol for the atoken"
+    - &atoken_name
+      name: atoken_name
+      description: "Name for the atoken"
+    - &underlying_decimals
+      name: underlying_decimals
+      description: "Decimals for the underlying token"
+    - &underlying_symbol
+      name: underlying_symbol
+      description: "Symbol for the underlying token"

models/aave/aave_sources.yml

@@ -0,0 +1,244 @@
+version: 2
+
+sources:
+  - name: aave_v3_optimism
+    description: "Decoded contracts for Aave on Optimism"
+
+    tables:
+    - name: AToken_evt_Initialized
+      description: "aToken initialization events, supply tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - &contract_address
+          name: contract_address
+          description: "Aave token contract address"       
+        - &evt_block_number
+          name: evt_block_number
+          description: "Event block number"  
+        - &evt_block_time
+          name: evt_block_time
+          description: "Timestamp for block event time in UTC"
+        - &evt_index
+          name: evt_index
+          description: "Event index"  
+        - &evt_tx_hash
+          name: evt_tx_hash
+          description: "Transaction hash of the event" 
+        - &underlyingAsset
+          name: underlyingAsset
+          description: "Underlying ERC20 token that that the aToken represents"
+        - &aTokenDecimals
+          name: aTokenDecimals
+          description: "Decimals of the aToken"
+        - &aTokenSymbol
+          name: aTokenSymbol
+          description: "Symbol of the aToken"
+        - &aTokenName
+          name: aTokenName
+          description: "Name of the aToken"
+
+    - name: StableDebtToken_evt_Initialized
+      description: "aToken initialization events, stable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - &debtTokenDecimals
+          name: debtTokenDecimals
+          description: "Decimals of the aToken"
+        - &debtTokenSymbol
+          name: debtTokenSymbol
+          description: "Symbol of the aToken"
+        - &debtTokenName
+          name: debtTokenName
+          description: "Name of the aToken" 
+
+    - name: VariableDebtToken_evt_Initialized
+      description: "aToken initialization events, variable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *debtTokenDecimals
+        - *debtTokenSymbol
+        - *debtTokenName
+
+  - name: aave_v3_arbitrum
+    description: "Decoded contracts for Aave on Arbitrum"
+
+    tables:
+    - name: AToken_evt_Initialized
+      description: "aToken initialization events, supply tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address      
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index 
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *aTokenDecimals
+        - *aTokenSymbol
+        - *aTokenName
+
+    - name: StableDebtToken_evt_Initialized
+      description: "aToken initialization events, stable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *debtTokenDecimals
+        - *debtTokenSymbol
+        - *debtTokenName
+
+    - name: VariableDebtToken_evt_Initialized
+      description: "aToken initialization events, variable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *debtTokenDecimals
+        - *debtTokenSymbol
+        - *debtTokenName
+
+  - name: aave_v3_polygon
+    description: "Decoded contracts for Aave on Polygon"
+    tables:
+    - name: AToken_evt_Initialized
+      description: "aToken initialization events, supply tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address      
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index 
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *aTokenDecimals
+        - *aTokenSymbol
+        - *aTokenName
+
+    - name: StableDebtToken_evt_Initialized
+      description: "aToken initialization events, stable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *debtTokenDecimals
+        - *debtTokenSymbol
+        - *debtTokenName
+
+    - name: VariableDebtToken_evt_Initialized
+      description: "aToken initialization events, variable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *debtTokenDecimals
+        - *debtTokenSymbol
+        - *debtTokenName
+
+  - name: aave_v3_avalanche_c
+    description: "Decoded contracts for Aave on Avalanche"
+    tables:
+    - name: AToken_evt_Initialized
+      description: "aToken initialization events, supply tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address      
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index 
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *aTokenDecimals
+        - *aTokenSymbol
+        - *aTokenName
+
+    - name: StableDebtToken_evt_Initialized
+      description: "aToken initialization events, stable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *debtTokenDecimals
+        - *debtTokenSymbol
+        - *debtTokenName
+
+    - name: VariableDebtToken_evt_Initialized
+      description: "aToken initialization events, variable debt tokens."
+      freshness:
+        warn_after: { count: 12, period: hour }
+        error_after: { count: 24, period: hour }
+      loaded_at_field: evt_block_time
+      columns:
+        - *contract_address  
+        - *evt_block_number
+        - *evt_block_time
+        - *evt_index
+        - *evt_tx_hash
+        - *underlyingAsset
+        - *debtTokenDecimals
+        - *debtTokenSymbol
+        - *debtTokenName