Add dex CI testing pipeline for base_trades modifications #7429

Hosuke · 2025-01-07T12:17:55Z

Previously, our base_trades testing workflow only validated the basic structure of trades but did not check for reasonable USD amounts. This could lead to issues where trades with unreasonable amounts (e.g., >$1B) would only be caught after deployment.

This PR adds USD amount validation while significantly reducing computational overhead:

Test Framework Enhancement:
- Add dex_ci_trades_test view that enriches base_trades with token information
- Add data test to validate amount_usd:
```
- dbt_utils.accepted_range:
    max_value: 1000000000  # Catch trades > $1B
```
Performance Optimization:
- Directly enrich modified base_trades with token information, skipping intermediate view materializations:
```
-- Direct enrichment without building platform_trades -> trades -> trades_enrich
SELECT * FROM (
    {{ enrich_dex_trades(base_trades = 'base_trades', ...) }}
)
```
- Only process previous day's data for price availability
- This dramatically reduces computation by avoiding the full dex.trades model hierarchy
Workflow Improvements:
- Simplify base_trades file detection logic
- Always compare against main branch for consistency

The changes not only catch potential USD amount issues early but also significantly improve CI performance by avoiding the overhead of building multiple intermediate views in the dex.trades model hierarchy.

- Add dynamic transaction columns macro for multi-chain support - Create CI test view to validate base_trades changes - Add comprehensive schema and tests matching dex.trades - Optimize workflow to efficiently test base_trades modifications This change improves the CI process by: 1. Only testing modified base_trades 2. Processing one day of data for quick validation 3. Using the same data quality checks as production 4. Preventing full dex build until tests pass

- Remove unnecessary transaction columns handling - Simplify base_trades selection - Keep core functionality for amount_usd validation

…chema - Remove YAML anchor references - Add complete column definitions - Keep all tests and descriptions intact

- Add get_modified_base_trades macro to read changed files from env - Update test view to automatically include modified models - Enhance workflow to pass modified files information This change enables fully automated testing of modified base_trades models without requiring manual updates to the test view.

- Replace modules.os.environ with dbt var() function - Update workflow to pass modified files via --vars - Keep the same functionality but use dbt-native approach

- Remove transaction-related columns since we no longer join with transactions table - Keep only columns that come from base_trades and token enrichment

- Add fetch-depth: 0 to get full git history - Use git fetch to get target branch - Compare target branch with current branch in PR environment - Compare with previous commit in non-PR environment

- Use source() instead of ref() to reuse existing base_trades views - Add source definitions for base_trades tables - Fix working directory in workflow - Remove project-dir parameter This change improves performance by avoiding recomputation of base_trades models during CI testing.

- Move test-base-trades step into dbt_run.yml workflow - Add base_trades_changes and modified_base_trades inputs - Run base_trades tests after initial models to reuse computed views This change improves performance by running tests after base_trades models are already computed.

… conflict - Wrap macro call in SELECT * FROM to avoid WITH clause conflict - Keep base_union CTE separate from macro's CTEs

- Get git_schema from GIT_SHA environment variable - Reference base_trades tables from temporary schema - This ensures we use the tables created in the current CI run

- Replace target.database with test_schema - This matches where the base_trades tables are actually created

- Move base_trades check from dex.yml into dbt_run.yml - Remove inputs for base_trades changes and modified files - Use step outputs directly in dbt_run.yml - Simplify dex.yml by removing check-base-trades job

- Use delta_prod catalog - Combine git_schema and model_name in schema path - This matches how dbt creates temporary tables

- Add mock fields for testing only - Use evt_tx_hash as a placeholder - Keep production code unchanged

- Use cast(null as varbinary) for tx_from and tx_to - These fields are only for testing and not used in production - Keep production code unchanged