Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

needs to use {{this}} format for incremental #3341

Closed
wants to merge 18 commits into from
Closed

needs to use {{this}} format for incremental #3341

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
dffb01f
needs to use {{this}} format for incremental
andrewhong5297 May 14, 2023
bebf854
should always be next day only
andrewhong5297 May 14, 2023
3945b01
Add uniqueness test
Hosuke May 15, 2023
7afffa3
keep block_slot
andrewhong5297 May 16, 2023
18838b7
Merge branch 'solana_daily_balances_fix' of https://github.com/dunean…
andrewhong5297 May 16, 2023
555b657
Merge branch 'main' into solana_daily_balances_fix
Hosuke May 17, 2023
5ff9b39
test 30
andrewhong5297 May 20, 2023
710f461
Try to fix the incremental merge checking bug
Hosuke May 20, 2023
99c8b33
Merge branch 'main' into solana_daily_balances_fix
Hosuke May 20, 2023
0464737
Merge branch 'main' into solana_daily_balances_fix
Hosuke May 23, 2023
53e7fed
remove 30 day limit
andrewhong5297 May 23, 2023
863f146
Merge branch 'main' into solana_daily_balances_fix
jeff-dude May 23, 2023
f85eaa2
add space
andrewhong5297 May 23, 2023
9b83a82
Add not_null tests
Hosuke May 24, 2023
b1870e5
Merge branch 'main' into solana_daily_balances_fix
jeff-dude May 25, 2023
f0fff46
Merge branch 'main' into solana_daily_balances_fix
Hosuke May 26, 2023
7d38cf0
Remove not_null test for latest_balance
Hosuke May 27, 2023
28654d7
Merge branch 'main' into solana_daily_balances_fix
Hosuke May 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions models/solana_utils/solana_utils_daily_balances.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
materialized='incremental',
file_format = 'delta',
incremental_strategy='merge',
unique_key = ['token_mint_address', 'address','day'],
unique_key = ['token_mint_address', 'address', 'day'],
post_hook='{{ expose_spells(\'["solana"]\',
"sector",
"solana_utils",
Expand All @@ -16,15 +16,16 @@ WITH
SELECT
address
, date_trunc('day', block_time) as day
, token_mint_address
, coalesce(token_mint_address, '') as token_mint_address
, cast(post_balance as double)/1e9 as sol_balance --lamport -> sol
, post_token_balance as token_balance --tokens are already correct decimals in this table
, token_balance_owner
, block_slot
, row_number() OVER (partition by address, date_trunc('day', block_time) order by block_slot desc) as latest_balance
FROM {{ source('solana','account_activity') }}
WHERE tx_success
{% if is_incremental() %}
AND block_time >= date_trunc("day", now() - interval '1 day')
AND block_time > (SELECT max(day) FROM {{this}})
Copy link
Collaborator

@0xRobin 0xRobin May 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should probably use >= here to not drop later incremental updates from the same day?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since day is derived from date_trunc(...), it represents the lower bound of the time, I think > will include all events in that date after UTC 00:00.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to update on same day, only next day. To prevent duplicates.

There is the edge case of missing an address if their balance changes later that day but not the next day.

But ignoring that case for now!

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah @Hosuke you're right! I was confused but indeed because we compare block_time with the day timestamp this works and there's no real difference between > and >=. 👍

{% endif %}
)

Expand All @@ -37,4 +38,4 @@ SELECT
, token_balance_owner
, now() as updated_at
FROM updated_balances
WHERE latest_balance = 1
WHERE latest_balance = 1
14 changes: 13 additions & 1 deletion models/solana_utils/solana_utils_schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,12 @@ models:
columns:
- &token_mint_address
name: token_mint_address
tests:
- not_null
- &address
name: address
tests:
- not_null
- &updated_at
name: updated_at

Expand All @@ -30,9 +34,17 @@ models:
tags: ['solana','balances']
description: >
get the daily balances for each address, from which we can materialize a latest balance later on.
tests:
- dbt_utils.unique_combination_of_columns:
combination_of_columns:
- token_mint_address
- address
- day
columns:
- &day
name: day
tests:
- not_null
- *token_mint_address
- *address
- &sol_balance
Expand All @@ -55,7 +67,7 @@ models:
- *address
- *sol_balance
- *token_balance
- *token_mint_address
- name: token_mint_address
- *token_balance_owner
- *updated_at