feat: calculate the stats via stats queue #76
Conversation
jrwbabylonlab
commented
Apr 26, 2024
jrwbabylonlab
commented
- Pending merge of this PR: feat: add stats queue staking-queue-client#14
- This PR eliminates the logical database sharding for finality provider stats. Instead, we've redirected the stats calculation to its own dedicated queue. This abstraction separates it from the core state transition logic. The advantage of this approach is the implementation of a retryable delayed queue. If the stats calculation fails due to a database lock (e.g., trying to update the same record in a collection), it will be reprocessed several minutes later.
- There is also a pending task to retry failed transactions, which will further enhance reliability. However, this can be addressed later: add retry for db transactions #63
jrwbabylonlab
requested review from
vitsalis,
gitferry,
liam-icheng-lai and
gusin13
and removed request for
liam-icheng-lai
| @@ -19,6 +19,7 @@ db.unbonding_queue.createIndex({'unbonding_tx_hash_hex': 1}, {unique: true}); | |||
| db.timelock_queue.createIndex({'expire_height': 1}, {unique: false}); | |||
| db.delegations.createIndex({'staker_pk_hex': 1, 'staking_tx.start_height': -1}, {unique: false}); | |||
| db.staker_stats.createIndex({'active_tvl': -1, '_id': 1}, {unique: false}); | |||
| db.finality_providers_stats.createIndex({'active_tvl': -1, '_id': 1}, {unique: false}); | |||
There was a problem hiding this comment.
@liam-icheng-lai or @filippos47 i will need your help on updating the DB indexes to include all of them from line 18 to 22. I think we already have the index for 18-20. but not for 21 and 22.
| } | ||
|
|
||
| var finalityProviderDetailsPublic []FpDetailsPublic | ||
| for _, fp := range resultMap.Data { |
There was a problem hiding this comment.
Won't this skip finality providers that are in the global params, but do not have any delegations? For example, consider that the system is live, and our finality providers registry contains fp1, fp2, fp3, while fp1, fp2, and fp4 have received delegations. Will the output contain fp1, fp2, fp3, and fp4 or just fp1, fp2, fp4? I think we should have the first.
There was a problem hiding this comment.
it will be p1, p2 and p4.
The UI will be showing a paginated and sorted by active TVL FP list, right? This would make fp3 not a natural fit into the paginated response as its tvl will be 0. But what we can do is by checking if we have more result from db, if no more items, we can append the 0 tvl FP from the global params.
There was a problem hiding this comment.
@vitsalis i had a quick look into the implementation if we want to make it return fp1,2,3,4 all together and work with pagination. It's a little bit more work to do than simply append the global params FP into the end of the return list.
I would prefer track this conversation and the implementation in a seperate ticket so it at least unblock the load test etc.
For context, if we wanna return fp1,2,3,4. we need to do below:
- Check if we are at the end of the pagination. i.e no more items to query in db.
- make a new query with provided FP pk to db. if PKs not exist in the db, we append the FP into the returned list with tvl of 0.
There was a problem hiding this comment.
It is a requirement to return fp1,2,3,4. Otherwise, when we launch the system, as soon as a single finality provider gets a delegation, all others won't appear. We have the pre-registration with the deposit for this reason -- you register so that you appear on the website and attract delegations. My recommendation is to do this in the current PR, as it is a core requirement of the functionality. If you think it is blocking, then we can do this in another PR, but it's a pre-requisite for testnet launch.
There was a problem hiding this comment.
noted. yeah, will make sure this is fixed before testnet launch (should be fixed by next week). would prefer to close off all existing PRs first and have the staking cap get out the way first.
#88
jrwbabylonlab
force-pushed
the
stats-calculation-through-stats-queue
branch
from
f6b2173 to
2065790
Compare
jrwbabylonlab
force-pushed
the
stats-calculation-through-stats-queue
branch
from
2065790 to
ce43ce8
Compare
