t1/t2 balance, t1/t2 active counts recalculations.

[ice-myles]

The problem

Due to bugs that were fixed after the app release launch, part of the users have got the wrong T1/T2 balances calculations. For example one of the bugs: when some user removed his account, his T1, T2 didn't get the T1/T2 active referrals change in proper way.
To reduce the inequality we must set the right T1/T2 balances (as all other balances were not affected) for everybody so that everyone is on equal terms.

The solution

To recalculate balances we should get active ranges for each user and correlate them with active ranges of T1/T2.

For example:

T0 = user1
T1 = user2
...
TN = userN (T1)

user1 (T0) active time ranges from clickhouse:

2023/10/12 14:49:37.593809 - 2023/10/12 14:51:37.593809 (2 min) 
2023/10/12 14:52:50.593809 - 2023/10/12 14:53:50.593809  (1 min)

user2 (T1) active time ranges from clickhouse:

2023/10/12 14:49:45.593809 - 2023/10/12 14:51:45.593809 (2 min)
2023/10/12 14:52:37.593809 - 2023/10/12 14:53:37.593809 (1 min)

So user1 ranges are crossed with user2 (T1):

2023/10/12 14:49:45.593809 - 2023/10/12 14:51:37.593809 (1 min 52 sec of joint activity for the first ranges)
2023/10/12 14:52:50.593809 - 2023/10/12 14:53:37.593809 (47 sec of joint activity for the second ranges)
etc

Then user1 ranges are crossed with all separate users from T1 to TN
Also the same is done for T2 users to get crossing time ranges and use active time ranges to recalculate balances.

Slashing

For the slashing case we need to know only when T0 (user for whom we recalculate T1/T2 balances) wasn't active (when his slashing_solo_rate > 0). In this case everything related to slashing does mine() function and more specifically it just set slashing rates and each elapsed time decrease the balances by slashing rate.

Only 1 additional thing that should be known here about implementation: created_at field from the clickhouse history is used for slashing calculations and last value (created_at) is always "rounded", it means it has values like 12:00, 13:00, 14:00, etc. Then when the slashing is finished and new mining session is started, we just need to slash time diff between mining_session_solo_started_at and created_at.

For example new mining session started at 12:37, then we need to slash last 37 minutes in the last history record before new mining session, and next 23 minutes are used then for the positive balances calculations.

Streaks and day offs

They are taken into the account. As theirs implementation assumes that several mining_session_solo_started_at fields can have the same value (due to streaks), this case is handled not to take into account the repeated time ranges. The implementation has the previousUserStartedAt, previousUserEndedAt, previousT1MiningSessionStartedAt, previousT1MiningSessionEndedAt, previousT2MiningSessionStartedAt, previousT2MiningSessionEndedAt variables to use them for streaks and day-offs detection.

Resurrect

Resurrection is done not through mine() function (as it was difficult to pass right time points) but by separate flag in the recalculations. If resurrect was happened for an user, just boolean flag is set to true, t1/t2 balances is being resurrected, and then resurrection is not repeated more for this user.

Adoption switch among the crossing time ranges

As base mining rate (BMR) can be changed, we need to take this change into the account while recalculating. For this case implementation sends the start and end values of the range to the splitByAdoptionTimeRanges function. This function splits the time range by adoptions values, for example:

N    timePoint                    BMR
---------------------------------------
1     2023/10/12 14:49:45.593809, 16
2     2023/10/12 14:49:57.593809,  8
3     2023/10/12 14:50:32.593809,  4
4     2023/10/12 14:51:37.593809,  2

The time between start and end as a rule is the time from AchievedAt field of adoptions (if any adoption switch is happened between this range).
So each (time point - previous time point) must be calculated with it's own BMR.
NOTE: The function all the time returns start and end time points with theirs actual adoptions.

New users while balances are recalculated

For new users while balances are recalculated recalculated key will be added as we don't need to recalculate anything for them (they were created after bugfixing).

The difference between actual and recalculated values problem

The difference between t1/t2 calculated and t1/t2 actual balances will be all the time due to mine() function calculates balances based on small pieces of time when some active_t1_referrals/active_t2_referrals could have been already decreased.
More details for the understanding: each user has active_t1_referrals and active_t2_referrals fields. For example user0 has:

active_t1_referrals = 10
active_t2_referrals = 5

when mine() function is called. The small piece of time, when balances are recalculated, use these values to multiply with T1/T2 coefficients and elapsed time (for example 40 ms). But 1 of 10 active t1 could stop be active after 20 ms from these 40 ms, though 20 ms are still used for the multiplying. So these small pieces of time can add balance a bit.

This balance recalculation implemenation has more accuracy as it is kept on exact time ranges from the history, but not on the T1/T2 active counts.

When is better to use recalculations

Recalculations is better to use at the beginning of the hour/minute for production/staging as here history has the fresh update.
For example if recalculation was launched at 11:58, and new history records will be appeared at 12:00 only, then this last hour will not be taken into the account.

Balances backup mode

When the balances are recalculated the first time, then the old t1/t2 balances (t1/t2 slashing rates and t1/t2 active counts) need to be stored to have possibility to restore them .
balances_backup_mode key should be added with value true to restore all stuff that was before recalculation. If to save not true (anything another) - it will work in recalculation mode.

NOTE: if 1 day passed from the time point when recalculation was started and then backup was used, the balances for this day will not be restored.

Actualise santa

update_santa_badges_levels_and_roles_tasks script actualises data in santa database.
If balance/friends invited were changed after the recalculations, then need to be changed friends invited/balance/socials/levels/roles/badges/etc fields in santa.

Actualise eskimo

Several fields in eskimo have the inequality, so we need to make them actual as well. For this is enough to update T1/T2 values in the referral_acquisition_history table.

Update t1/t2 values

WITH t2_table AS (
    SELECT
       SUM(1) as t2_count,
       u.id
  FROM users u
    JOIN USERS t1
       ON t1.referred_by = u.ID
              AND t1.id != u.id
              AND t1.username != t1.id
              AND t1.referred_by != t1.id
    JOIN USERS referrals
      ON referrals.referred_by = t1.ID
            AND referrals.id != t1.id
            AND referrals.username != referrals.id
            AND referrals.referred_by != referrals.id
  GROUP BY u.id
 ),
 calculated AS (
    SELECT 
       u.id AS id,
       COUNT(t1.id) AS t1_count,
       COALESCE(t2_count, 0) AS t2_count
  FROM users u
      LEFT JOIN t2_table
          ON t2_table.id = u.id
      LEFT JOIN USERS t1
          ON t1.referred_by = u.id
                 AND t1.username != t1.id
                 AND t1.referred_by != t1.id
  GROUP BY u.id, t2_count
)
UPDATE referral_acquisition_history rah
     SET t1 = calculated.t1_count,
            t2 = calculated.t2_count
  FROM calculated
  WHERE rah.user_id = calculated.id

T1 check

WITH t1_table AS (
   SELECT 
       COUNT(*) AS t1_count, u.referred_by AS id
   FROM users u
       JOIN USERS referrals
                ON (u.referred_by = referrals.id)
                       AND u.username != u.id
                       AND u.referred_by != u.id
  GROUP BY u.referred_by
)
SELECT id, t1_count, rah.t1 FROM t1_table
     JOIN referral_acquisition_history rah
     ON id = rah.user_id
WHERE rah.t1 != t1_count 

T2 check

WITH t2_table AS (
SELECT
  u.id, SUM(1) as t2_count
 FROM users u
 JOIN USERS t1
          ON t1.referred_by = u.ID
    AND t1.id != u.id
          AND t1.username != t1.id
          AND t1.referred_by != t1.id
    JOIN USERS referrals
   ON referrals.referred_by = t1.ID
   AND referrals.id != t1.id
            AND referrals.username != referrals.id
   AND referrals.referred_by != referrals.id
GROUP BY u.id
) 
SELECT id, t2_count, rah.t2 FROM t2_table
  JOIN referral_acquisition_history rah
      ON id = rah.user_id
WHERE rah.t2 != t2_count