Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move Regional Records computation to Auxiliary table #10122

Merged
merged 13 commits into from
Oct 29, 2024

Conversation

gregorbg
Copy link
Member

@gregorbg gregorbg commented Oct 22, 2024

Supersedes #10089, although I'm very grateful for the initial inspiration 😄
Supersedes #10120

The basic problem is that the query to pre-load existing records was breaking our necks. The simple solution of this PR is to combine Results data and Competitions data into one common, JOINed table which can be indexed.
That way, we can essentially reuse the same queries as before (with minor modifications to the table name etc.) but the indexes will make them reasonably fast.

Before

SELECT r.eventId, r.countryId, MIN(r.best) AS `value`
FROM (
  SELECT `Results`.`eventId`, `Results`.`competitionId`, `Results`.`countryId`, `Results`.`best`
  FROM `Results`
  WHERE `Results`.`best` > 0
) AS r
INNER JOIN (
  SELECT `Competitions`.`id`
  FROM `Competitions`
  WHERE `Competitions`.`end_date` < '2024-08-10' # This seemingly hard-coded value is read directly from the `Competitions` table one step before
) AS c
  ON c.id = r.competitionId
INNER JOIN (
  SELECT `competition_events`.`event_id`
  FROM `competition_events`
  WHERE `competition_events`.`competition_id` = 'StevenageAugust2024'
) AS ce
  ON ce.event_id = r.eventId
GROUP BY r.eventId, r.countryId

Benchmark on my local machine: 11390.9ms

After

⚠️ Please do not be confused by the table being referenced as Results. This is a necessary hack because otherwise the Rails model engine gets confused! It is only an alias that is freely choosable, and in terms of understanding the contents of this query, you can mentally replace Results with pancake.

SELECT eventId, countryId, MIN(best) AS `value`
FROM regional_records_lookup AS Results # This alias is here only to make sure that Ruby doesn't get confused
WHERE `Results`.`best` > 0
  AND `Results`.`competitionEndDate` < '2024-08-10'
  AND `Results`.`eventId` IN ('333', '222', '444', '555', '666', '777', '333bf', '333fm', '333oh', 'clock', 'minx', 'pyram', 'sq1')
GROUP BY `Results`.`eventId`, `Results`.`countryId`

The relevant index is on (eventId, countryId, competitionEndDate, best) (as well as (eventId, countryId, competitionEndDate, average) respectively for averages)

Benchmark on my local machine: 1804.1ms

@@ -29,6 +29,11 @@
huge).
</p>

<%= alert :warning do %>
This script relies on auxiliary tables which are computed as part of <%= link_to "running CAD", admin_compute_auxiliary_data_path, target: :_blank %>.
<b>It will not deliver accurate results for specific competitions if CAD hasn't been run after the results posting.</b>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe replace "after the results posting" with "after the 'import results' step", so that it won't get confused with last step?

lib/db_helper.rb Show resolved Hide resolved
@gregorbg gregorbg force-pushed the feature/regional-records-aux-table branch from e8de5c1 to fe25062 Compare October 29, 2024 03:15
@gregorbg gregorbg merged commit 2050836 into thewca:main Oct 29, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants