Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark and improve performance for Reports::Dynamic #292

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

mwarin
Copy link
Contributor

@mwarin mwarin commented Oct 12, 2023

Gather here all performace fixes to related to the reporting platform, which was introduced in:
#289

After putting into production and running against the production MongoDB, I could tell that this was going to need some tuning. It is slow and takes up an awful lot of resources. I can think of 3 approaches:

  1. Benchmark ruby code and see what can be easily improved there.
  2. Add indexes to MongoDB where missing.
  3. See what can be parallelized without too much fussing.

In production, I would like the following basic query to take less than 20m.
It's currently at 3h+.

dyna_rep = Reports::Dynamic.new(
  base: "holdings",
  decorations: ["holdings.ocn", "holdings.local_id"],
  restrictions: {"holdings.organization" => "ucmerced"},
)
dyna_rep.run

41575

@coveralls
Copy link

coveralls commented Oct 12, 2023

Coverage Status

coverage: 94.973% (+0.02%) from 94.951% when pulling a866b41 on report-platform-performance into 7952406 on main.

@mwarin mwarin force-pushed the report-platform-performance branch from be7b9e1 to b688e2f Compare October 18, 2023 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants