Added indexes #425
Added indexes #425
Conversation
| @@ -336,7 +336,11 @@ class Job(Document): | |||
| default=False | |||
| ) # Marked true when all retry steps have completed | |||
|
|
|||
| meta = {"collection": "ee2_jobs"} | |||
| # See https://docs.mongoengine.org/guide/defining-documents.html#indexes | |||
| # Hmm, are these indexes need to be + or - ? | |||
There was a problem hiding this comment.
Depends on if you want sorted results - if not, it doesn't really matter. With the status/queued index, if you do an equality query on status and a range query on queued you can sort descending on the queue time without having to do an in-memory sort. Ascending sorts would be in memory.
There was a problem hiding this comment.
Ok, with the change now oldest first sorts can be done without an in-memory sort
| meta = {"collection": "ee2_jobs"} | ||
| # See https://docs.mongoengine.org/guide/defining-documents.html#indexes | ||
| # Hmm, are these indexes need to be + or - ? | ||
| indexes = [("status", "batch_job"), ("status", "-queued")] |
There was a problem hiding this comment.
Hmm - would it make sense to have indexes with the username, e.g. (user, status, -queued)? Does EE2 not do any user specific queries?
There was a problem hiding this comment.
Only for the job browser bff
funcdef check_jobs_date_range_for_user(CheckJobsDateRangeParams params)
returns (CheckJobsDateRangeResults) authentication required;
funcdef check_jobs_date_range_for_all(CheckJobsDateRangeParams params)
returns (CheckJobsDateRangeResults) authentication required;
There was a problem hiding this comment.
yeah I see that the original impetus was for the bad jobs reaper, which I assume doesn't care about the user name.
that being said, indexes targeted for those functions might not be a bad idea
There was a problem hiding this comment.
Here's another call that searches on clusterid instead of job id
execution_engine2/lib/execution_engine2/sdk/EE2Status.py
Lines 52 to 53 in 1e268df
| meta = {"collection": "ee2_jobs"} | ||
| # See https://docs.mongoengine.org/guide/defining-documents.html#indexes | ||
| # Hmm, are these indexes need to be + or - ? | ||
| indexes = [("status", "batch_job"), ("status", "queued")] |
There was a problem hiding this comment.
What's the query for the status/batch_job index?
There was a problem hiding this comment.
Do we figure this out using .explain() ? |
|
yep |
|
Well, that and just thinking through how the queries will use the indexes and how selective the indexes will be for the query |
Description of PR purpose/changes
{"status": Status.queued.value, "queued": {"$lt": before_days}}
{"status": Status.created.value, "created": {"$lt": before_days}}
I don't think there's any indexes on status or the timestamp, so this is going to be a tablescan. I'd recommend making compound indices on state/queued and state/created
I'd just set them up in MongoUtil. That's more or less how I do it for all my servers, for example
execution_engine2/bin/PurgeBadJobs.py
Line 102 in bc0644a
execution_engine2/bin/PurgeBadJobs.py
Line 81 in bc0644a
Hmm I wonder which type of indexes are best, and if I need a different index on 'created' because it actually needs to know if its a batch job or not and the ID record not a field called created.