Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch up job archiving code a bit #2139

Merged
merged 4 commits into from
Mar 12, 2024
Merged

Batch up job archiving code a bit #2139

merged 4 commits into from
Mar 12, 2024

Conversation

sastels
Copy link
Collaborator

@sastels sastels commented Mar 12, 2024 •
edited
Loading

Summary | Résumé

Task that archives jobs is archiving about 230 jobs / minute, which is pretty slow given that we expect it to do about 40K per day. This PR batches up both the database operation to mark the jobs as archived as well as deleting the files from s3.

Related Issues | Cartes liées

Test instructions | Instructions pour tester la modification

  • create several (5 or so) jobs, either through admin or api directly
  • change this line to "schedule": 60, (to run the task every minute)
  • change this line to end_date = today + timedelta(days=7) (to remove all jobs, not just old ones)
  • change the query limit from 20000 to 2
  • purge your queues with make run-celery-purge
  • run celery and api with make run-celery-local and make run
  • After a minute you should see something like (intermingled with other lines):
[2024-03-04 18:44:48,955: INFO/MainProcess] Task remove_sms_email_jobs[a813a22c-94df-4dbd-9bef-7129278672a3] received
...
[2024-03-04 18:44:48,974: INFO/ForkPoolWorker-2] Archiving 2 jobs.
...
[2024-03-04 18:44:50,110: INFO/ForkPoolWorker-2] Archiving 2 jobs.
...
[2024-03-04 18:44:51,072: INFO/ForkPoolWorker-2] Archiving 1 jobs.

Bonus points!

  • check the jobs in the database before / after the task runs to see the "archived" field flip
  • check that the jobs are in staging s3 before but not after the task runs

Release Instructions | Instructions pour le déploiement

Check the speed that jobs are being archived. Should be much faster than 230 / minute.

Reviewer checklist | Liste de vérification du réviseur

  • This PR does not break existing functionality.
  • This PR does not violate GCNotify's privacy policies.
  • This PR does not raise new security concerns. Refer to our GC Notify Risk Register document on our Google drive.
  • This PR does not significantly alter performance.
  • Additional required documentation resulting of these changes is covered (such as the README, setup instructions, a related ADR or the technical documentation).

⚠ If boxes cannot be checked off before merging the PR, they should be moved to the "Release Instructions" section with appropriate steps required to verify before release. For example, changes to celery code may require tests on staging to verify that performance has not been affected.

@sastels sastels mentioned this pull request Mar 12, 2024
Closed
1 task
@sastels sastels marked this pull request as ready for review March 12, 2024 18:00
ben851
ben851 approved these changes Mar 12, 2024
View reviewed changes
Copy link
Contributor

@ben851 ben851 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

jimleroyer
jimleroyer reviewed Mar 12, 2024
View reviewed changes
app/aws/s3.py Outdated
response = bucket.delete_objects(Delete={"Objects": [{"Key": key} for key in object_keys]})
except botocore.exceptions.ClientError:
current_app.logger.exception("Couldn't delete any objects from bucket %s.", bucket.name)
raise
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why log it to raise it? we could just let celery log it itself no? We might end up with two exceptions here per my understanding.

app/celery/nightly_tasks.py Outdated
for job in jobs:
s3.remove_job_from_s3(job.service_id, job.id)
dao_archive_job(job)
current_app.logger.info("Job ID {} has been removed from s3.".format(job.id))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could dump the list instead of iterating through each individual entry as well.

jimleroyer
jimleroyer approved these changes Mar 12, 2024
View reviewed changes
Copy link
Member

@jimleroyer jimleroyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sastels sastels merged commit 71a1268 into main Mar 12, 2024
4 checks passed
@sastels sastels deleted the batch-job-archiving branch March 12, 2024 20:28
sastels added a commit that referenced this pull request Mar 19, 2024
@sastels
Revert "Batch up job archiving code a bit (#2139)"
9c7d650 
This reverts commit 71a1268.
@sastels sastels mentioned this pull request Mar 19, 2024
Merged
sastels added a commit that referenced this pull request Mar 19, 2024
@sastels
Revert "Batch up job archiving code a bit (#2139)" (#2142)
604ec61 
This reverts commit 71a1268.
@sastels sastels restored the batch-job-archiving branch March 19, 2024 17:14
@sastels sastels mentioned this pull request Mar 19, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ben851 ben851 ben851 approved these changes

@jimleroyer jimleroyer jimleroyer approved these changes

@jzbahrai jzbahrai Awaiting requested review from jzbahrai

@P0NDER0SA P0NDER0SA Awaiting requested review from P0NDER0SA

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sastels @jimleroyer @ben851