Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix daily limit counting when scheduling jobs #2112

Merged
merged 9 commits into from
Feb 22, 2024

Conversation

whabanks
Copy link
Contributor

@whabanks whabanks commented Feb 13, 2024

Summary | Résumé

Jobs scheduled for > today were being counted toward the current day's daily limit, causing services to hit their daily sending limit before they should have.

Depends on this Utils PR

Changelog:

  • When scheduling a job, the daily limits in Redis will only be incremented if the job was scheduled for the current day
  • When refreshing daily count keys from the DB the underlying query will now include counts from jobs scheduled for the current day
  • Experimenting with a new decorator for feature flags
  • Added the Poetry installation dir to the $PATH when running the postCreate hook on dev container creation. This should resolve an occasional issue where poetry could not be found once the container finishes building.
  • Refactored create_job in app/job/rest.py to reduce it's cyclomatic complexity < 15.

Testing

Redis keys to observe throughout the tests

  • email-{service_id}-count
  • {service_id}-{current_date}-count

Notes:

  • The last two tests need to be performed in succession if you do not currently have any jobs that were scheduled for the current day and then canceled.
  • The overall message limit key should not change through these tests. This key is incremented once the scheduled job fires and the notifications are added to the DB and processed.

Jobs scheduled for tomorrow or later do not increment Redis counts

  1. Create a few one off email notifications.
  2. Verify that the counts for both keys in Redis are accurate
  3. Schedule a job for tomorrow or later
  4. Note that the counts were not incremented in either key

Jobs scheduled within the current day increment the Redis counts

  1. Create a job scheduled for no later than the current day at 23:59:59
  2. Note that the email count key was incremented

Canceling a job scheduled for the current day decrements the Redis counts

  1. Via Admin, cancel the job that you scheduled in the previous test
  2. Note that the email count key was decremented

Jobs scheduled within the current day, but were canceled, are not counted when the email-{service_id}-count key is refreshed from the DB

  1. Take note of the current value of the email count key
  2. Delete the aforementioned key in Redis
  3. Send a one off notification, this should trigger the email count key to be refreshed from the DB
  4. Note that re-populated value matches the count noted in step 1 (+1 for the one off sent in step 4).
    • i.e. The canceled job was not counted by the underlying query that refreshes the Redis key.

- When scheduling a job, the daily limits in Redis will only be
  incremented if the job was scheduled for the current day
- When refreshing daily count keys from the DB the underlying query will
  now include counts from jobs scheduled for the current day
- Experimenting with a new decorator for feature flags
- Fixed an issue with the dev container where the poetry installation will
  sometimes not be detected by adding the installation dir to the $PATH
Copy link
Member

@andrewleith andrewleith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but I'm not sure why the tests are failing. The new utils is tagged so I thought it would pass.

app/dao/services_dao.py Show resolved Hide resolved
@whabanks
Copy link
Contributor Author

Looks good but I'm not sure why the tests are failing. The new utils is tagged so I thought it would pass.

Yea, I've had this happen once before, where it acts like the tag didn't stick. I had to bump the version again but luckily it looks like there's another bump on the way. I'll bump the version in this PR once that is merged. Hopefully it resolves itself.

@whabanks whabanks merged commit 9d1d445 into main Feb 22, 2024
4 checks passed
@whabanks whabanks deleted the fix/scheduled-job-daily-limit-counts branch February 22, 2024 17:29
@whabanks whabanks mentioned this pull request Feb 22, 2024
increment_email_daily_count_send_warnings_if_needed(service, len(list(recipient_csv.get_rows())))
if scheduled_for is None or not scheduled_for.date() > datetime.today().date():
increment_email_daily_count_send_warnings_if_needed(
authenticated_service, len(list(recipient_csv.get_rows())) # noqa: F821
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still failing local smoke tests. authenticated_service doesnt seem to have the id property this method expects it to have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants