Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beat unique fix #6

Merged
merged 9 commits into from
Oct 6, 2023
Merged

Beat unique fix #6

merged 9 commits into from
Oct 6, 2023

Conversation

TkTech
Copy link
Owner

@TkTech TkTech commented Oct 5, 2023

It's possible for the celery beat scheduler to skip calling apply_async(), if it's unable to import the actual task. In these cases, it'll resort to using send_task(), see https://github.com/celery/celery/blob/main/celery/beat.py#L406. This means that our unique lock may never be checked, since we hook it into Task.apply_async().

This change adds verification of the lock at the time a task is run, while keeping the early lock as well.

This is a known issue with other similar libraries, see:

@TkTech TkTech requested a review from dominicplouffe October 5, 2023 16:17
@TkTech TkTech self-assigned this Oct 5, 2023
@codecov
Copy link

codecov bot commented Oct 5, 2023

Codecov Report

Attention: 15 lines in your changes are missing coverage. Please review.

Comparison is base (43727fa) 58.74% compared to head (fbbcc5a) 60.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #6      +/-   ##
==========================================
+ Coverage   58.74%   60.00%   +1.25%     
==========================================
  Files           7        7              
  Lines         286      295       +9     
==========================================
+ Hits          168      177       +9     
  Misses        118      118              
Files Coverage Δ
celery_heimdall/__init__.py 100.00% <100.00%> (ø)
celery_heimdall/config.py 100.00% <100.00%> (ø)
celery_heimdall/errors.py 100.00% <100.00%> (ø)
celery_heimdall/contrib/inspector/monitor.py 0.00% <0.00%> (ø)
celery_heimdall/contrib/inspector/cli.py 0.00% <0.00%> (ø)
celery_heimdall/task.py 93.79% <89.13%> (+0.41%) ⬆️
celery_heimdall/contrib/inspector/models.py 0.00% <0.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@TkTech TkTech merged commit b6709f6 into main Oct 6, 2023
@jzelez
Copy link

jzelez commented Jan 10, 2025

Hey, first of all - awesome work on this package.

Secondly, I've noticed an issue with this approach - namely, when you call the decorated function directly, the lock is still acquired (while it shouldn't in this case) and it is then never released, because it doesn't use any of the task api calls/signals.

I've subclassed the HeimdallTask to handle this. Basically, bypassing the acquire lock behaviour if the task/function is called directly.

    def __call__(self, *args, **kwargs):
        if self.request.called_directly:
            return self.run(*args, **kwargs)

        return super().__call__(*args, **kwargs)

@TkTech
Copy link
Owner Author

TkTech commented Jan 10, 2025

Great catch! It may still be desirable to have the lock used when running directly, so we'll need to make this a configuration option and move cleanup into the call. We'll add this and a test for it in the refactor in #10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants