Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Scrapy 2.10 and RETRY_EXCEPTIONS setting #41

Open
Prometheus3375 opened this issue Sep 13, 2023 · 0 comments · May be fixed by #42
Open

Add support for Scrapy 2.10 and RETRY_EXCEPTIONS setting #41

Prometheus3375 opened this issue Sep 13, 2023 · 0 comments · May be fixed by #42

Comments

@Prometheus3375
Copy link

Prometheus3375 commented Sep 13, 2023

In Scrapy 2.10 RetryMiddleware was updated (PR scrapy/scrapy#5929) to deprecate EXCEPTIONS_TO_RETRY tuple. There is an error (issue scrapy/scrapy#6049) that makes RetryUserAgentMiddleware instances to throw AttributeError upon accessing EXCEPTIONS_TO_RETRY.

To resolve this problem and add support for new RETRY_EXCEPTIONS setting, __init__ must be updated in the following way:

    def __init__(self, crawler: Crawler):
        RetryMiddleware.__init__(self, crawler.settings)
        RandomUserAgentBase.__init__(self, crawler)

        if hasattr(self, 'exceptions_to_retry'):
            self.EXCEPTIONS_TO_RETRY = self.exceptions_to_retry

Such solution maintains support for previous Scrapy versions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants