How to handle 429 status code?

[jackfsuia]

I got a list of this error "WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Request blocked - received 429 status code" after I crawled about 300 pages. What should i do to continue crawling more? Like, is there any configurations for sleeping for a few seconds so the requests wont be too often?

[janbuchar]

https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions#maxRequestsPerMinute This option may help you if you experiment with it a bit 🙂