CheerioCrawler rate limits?

[sharkstate]

Im using CheerioCrawler and notice that after a certain number of crawler loops (crawling 10 pages per crawl) CheerioCrawler stops processing the posted urls, eg 10 urls are posted through await crawler.run(urls); but it jumps directly to "CheerioCrawler: All requests from the queue have been processed, the crawler will shut down".

I cant find info about this limit, how it works, and if it can be adjusted?

Thanks!

[B4nan]

There is no limit (at least not enabled by default), the problem you are facing is because you are reusing the same crawler instance, with the same storage and its own state. The state is locally stored in the storage folder, there are several ways to deal with this problem.

We actually made some improvements in this field in the latest version (#2056), so with v3.5.3 you can do this:

const getCrawler = () => {
    return new CheerioCrawler({
        requestHandler: async () => {},
    }, new Configuration({
        storageClient: new MemoryStorage({
            persistStorage: false,
        }),
    }));
};

const a = getCrawler();

await a.run([
    { url: 'https://example.org/' },
]);

const b = getCrawler();

await b.run([
    { url: 'https://example.org/' },
]);

We are working on some guides around this topic.

[sharkstate]

Thank you! This works great.