Programmatically stop crawling

[blinjrm]

How can I programmatically stop the crawler when a condition is met?

I want to extract N products from an ecommerce website, which usually requires more than N requests because not all links point to a product page.

I know of 2 ways:

• throw a CriticalError → works but is not elegant.
• use maxRequestsPerCrawl → does not work because not all requests point to a product page, so I end up with less products than intended.

I'm looking for something like

if (productsCrawled >= maxProductsPerCrawl) {
    crawler.stop();
};

Thanks for your help.

[Strajk]

Related discussion https://discord.com/channels/801163717915574323/1138931745442500698/1138931745442500698
TLDR I'm using crawler.stats.state, my own checks, and process.exit... but also looking for more idiomatic solution

[B4nan]

There is crawler.teardown(), it will stop the autoscaled pool and clean up some event handlers (plus trigger persist state event).