not able to call two different crawler with same url

[ganesh-swami]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

• First call the CheerioCrawler with the url
• after few seconds call the PlaywrightCrawler with the same url
• and for CheerioCrawler it will show =>INFO CheerioCrawler: Finished! Total 1 requests: 1 succeeded, 0 failed. {"terminal":true}
• but for PlaywrightCrawler crawler it doen't crawel and show => INFO PlaywrightCrawler: Finished! Total 0 requests: 0 succeeded, 0 failed. {"terminal":true}

Why I do in this way :
Because i am checking if i didn't found content with CheerioCrawler it means it is using modern js tech stack and i have to get data of this using PlaywrightCrawler. So for some websites i have to call it both first cheerio and then playwright once cheerio is done. but it show bellow things for playwright => INFO PlaywrightCrawler: Finished! Total 0 requests: 0 succeeded, 0 failed. {"terminal":true}

Also i have checked and verify that second one doens't crawl at all .

Note :

• If i remove one of the crawler it will start working for other crawler
• If i pass two different urls then both works fine

What could be the possible solution. is this bug or i am expecting something which is not possible with crawlee.

Any help will be appreciated
Thanks

Code sample

code of CheerioCrawler: 
const crawler = new CheerioCrawler({
    async requestHandler({ $, request }) {
        const title = $('title').text();
    }
})
await crawler.run(['https://hanuman-chalisa.co.in/']);

code of PlaywrightCrawler: 

const crawler2 = new PlaywrightCrawler({
    requestHandler: async ({ page, parseWithCheerio }) => {
        await page.waitForSelector('body h1');
        const $ = await parseWithCheerio();
        
    },
});

await crawler2.run(['https://hanuman-chalisa.co.in/']);

Package version

3.5

Node.js version

18+

Operating system

windlow11

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

Your code will reuse the same default/implicit RequestQueue instance for both, resulting in the described behavior. You need to create your own named RQ instance and pass it to the crawler options to have a separate state. Also note that since you will be using named RQ, you will need to purge it manually, as only the default queue is purged automatically on start.

const rq1 = await RequestQueue.open(); // default, will be purged automatically
const rq2 = await RequestQueue.open('rq2');  // named, we need to drop it at the end of the script

// code of CheerioCrawler: 
const crawler = new CheerioCrawler({
    requestQueue: rq1, // this is technically not necessary, default/implicit RQ is always there
    async requestHandler({ $, request }) {
        const title = $('title').text();
    }
})
await crawler.run(['https://hanuman-chalisa.co.in/']);

//code of PlaywrightCrawler: 
const crawler2 = new PlaywrightCrawler({
    requestQueue: rq2, // using another RQ instance to have a separate state
    requestHandler: async ({ page, parseWithCheerio }) => {
        await page.waitForSelector('body h1');
        const $ = await parseWithCheerio();
    },
});

await crawler2.run(['https://hanuman-chalisa.co.in/']);

await rq2.drop(); // drop the named RQ, otherwise it would skip the requests processed in the previous run of the script

[yossi-s]

@B4nan Is it possible to drop or marked as not handled a few urls from the RequestQueue and run it again? (in case I use the same RequestQueue)