Crawlee forces me to install full puppeteer eventough I am giving it the executablePath

[ali-habibzadeh]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

Trying to launch a crawler by passing executablePath so I don't have to bundle Chrome with my code but it seems Crawlee's PuppeteerCrawler is locked to Puppeteer and can't work with puppeteer-core. Is there any workaround for this?

import { EnqueueStrategy, PuppeteerCrawler } from "crawlee";
import express from "express";
import { Page } from "puppeteer-core";
import { getExecutablePath } from "./executable-path";

const app = express();
app.listen(process.env.PORT || 4100);

const startCrawl = async (props: { startUrl: string; onRequest: (page: Page) => Promise<void> }) => {
  const crawler = new PuppeteerCrawler({
    requestHandlerTimeoutSecs: 120,
    launchContext: { launchOptions: { executablePath: getExecutablePath() } },
    maxConcurrency: 10,
    async requestHandler({ enqueueLinks, log, request, page }) {
      await props.onRequest(page);
      log.info(`Added: ${request.url}`);
      await enqueueLinks({ strategy: EnqueueStrategy.SameDomain });
    },
  });
  await crawler.run([props.startUrl]);
};

Throws

Error: Cannot find module 'puppeteer'

Using useChrome: true also does not help with using the local browser. it still errors that there is no Puppeteer

Code sample

No response

Package version

3.10.2

Node.js version

20

Operating system

osx

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

Error: Cannot find module 'puppeteer'

You need to install puppeteer package, that's your problem, it's not about executable paths. They no longer install browsers in that package, so why do you want to use just puppeteer-core?

[ali-habibzadeh]

Because I am using a separate service for Chrome to make my application scalable. That means it's pointless to ship with the browser binary. puppeteer-core is the official package for this that ships without browser's binary but only the library. I see you have hardcoded it to puppeteer with chrome.

I am sure you have good design decision behind it.

[B4nan]

I just told you this is not the case anymore, they don't ship any browser with puppeteer package.

[ali-habibzadeh]

It's fine I can move it to your basic crawler and do the puppteer stuff inside the request handler

[B4nan]

cc @vladfrangu, at least that's how I understood it, since we now need to install the browsers explicitly ourselves.

if its not like that, I guess we should use puppetteer-core internally?