How to run Nodriver on Linux or Lambda?

[raunaqss]

Hi there,

Hope you're having a pleasant day.

Firstly, thanks for making Nodriver and sharing it for all of us to use, it's unbelievably quick and sneaky.

I'm trying to get it working on Linux or AWS Lambda. Using a similar image as this one here for lambda: https://github.com/umihico/docker-selenium-lambda

Here's how my function looks:

async def scrape_with_nodriver(url):
    proxy_server = config('PROXY_SERVER_US')
    chrome_binary_path = config('CHROME_BINARY_PATH', default='/opt/chrome/chrome')
    browser = await nodriver.start(browser_executable_path=chrome_binary_path)
    page = await browser.get(url)
    await page.select('div.element')
    html_source = await page.get_content()
    product_details = parse_product_details(html_source)
    return product_details

However, I'm getting the following error:

{ "errorMessage": "\n ---------------------\n Failed to connect to browser\n ---------------------\n One of the causes could be when you are running as root.\n In that case you need to pass no_sandbox=True \n ", "errorType": "Exception", "requestId": "1d3a2a26-1a2e-4d8b-887d-8375fe504b57", "stackTrace": [ " File \"/var/task/main.py\", line 16, in handler\n return uc.loop().run_until_complete(wayfair_detail_handler.handler(event, context))\n", " File \"/var/lang/lib/python3.12/asyncio/base_events.py\", line 685, in run_until_complete\n return future.result()\n", " File \"/var/task/app/wayfair_detail_handler_uc.py\", line 18, in handler\n product_details = await scrape_with_nodriver(url)\n", " File \"/var/task/app/wayfair_detail_handler_uc.py\", line 27, in scrape_with_nodriver\n browser = await nodriver.start(proxy_server=proxy_server, browser_executable_path=chromedriver_path)\n", " File \"/var/lang/lib/python3.12/site-packages/nodriver/core/util.py\", line 74, in start\n return await Browser.create(config)\n", " File \"/var/lang/lib/python3.12/site-packages/nodriver/core/browser.py\", line 87, in create\n await instance.start()\n", " File \"/var/lang/lib/python3.12/site-packages/nodriver/core/browser.py\", line 343, in start\n raise Exception(\n" ] }

The above message is from AWS Lambda but I'm getting the same error on Linux as well. For linux I've set the browser_executable_path to /opt/google/chrome/chrome (using /opt/google/chrome results in permission errors for some reason).

Any suggestions for how I could get it to work?

Thanks for reading!

Cheers and regards 🫶🏻

[Emoncht]

Try not to run it with root user. I suggest to create a seperate user.

[raunaqss]

I'm going to try this out and get back on whether it worked.

[ganhj99]

@raunaqss Is this working already on your end?

[strukiii]

I'm running into the same exact issue, would be great to get some clarification here and have this work out of the box. The old undetected-chromedriver works fine on my same ubuntu server interestingly. To add to this, when using headless mode I get this error.

  File "/home/carter/Scrapers/auth_proxy_test.py", line 62, in <module>
    Scraper()
  File "/home/carter/Scrapers/auth_proxy_test.py", line 12, in __init__
    loop().run_until_complete(self.run())
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/carter/Scrapers/auth_proxy_test.py", line 16, in run
    browser = await start(
              ^^^^^^^^^^^^
  File "/home/carter/Scrapers/venv/lib/python3.12/site-packages/nodriver/core/util.py", line 96, in start
    return await Browser.create(config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/carter/Scrapers/venv/lib/python3.12/site-packages/nodriver/core/browser.py", line 91, in create
    await instance.start()
  File "/home/carter/Scrapers/venv/lib/python3.12/site-packages/nodriver/core/browser.py", line 394, in start
    await self.connection.send(cdp.target.set_discover_targets(discover=True))
  File "/home/carter/Scrapers/venv/lib/python3.12/site-packages/nodriver/core/connection.py", line 420, in send
    await self._prepare_headless()
  File "/home/carter/Scrapers/venv/lib/python3.12/site-packages/nodriver/core/connection.py", line 499, in _prepare_headless
    response, error = await self._send_oneshot(
    ^^^^^^^^^^^^^^^
TypeError: cannot unpack non-iterable NoneType object
successfully removed temp profile /tmp/uc_ip90k5fm