python3Packages.selenium: ModuleNotFoundError: No module named 'selenium.webdriver.common.devtools'

[milahu]

ModuleNotFoundError: No module named 'selenium.webdriver.common.devtools'
FileNotFoundError: [Errno 2] No such file or directory: '/nix/store/j5xnifvfvbsf66qj50v9ns355xfv3hsr-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/devtools'

problem: python3Packages.selenium is built from github source, not from pypi source

selenium.nix

nixpkgs/pkgs/development/python-modules/selenium/default.nix

buildPythonPackage rec {
  pname = "selenium";
  version = "4.14.0";
  format = "setuptools";

  disabled = pythonOlder "3.7";

  src = fetchFromGitHub {
    owner = "SeleniumHQ";
    repo = "selenium";
    # check if there is a newer tag with or without -python suffix
    rev = "refs/tags/selenium-${version}";
    hash = "sha256-cTMCKfFLUlJDbTUQA3Z/pKCE1RQQRMb4K8hKKn9HqvU=";
  };

upstream issue: SeleniumHQ/selenium#9010

ModuleNotFoundError: No module named 'selenium.webdriver.common.devtools'

Fixed in SeleniumHQ/selenium@a420091

upstream uses the gn build system to fix the issue, but the pypi "Source Distribution" is still broken
the pypi "Built Distribution" works, because it has the selenium/webdriver/common/devtools/*/*.py files

fix 1

in setup.py use find_packages
and add an empty selenium/webdriver/common/devtools/__init__.py file

fix 2

in setup.py use find_namespace_packages
because there is no selenium/webdriver/common/devtools/__init__.py

When you use find_packages(), all directories without an __init__.py file will be ignored. On the other hand, find_namespace_packages() will scan all directories.

done in selenium.nix

fix 3

use the pypi "Built Distribution"
this works, because it has the selenium/webdriver/common/devtools/*/*.py files

fix 4

fix the buildPhase to produce the missing files
see the upstream issue: SeleniumHQ/selenium#9010

---

setuptools/command/build_py.py:207: _Warning: Package 'selenium.webdriver.common.devtools.v119' is absent from the `packages` configuration.

$ nix log /nix/store/23rs2vj83drvqw0fy4631kp11f4v30ma-python3.11-selenium-4.17.2

!!
  check.warn(importable)
/nix/store/7fnhc3nd125lq9vzw6bghsvbnmcc8mc1-python3.11-setuptools-69.0.2/lib/python3.11/site-packages/setuptools/command/build_py.py:207: _Warning: Package 'selenium.webdriver.common.devtools.v119' is absent from the `packages` configuration.
!!

        ********************************************************************************
        ############################
        # Package would be ignored #
        ############################
        Python recognizes 'selenium.webdriver.common.devtools.v119' as an importable package[^1],
        but it is absent from setuptools' `packages` configuration.

        This leads to an ambiguous overall configuration. If you want to distribute this
        package, please make sure that 'selenium.webdriver.common.devtools.v119' is explicitly added
        to the `packages` configuration field.

        Alternatively, you can also rely on setuptools' discovery methods
        (for example by using `find_namespace_packages(...)`/`find_namespace:`
        instead of `find_packages(...)`/`find:`).

        You can read more about "package discovery" on setuptools documentation page:

        - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

        If you don't want 'selenium.webdriver.common.devtools.v119' to be distributed and are
        already explicitly excluding 'selenium.webdriver.common.devtools.v119' via
        `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
        you can try to use `exclude_package_data`, or `include-package-data=False` in
        combination with a more fine grained `package-data` configuration.

        You can read more about "package data files" on setuptools documentation page:

        - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


        [^1]: For Python, any directory (with suitable naming) can be imported,
              even if it does not contain any `.py` files.
              On the other hand, currently there is no concept of package data
              directory, all directories are treated like packages.
        ********************************************************************************

what packages are missing

$ nix log /nix/store/23rs2vj83drvqw0fy4631kp11f4v30ma-python3.11-selenium-4.17.2 | grep "Python recognizes" 
        Python recognizes 'selenium.webdriver.common.actions' as an importable package[^1],
        Python recognizes 'selenium.webdriver.common.bidi' as an importable package[^1],
        Python recognizes 'selenium.webdriver.common.devtools.v119' as an importable package[^1],
        Python recognizes 'selenium.webdriver.common.devtools.v120' as an importable package[^1],
        Python recognizes 'selenium.webdriver.common.devtools.v121' as an importable package[^1],
        Python recognizes 'selenium.webdriver.common.devtools.v85' as an importable package[^1],
        Python recognizes 'selenium.webdriver.safari' as an importable package[^1],
        Python recognizes 'selenium.webdriver.webkitgtk' as an importable package[^1],
        Python recognizes 'selenium.webdriver.wpewebkit' as an importable package[^1],

setup.py

    'packages': ['selenium',
                 'selenium.common',
                 'selenium.webdriver',
                 'selenium.webdriver.chromium',
                 'selenium.webdriver.chrome',
                 'selenium.webdriver.common',
                 'selenium.webdriver.support',
                 'selenium.webdriver.firefox',
                 'selenium.webdriver.ie',
                 'selenium.webdriver.edge',
                 'selenium.webdriver.remote',
                 'selenium.webdriver.support', ],
    'include_package_data': True,

repro.py

repro.py from https://stackoverflow.com/questions/66227508

#!/usr/bin/env python3

import trio # async library that selenium uses
from selenium import webdriver

async def start_listening(listener):
    async for event in listener:
        print(event)

async def main():
    driver = webdriver.Chrome()

    async with driver.bidi_connection() as connection:
        session, devtools = connection.session, connection.devtools

        # await session.execute(devtools.fetch.enable())
        await session.execute(devtools.network.enable())

        # listener = session.listen(devtools.fetch.RequestPaused)
        listener = session.listen(devtools.network.ResponseReceived)
        async with trio.open_nursery() as nursery:
            nursery.start_soon(start_listening, listener) # start_listening blocks, so we run it in another coroutine
            url = "https://httpbin.org/get" # json response
            driver.get(url)

trio.run(main)

simply using the pypi source does not work: the selenium-manager binary is missing at /lib/python3.11/site-packages/selenium/webdriver/common/linux/selenium-manager

{ lib
, fetchPypi
, buildPythonPackage
, setuptools
, wheel
, certifi
, trio
, trio-websocket
, typing-extensions
, urllib3
}:

buildPythonPackage rec {
  pname = "selenium";
  version = "4.17.2";
  pyproject = true;

  src = fetchPypi {
    inherit pname version;
    hash = "sha256-1D1pcuUWhV+yQu+c5M51kFexFQcOcC57HBAy/ns42Hs=";
  };

  # relax versions
  # fix: typing-extensions~=4.9 not satisfied by version 4.8.0
  # https://github.com/SeleniumHQ/selenium/blob/trunk/py/setup.py

  # also add selenium.webdriver.common.devtools.* packages
  # fix: ModuleNotFoundError: No module named 'selenium.webdriver.common.devtools'
  # https://github.com/milahu/nixpkgs/issues/20

  postPatch = ''
    sed -i 's/[~>]=.*"/"/' setup.py
    substituteInPlace setup.py \
      --replace \
        "    'packages': [" \
        "    'packages': find_namespace_packages(where='.'), '_packages': [" \
      --replace \
        "from setuptools import setup" \
        "from setuptools import setup, find_namespace_packages" \
  '';

  nativeBuildInputs = [
    setuptools
    wheel
  ];

  propagatedBuildInputs = [
    certifi
    trio
    trio-websocket
    typing-extensions
    urllib3
  ];

  pythonImportsCheck = [ "selenium" ];

  meta = with lib; {
    description = "Bindings for Selenium WebDriver";
    homepage = "https://selenium.dev/";
    license = licenses.asl20;
    maintainers = with maintainers; [ ];
  };

}

$ ./repro.py
Traceback (most recent call last):
  File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/driver_finder.py", line 38, in get_path
    path = SeleniumManager().driver_location(options) if path is None else path
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/selenium_manager.py", line 89, in driver_location
    args = [str(self.get_binary()), "--browser", browser]
                ^^^^^^^^^^^^^^^^^
  File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/selenium_manager.py", line 73, in get_binary
    raise WebDriverException(f"Unable to obtain working Selenium Manager binary; {path}")
selenium.common.exceptions.WebDriverException: Message: Unable to obtain working Selenium Manager binary; /nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/linux/selenium-manager


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/src/milahu/opensubtitles-scraper/./aiohttp_chromium/test/stream-response/Network.dataReceived.selenium.2.py", line 38, in <module>
    trio.run(main)
  File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/trio/_core/_run.py", line 2251, in run
    raise runner.main_task_outcome.error
  File "/home/user/src/milahu/opensubtitles-scraper/./aiohttp_chromium/test/stream-response/Network.dataReceived.selenium.2.py", line 14, in main
    driver = webdriver.Chrome()
             ^^^^^^^^^^^^^^^^^^
  File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/chromium/webdriver.py", line 49, in __init__
    self.service.path = DriverFinder.get_path(self.service, options)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/ph32hh8p5vd8s3i2wk9vfyk06p7j7xv1-python3-3.11.7-env/lib/python3.11/site-packages/selenium/webdriver/common/driver_finder.py", line 41, in get_path
    raise NoSuchDriverException(msg) from err
selenium.common.exceptions.NoSuchDriverException: Message: Unable to obtain driver for chrome using Selenium Manager.; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/driver_location

similar issues

• Some files of selenium python module are not packaged NixOS/nixpkgs#230232
• pythonPackages.selenium: FileNotFoundError: [Errno 2] No such file or directory, 'getAttribute.js' NixOS/nixpkgs#181035

ping

ping maintainer @jraygauthier

ping contributors @arkivm @sdht0 @AleXoundOS @mweinelt @contrun @SuperSandro2000 @lapp0 @polykernel @paulhersch @reflektoin