Building a PEX for multiple platforms doesn't affect sys.platform in extras_require

[kevincon]

I'm trying to build a cross-platform (Mac and Linux) PEX that contains the doit library, but it seems as if pex is not doing anything to make the target platform be respected by the usages of sys.platform in the conditions of the extras_require dictionary in its setup.py file.

The following command results in the failure I'm running into on my Mac (running 10.12.6):

➜  pex -v --disable-cache --python=python3 doit --platform macosx_10.12-x86_64 --platform linux-x86_64 -o doit.pex
  doit 0.31.1 pex :: Resolving distributions :: Packaging MacFSEvents
  cloudpickle 0.5.2
  MacFSEvents 0.8.1
pex: Target package WheelPackage('file:///private/var/folders/b7/qgjhz8kx5fvc6kz_fnx55b600000gq/T/tmpt_08qmf1/MacFSEvents-0.8.1-cp36-cp36m-macosx_10_12_x86_64.whl') is not compatible with CPython-3.6.4 / linux-x86_64
Traceback (most recent call last):
  File "/usr/local/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/pex/bin/pex.py", line 663, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/usr/local/lib/python3.6/site-packages/pex/bin/pex.py", line 601, in build_pex
    for dist in resolveds:
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 438, in resolve_multi
    allow_prereleases):
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 376, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 209, in resolve
    dist = self.build(package, resolvable.options)
  File "/usr/local/lib/python3.6/site-packages/pex/resolver.py", line 177, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('https://pypi.python.org/packages/28/2e/1ff399cfd2a6a8ebb65152203c920643c2169aa507b2e96559d19baeb7c3/MacFSEvents-0.8.1.tar.gz#md5=45759553d58bab6f7c8d6187fb0fe12f') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

It appears as if pex is succeeding in building the part of the PEX for macosx_10.12-x86_64 (makes sense since that's my host OS), but failing on the part for linux-x86_64.

Shouldn't pex not try to use MacFSEvents for the linux-x86_64 platform since the choice between MacFSEvents and pyinotify is guarded by checking sys.platform in the extras_require dictionary?

I also don't actually need either MacFSEvents nor pyinotify for my purposes, so alternatively is there a way to tell pex to ignore the extras_require dictionary?

If instead this is something that should be fixed/adjusted in the doit library, what do you recommend I change in a potential pull request for that library?

[kwlzn]

respecting these extras environmental markers is a known gap in pex - and (I believe) a known gap in general for cross-platform python resolution.

most interactions with pip involve resolving and installation packages only on the local machine. over time this routine mode of operation has lead to standardization of various forms of the baked in sys.platform logic in dependencies you linked to (see also PEP 508 "environmental markers", the wheel docs, python code directly in setup.py, etc). behind this tho, there's a baked in age-old assumption that "the platform executing the resolver logic is the same one you're resolving for", which is what leads to the gap from pip vs pex or other multi-platform resolvers where this assumption isn't true.

because of these age old assumptions, this is a semi-hard problem to solve for generally - and for not much benefit. you'd effectively have to emulate a platform to the best of your ability - stubbing out all of the environmental markers/sys/platform/etc - in a running python interpreter.

an interesting datapoint here would be to know whether or not e.g. pip download --platform exhibited the same behavior, because that's the only officially "blessed" analog I can think of (and may be a solution we can glean from).

[kwlzn]

there's also no exclude syntax that I'm aware of for python requirements, tho that'd be nice in cases like this and could be worth pursuing.

beyond that, it seems like the only way to make this particular package cross-platform-resolver friendly would be to drop the env marker extras in favor of making users explicitly define those (e.g. doit[linux] vs doit[osx]) - but that's likely not a generally acceptable solution.

an inversion of that idea that may be a viable escape hatch in the longer term on a per-platform axis tho, could be to implement a pex --intransitive mode that would force you to explicitly specify the transitive set of requirements. this would not solve for the "need MacFSEvents only for OSX && pyinotify only for Linux" problem tho because there's no way to specify e.g. a combination of (requirement, platform) as requirement specification.

so yeah, a harder problem than it appears on the surface.

[jsirois]

I just noticed this issue. AFAICT it has since been ~solved on all points:

• Environment markers are now fully supported, even for --platform (Fix --platform resolve handling of env markers. #1367 in 2021) and the newer --complete-platform (Add support for --complete-platform. #1609 in 2022).
• An --intransitive mode is now supported (Add an --intransitive option. #775 in 2019).
• An --exclude facility has been implemented (Implement support for --exclude <req>. #2281 / Option to exclude dependencies from pex file #2097 very recent development).

All that said, the OP use case fails still - as it should. The Command line explicitly requests 2 platforms are satisfied in the PEX and, since both MacFSEvents and pyinotify are sdist only (on PyPI), this cannot be done. One of the 2 requested platform's sdists will always fail to build:

$ pex --disable-cache --python=python3 doit --platform macosx_10.12-x86_64-cp-36-cp36m --platform linux-x86_64-cp-36-cp36m -o doit.pex
pid 403838 -> /tmp/tmp7ht1pz5f/venvs/2e166bfd4eedf783ee2a3e3b89b13708f8137918/0de1795ad4486f45ee94cecd983e9905b6b11dc9/bin/python -sE /tmp/tmp7ht1pz5f/venvs/2e166bfd4eedf783ee2a3e3b89b13708f8137918/0de1795ad4486f45ee94cecd983e9905b6b11dc9/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /tmp/tmp7ht1pz5f/pip/23.2/pip_cache wheel --no-deps --wheel-dir /tmp/tmp7ht1pz5f/built_wheels/sdists/MacFSEvents-0.8.4.tar.gz/bf7283f1d517764ccdc8195b21631dbbac1c506b920bf9a8ea2956b3127651cb/cp312-cp312-manylinux_2_35_x86_64.74524750d4dc4d75af16aa137780564b.work /tmp/tmp7ht1pz5f/downloads/resolver_download.wx91asi9/cp36-cp36m-macosx_10_12_x86_64/MacFSEvents-0.8.4.tar.gz --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-312
      copying fsevents.py -> build/lib.linux-x86_64-cpython-312
      running build_ext
      building '_fsevents' extension
      creating build/temp.linux-x86_64-cpython-312
      x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/tmp7ht1pz5f/venvs/2e166bfd4eedf783ee2a3e3b89b13708f8137918/0de1795ad4486f45ee94cecd983e9905b6b11dc9/include -I/usr/include/python3.12 -c _fsevents.c -o build/temp.linux-x86_64-cpython-312/_fsevents.o
      _fsevents.c:2:10: fatal error: CoreFoundation/CoreFoundation.h: No such file or directory
          2 | #include <CoreFoundation/CoreFoundation.h>
            |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for MacFSEvents
ERROR: Failed to build one or more wheels

I tried out the newish --exclude feature here and it does not work since it only excludes post resolve; i.e.: after [download dists, build wheels, install wheels], which is how a Pex resolve is now composed. I'll take up this issue and close out with a fix that plumbs --exclude deeper to avoid the build wheels and install wheels steps in the Pex resolve.

[AlexTereshenkov]

Thank you, @jsirois, I have also spent a few hours trying to get Pants pass the --exclude flag to pex so that a requirement that is not listed in a lockfile would be ignored, and can confirm it fails at the resolve stage.

I attempt to exclude python3-tabulate package that is not declared in the lockfile:

22:01:24.44 [DEBUG] spawned local process as Some(74171) for Process { argv: ["/home/user.name/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin/python", "./pex", "--tmpdir", ".tmp", "--jobs", "1", "--no-emit-warnings", "--pip-version", "24.0", "--python-path", "/home/user.name/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin:/home/user.name/code/cheeseshop-query/.venv/bin:/home/user.name/.npm-global/bin:/home/user.name/bin/:/usr/local/go/bin:/home/user.name/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/user.name/.local/bin", "--output-file", "requirements.pex", "--exclude=python3-tabulate", "--python", "/usr/bin/python3.9", "--sources-directory=source_files", "click", "loguru", "packaging", "python-dateutil", "python3-tabulate", "requests", "types-python-dateutil", "types-requests", "typing-extensions", "--lock", "requirements/requirements.lock", "--no-pypi", "--index=https://pypi.org/simple/", "--manylinux", "manylinux2014", "--exclude", "python3-tabulate", "--layout", "packed"], env: {"CPPFLAGS": "", "LANG": "en_US.UTF-8", "LDFLAGS": "", "PATH": "/home/user.name/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin:/home/user.name/code/cheeseshop-query/.venv/bin:/home/user.name/.npm-global/bin:/home/user.name/bin/:/usr/local/go/bin:/home/user.name/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/user.name/.local/bin", "PEX_IGNORE_RCFILES": "true", "PEX_PYTHON": "/home/user.name/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin/python", "PEX_ROOT": ".cache/pex_root"}, working_directory: None, input_digests: InputDigests { complete: DirectoryDigest { digest: Digest { hash: Fingerprint, size_bytes: 328 }, tree: "Some(..)" }, nailgun: DirectoryDigest { digest: Digest { hash: Fingerprint, size_bytes: 0 }, tree: "Some(..)" }, inputs: DirectoryDigest { digest: Digest { hash: Fingerprint, size_bytes: 328 }, tree: "Some(..)" }, immutable_inputs: {}, use_nailgun: {} }, output_files: {}, output_directories: {RelativePath("requirements.pex")}, timeout: None, execution_slot_variable: None, concurrency_available: 9, description: "Building 9 requirements for requirements.pex from the requirements/requirements.lock resolve: click, loguru, packaging, python-dateutil, python3-tabulate, requests, types-python-dateutil, types-requests, typing-extensions", level: Info, append_only_caches: {CacheName("pex_root"): RelativePath(".cache/pex_root"), CacheName("python_build_standalone"): RelativePath(".python-build-standalone")}, jdk_home: None, cache_scope: Successful, execution_environment: ProcessExecutionEnvironment { name: None, platform: Linux_x86_64, strategy: Local }, remote_cache_speculation_delay: 0ns, attempt: 0 }

and it fails with

ProcessExecutionFailure: Process 'Building 9 requirements for requirements.pex from the requirements/requirements.lock resolve: click, loguru, packaging, python-dateutil, python3-tabulate, requests, types-python-dateutil, types-requests, typing-extensions' failed with exit code 1.
stdout:

stderr:
Failed to resolve compatible artifacts from lock requirements/requirements.lock for 1 target:
1. /usr/bin/python3.9:
    Failed to resolve all requirements for cp39-cp39-manylinux_2_36_x86_64 interpreter at /usr/bin/python3.9 from requirements/requirements.lock:

Configured with:
    build: True
    use_wheel: True

Dependency on python3-tabulate (via: python3-tabulate) not satisfied, no candidates found.

It would be superb to be able to ask PEX not to search for a package in a lockfile. The use case is that one may have most packages resolved in the lockfile, but there will be a few that are not there (but will be available on the PATH at runtime).

[AlexTereshenkov]

Unless I have overseen something, this applies also to running pex without a lockfile:

ERROR: Could not find a version that satisfies the requirement python3-tabulate (from versions: none)
ERROR: No matching distribution found for python3-tabulate

[jsirois]

@AlexTereshenkov it would be really useful if you could provide full info (the lock file in question). As it stands it's not clear to me your report is totally germane to this particular issue. You seem to be describing wanting the ability to pass --exclude not-in-input-requirements-or-in-resulting-lockfile-at-all.

[AlexTereshenkov]

Oh great to catch up again, John :) thanks for taking a look!

I have a PR https://github.com/AlexTereshenkov/cheeseshop-query/pull/39/files in case you want to run things locally for yourself. $ PEX_EXTRA_SYS_PATH="/home/user.name/local/lib/python3.11/site-packages" pants --no-local-cache test tests/cli works great, but only if I put some dummy wheel to get installed for the "provided" package; I have edited the lockfile manually to be able to run the tests.

In this PR AlexTereshenkov/cheeseshop-query#40 I have everything just as in the first one except the lockfile is left as is (automatically generated). $ PEX_EXTRA_SYS_PATH="/home/user.name/local/lib/python3.11/site-packages" pants --no-local-cache --python-enable-resolves=false test tests/cli with resolves disabled leads to failures as well which is expected.

[AlexTereshenkov]

FWIW I am actually fine to run the commands with no / empty lockfile - the primary use case is that none of the dependencies are going to be provided via Python packages, so all of them will be available only at runtime. I'd love pex to totally ignore any 3rd party requirements, similar to how it's done for the pex_binary target: https://www.pantsbuild.org/2.19/reference/targets/pex_binary#include_requirements

[jsirois]

@AlexTereshenkov your primary use case is tortured via Pants as far as I can tell. IIUC you want a PEX of only your 1st party code (since all 3rdparty code will be provided by pre-installed packages visible to the host interpreter). That's trivial: pex -D src/ -o sources.pex.

I will look at the info you provided above when I get back to a keyboard on the 19th, but I suggest opening a dedicated issue if needed.

[jsirois]

I tried out the newish --exclude feature here and it does not work since it only excludes post resolve; i.e.: after [download dists, build wheels, install wheels], which is how a Pex resolve is now composed. I'll take up this issue and close out with a fix that plumbs --exclude deeper to avoid the build wheels and install wheels steps in the Pex resolve.

I have fixed up exclude in this way, but it still does not solve the OP case fully since pyinotify uses an aggressive setup.py that sys.exits if not being executed on Linux: https://github.com/seb-m/pyinotify/blob/0f3f8950d12e4a6534320153eed1a90a778da4ae/setup.py#L27-L30 So, even with --exclude fixed to prevent building of sdists, the pip download resolve step still needs to run python setup.py egg_info to get dependency metadata, and that aborts on Mac:

...
E         Running command python setup.py egg_info
E         inotify is not available on macosx-10.9-universal2
E         error: subprocess-exited-with-error
E         
E         × python setup.py egg_info did not run successfully.
E         │ exit code: 1
E         ╰─> See above for output.
...

To get around this sort of aggressive behavior in a setup.py would require patches to Pip itself to honor some form of --exclude that Pex could pass through.

[cognifloyd]

pyinotify has been a thorn in my side for years. If there's something PEX could do to work around that awful (or "aggressive" as you put it so nicely) setup.py, like patching pip somehow, that would be amazing!

[jsirois]

@cognifloyd you can always use a vcs requirement pointing to your fork + patch. Since pyinotify is a sdist anyhow, this is next to no skin off your back IIUC - you just need to add a top-level fake requirement.

[jsirois]

If there's something PEX could do to work around that awful (or "aggressive" as you put it so nicely) setup.py, like patching pip somehow, that would be amazing!

Alright @cognifloyd b3695e1 makes this so. The same trick should make it easy to implement --override in a follow up to change a transitive dependency instead of eliminate it.

[jsirois]

Filed #2425 to track --override.

[jsirois]

Alrighty, at long last I think this can be considered complete. With the improved --exclude released in 2.4.0, the OP example can be satisfied and this is now enshrined in an integration test.