Ignore package?

[ariscn]

I'm using pex to build from several different requirements.txt files. The catch is that I'm on OS X and building for Linux. Even with --platform, I'm unable to run my pex on Linux because of missing packages that don't exist on the platform, most notably gnureadline.

I can work around this by maintaining a separate requirements file with gnureadlines removed, but this is a bit hacky. Is there a way to have pex ignore a package at either build time or runtime?

[kwlzn]

@ariscn there's no way to make pex ignore a package at build or runtime, other than to conditionally remove the package from the input set before passing in the requirements to pex. this could be as simple as a build script or Makefile that parameterizes the linux-specific requirements inputs when building. it would be contrary to pex's philosophy and emphasis on repeatability/~immutability to arbitrarily ignore packages at runtime - as that could affect the entire dependency set in strange ways.

however, in general building on OSX and targeting a Linux runtime is a very common case given some forethought. the approach we use for platform-native deps is to pre-compile those into binary distributions (e.g. eggs or wheels) on a Linux machine (via something like: pip wheel --no-cache-dir --wheel-dir=./output_dir/ $requirement) and then make them available to the OSX machine via an HTTP/directory store. when using the --platform flag combined with --find-links against the HTTP store containing the Linux bdists, pex is smart enough to compose a pex that will run on Linux from the available pre-compiled bdists. you can also pre-compile your OSX deps in the same way to keep things uniform.

another common solution is to eliminate platform-specific deps altogether (e.g. use pure-python libs like ldap3 in favor of python-ldap), but this may not always be possible. pre-building bdists is probably more ideal for you if you have a hard requirement on platform-native deps.

[benley]

Rather than conditionally ignoring dependencies, is there any way to specify build options for requirements? For example, in order to build python-ldap with pip on MacOS, it's necessary to add some flags to help it find sasl.h:

pip install python-ldap \
    --global-option=build_ext \
    --global-option="-I$(xcrun --show-sdk-path)/usr/include/sasl"

Is there any way to accomplish the equivalent using pex, or is hand-building a egg/wheel bdist the recommended workaround for such things?

[kwlzn]

@benley yeah - the recommended approach would be to build a bespoke bdist and then let pex resolve against bdists at build time. most folks create a "wheelhouse" type store that just contains wheels for this purpose and then use --find-links against that store. then you'd just populate the wheelhouse via pip or other mechanisms and then let pex simply resolve against the pre-built bdists.

this will not only make your builds generally faster (less compilation work to be done on each build), but also more repeatable and reliable (eliminates an entire class of problems that arise from environmental variance in compilation from dev machine to dev machine).

[jsirois]

@ariscn Pex has since grown support for --exclude which acts like your suggested --ignore:

• Option to exclude dependencies from pex file #2097
• https://github.com/pex-tool/pex/releases/tag/v2.4.0

@benley Pex switched to using Pip as its resolver in 2.x and you can now pass Pip options. For your case I downloaded the libsasl tarball, unpacked it and made a top-level symlink in the unpacked tarball from sasl -> include. This allows:

:; PIP_GLOBAL_OPTION="build_ext -I/home/jsirois/downloads/cyrus-sasl-2.1.28" pex --disable-cache --use-pip-config --no-use-pep517 python-ldap -- -c 'import ldap; print(ldap.__file__)'
/tmp/tmp_rhh4yy3/installed_wheels/870fbfbb0b77ebefe6e127072f1b007c3209e60f04b111df97600f7f37ee9653/python_ldap-3.4.4-cp311-cp311-linux_x86_64.whl/ldap/__init__.py

And to show it really works, this fails with the -I option typo-ed:

:; PIP_GLOBAL_OPTION="build_ext -I/TYPO/jsirois/downloads/cyrus-sasl-2.1.28" pex --disable-cache --use-pip-config --no-use-pep517 python-ldap -- -c 'import ldap; print(ldap.__file__)'
...
pip:   running build_ext
pip:   building '_ldap' extension
pip:   creating build/temp.linux-x86_64-cpython-311
pip:   creating build/temp.linux-x86_64-cpython-311/Modules
pip:   gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -DHAVE_SASL -DHAVE_TLS -DLDAPMODULE_VERSION=3.4.4 "-DLDAPMODULE_AUTHOR=python-ldap project" "-DLDAPMODULE_LICENSE=Python style" -IModules -I/tmp/tmpgg8_vzzy/venvs/960deab6b1961767adc5420cc4589b505810b49d/bcd5cfc9f138f1afceeb38adfb3b9b1c5b07dbe9/include -I/home/jsirois/.pyenv/versions/3.11.9/include/python3.11 -c Modules/LDAPObject.c -o build/temp.linux-x86_64-cpython-311/Modules/LDAPObject.o
pip:   Modules/LDAPObject.c:16:10: fatal error: sasl/sasl.h: No such file or directory
pip:      16 | #include <sasl/sasl.h>
pip:         |          ^~~~~~~~~~~~~
pip:   compilation terminated.
pip:   error: command '/usr/bin/gcc' failed with exit code 1
pip:   ----------------------------------------
pip:   ERROR: Failed building wheel for python-ldap
pip: ERROR: Failed to build one or more wheels
...