Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix --exclude. #2409

Merged
merged 10 commits into from
Jun 12, 2024
Merged

Fix --exclude. #2409

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
71ae3c9
Add support for testing if one SpecifierSet includes another.
jsirois Jun 4, 2024
99a6f38
Re-factor pex.asserts to split out reportable error message generation.
jsirois Jun 4, 2024
f2be4ec
Extract platform or "universal" logic.
jsirois Jun 4, 2024
5479bcd
Allow PatchSets to be additive.
jsirois Jun 4, 2024
2967b7d
Add machinery to support deep excludes.
jsirois Jun 4, 2024
5591f75
Plumb deep excludes and add tests.
jsirois Jun 9, 2024
c1b7e64
Run `tox -evendor` for selective un-vendoring.
jsirois Jun 10, 2024
372016f
Selectively un-vendor only what's needed.
jsirois Jun 10, 2024
d24fb32
Fix inexact version handling.
jsirois Jun 10, 2024
ed6b49c
Simplify exclude culling by performing it in the merge pass.
jsirois Jun 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
*.pyc
*.pyo
__pycache__/

*~

/.mypy_cache/
Expand Down
2 changes: 1 addition & 1 deletion pex/build_system/pep_517.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ def _default_build_system(
)
)
resolved_reqs.add("setuptools")
extra_env.update(__PEX_UNVENDORED__="1")
extra_env.update(__PEX_UNVENDORED__="setuptools")
else:
requires = [
selected_pip_version.setuptools_requirement,
Expand Down
2 changes: 1 addition & 1 deletion pex/interpreter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1594,7 +1594,7 @@ def spawn_python_job(
if expose:
# In order to expose vendored distributions with their un-vendored import paths in-tact, we
# need to set `__PEX_UNVENDORED__`. See: vendor.__main__.ImportRewriter._modify_import.
subprocess_env["__PEX_UNVENDORED__"] = "1"
subprocess_env["__PEX_UNVENDORED__"] = ",".join(expose)

pythonpath.extend(third_party.expose(expose, interpreter=interpreter))

Expand Down
6 changes: 4 additions & 2 deletions pex/pip/download_observer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@
import os
import pkgutil

from pex import third_party
from pex.common import safe_mkdtemp
from pex.pip.log_analyzer import LogAnalyzer
from pex.third_party import isolated
from pex.typing import TYPE_CHECKING

if TYPE_CHECKING:
Expand Down Expand Up @@ -77,7 +77,9 @@ def emit_patches(self, package):
"Given: {package!r}".format(package=package)
)

import_paths = list(third_party.expose(["pex"]))
patches_dir = safe_mkdtemp()
import_paths.append(patches_dir)
patches_package = os.path.join(patches_dir, package)
os.mkdir(patches_package)

Expand All @@ -92,7 +94,7 @@ def emit_patches(self, package):
print("from . import {module}".format(module=patch.module), file=fp)
print("{module}.patch()".format(module=patch.module), file=fp)

return patches_dir, isolated().chroot_path
return tuple(import_paths)

def add(self, patch_set):
# type: (PatchSet) -> PatchSet
Expand Down
2 changes: 1 addition & 1 deletion pex/pip/tool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -358,7 +358,7 @@ def _spawn_pip_isolated(
with ENV.strip().patch(
PEX_ROOT=ENV.PEX_ROOT,
PEX_VERBOSE=str(ENV.PEX_VERBOSE),
__PEX_UNVENDORED__="1",
__PEX_UNVENDORED__="setuptools",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue before this change was the un-vendoring was global and just above in pex/pip/download_observer.py, pex was newly exposed to Pip to support patches that use Pex code (the --exclude patches leverage this). That pex code, including everything under pex.vendor._vendored was being exposed which broke pex.third_party vendored imports for Python2.7 where the meta-path importer has different precedence than under newer Pythons. Since Pex itself never imports setuptools, exposing just that serves both the Pex patches of Pip (which don;t need pex.third_party.setuptools) and vendored Pip itself (which needs setuptools to build sdists). Ideally Pex would not re-write setuptools imports at all (since it never does a pex.third_party import of the code), but Lambdex does and this change would break Lambdex. A test caught this and serves as a backstop to not break Lambdex users.

**extra_env
) as env:
# Guard against API calls from environment with ambient PYTHONPATH preventing pip PEX
Expand Down
5 changes: 1 addition & 4 deletions tests/resolve/lockfile/test_json_codec.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,7 @@

import attr # vendor:skip
else:
if "__PEX_UNVENDORED__" in __import__("os").environ:
import attr # vendor:skip
else:
import pex.third_party.attr as attr
import pex.third_party.attr as attr


def test_roundtrip(tmpdir):
Expand Down
3 changes: 1 addition & 2 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,6 @@ commands =
description = Run Pex's vendored pip.
skip_install = true
setenv =
__PEX_UNVENDORED__ = 1
PYTHONPATH = {env:PYTHONPATH:}{:}{toxinidir}/pex/vendor/_vendored/pip
SETUPTOOLS_USE_DISTUTILS = stdlib
commands =
Expand All @@ -237,7 +236,7 @@ commands =
description = Run Python with Pex's vendored setuptools on the sys.path.
skip_install = true
setenv =
__PEX_UNVENDORED__ = 1
__PEX_UNVENDORED__ = setuptools
PYTHONPATH = {env:PYTHONPATH:}{:}{toxinidir}/pex/vendor/_vendored/setuptools
SETUPTOOLS_USE_DISTUTILS = stdlib
commands =
Expand Down