Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse error if comment # starts in the middle of the line #179

Open
wkl3nk opened this issue Jul 9, 2024 · 1 comment
Open

Parse error if comment # starts in the middle of the line #179

wkl3nk opened this issue Jul 9, 2024 · 1 comment

Comments

@wkl3nk
Copy link

wkl3nk commented Jul 9, 2024

Version: 0.10.0, 0.12.0
To reproduce, the following repository can be used for scanning:
https://github.com/danielhrisca/asammdf

This contains a dependency to numexpr that itself has a line in its requirement.txt has has a comment that starts in the middle of the line:
numpy >= 1.23.0 # keep in sync with NPY_TARGET_VERSION (setup.py)

Here is the output of a scan with python-inspector:

asammdf % python-inspector --python-version 311 --operating-system linux --json-pdt /tmp/my-dependencies.json --analyze-setup-py-insecurely --requirement requirements.txt --verbose

Resolving dependencies...
direct_dependencies:
 DependentPackage(purl='pkg:pypi/canmatrix', extracted_requirement='canmatrix[arxml,dbc]>=1.0', scope='install')
 DependentPackage(purl='pkg:pypi/lz4', extracted_requirement='lz4', scope='install')
 DependentPackage(purl='pkg:pypi/numexpr', extracted_requirement='numexpr', scope='install')
 DependentPackage(purl='pkg:pypi/numpy', extracted_requirement='numpy>=1.23.0', scope='install')
 DependentPackage(purl='pkg:pypi/pandas', extracted_requirement='pandas', scope='install')
 DependentPackage(purl='pkg:pypi/typing-extensions', extracted_requirement='typing_extensions', scope='install')
 DependentPackage(purl='pkg:pypi/python-dateutil', extracted_requirement='python-dateutil', scope='install')
 DependentPackage(purl='pkg:pypi/isal', extracted_requirement='isal; platform_machine == "x86_64" or platform_machine == "AMD64"', scope='install')
 DependentPackage(purl='pkg:pypi/lxml', extracted_requirement='lxml>=4.9.3', scope='install')
environment: Environment(python_version='311', operating_system='linux')
repos:
 PypiSimpleRepository(index_url='https://pypi.org/simple', credentials=None)
Traceback (most recent call last):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/packvers/requirements.py", line 102, in __init__
    req = REQUIREMENT.parseString(requirement_string)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/pyparsing/util.py", line 256, in _inner
    return fn(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/pyparsing/core.py", line 1200, in parse_string
    raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found '#'  (at char 16), (line:1, col:17)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolve_cli.py", line 257, in resolve_dependencies
    resolution_result: Dict = resolver_api(
                              ^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/api.py", line 280, in resolve_dependencies
    resolution, purls = resolve(
                        ^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/api.py", line 344, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/api.py", line 385, in get_resolved_dependencies
    resolver_results = resolver.resolve(requirements=requirements, max_rounds=max_rounds)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 427, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 239, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/resolvelib/resolvers.py", line 229, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 652, in get_dependencies
    return list(self._iter_dependencies(candidate))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 643, in _iter_dependencies
    for r in self.get_requirements_for_package(purl=purl, candidate=candidate):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 502, in get_requirements_for_package_from_pypi_simple
    yield from get_reqs_insecurely(
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 277, in get_reqs_insecurely
    yield from parse_reqs_from_setup_py_insecurely(setup_py=setup_py_location)
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/resolution.py", line 128, in parse_reqs_from_setup_py_insecurely
    for req in iter_requirements(level="", extras=[], setup_file=setup_py):
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/python_inspector/setup_py_live_eval.py", line 151, in iter_requirements
    pkg = Requirement(req)
          ^^^^^^^^^^^^^^^^
  File "/Users/klw1imb/.pyenv/versions/3.11.1/lib/python3.11/site-packages/packvers/requirements.py", line 104, in __init__
    raise InvalidRequirement(
packvers.requirements.InvalidRequirement: Parse error at "'# keep i'": Expected string_end

Is it allowed to have comments start in the middle of the line? I guess the Requirements File Format allows it when saying

A line that begins with # is treated as a comment and ignored. Whitespace followed by a # causes the # and the remainder of the line to be treated as a comment.

@sebassz
Copy link

sebassz commented Jul 10, 2024

This is most likely the same as mentioned in #152. There is already an open PR in the underlying packvers library to address this problem aboutcode-org/packvers#3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants