Poor error recovery for missing closed paren leads to confusing error

[Philogy]

Version Information

• vyper Version (output of vyper --version): 0.4.0+commit.e9db8d9
• OS: osx
• Python Version (output of python --version): Python 3.12.4

What's your issue about?

Parser/tokenizer.

Please include information like:

• full output of the error you received:

tokenize.TokenError: ('EOF in multi-line statement', (56, 0))

The above exception was the direct cause of the following exception:

vyper.exceptions.SyntaxException: EOF in multi-line statement

  line 56:0 
       55     def mint(to: address, id: uint256, nonce: uint8): nonpayable
  ---> 56
  --------^

• what command you ran: vyper src/AddressRequestMarket.vy
• the code that caused the failure (see this link for help with formatting code):

# SPDX-License-Identifier: MIT
# pragma version 0.4.0


REQUEST_LOCKED: constant(uint128) = max_value(uint128)

struct RequestKey:
    owner: address
    unlock_delay: uint256
    relevant_bits_mask: uint160
    desired_bits: uint160

struct Request:
    reward: uint128
    unlocked_at: uint128

struct PackedRequest:
    packed: bytes32

_request: HashMap[bytes32, uint256]

@external
@payable
def request(unlock_delay: uint256, relevant_bits_mask: uint160, desired_bits: uint160):
    pass

def _pack(request: Request) -> PackedRequest:
    return PackedRequest(packed=convert(
        concat(
            convert(request.reward, bytes16),
            convert(request.unlocked_at, bytes16)
        ),
        bytes32
    ))

def _unpack(request: PackedRequest) -> Request:
    x : uint256 = msg.value
    return Request(
        # Missing paren here:
        reward=convert(slice(request.packed, 0, 16, uint128),
        unlocked_at=convert(slice(request.packed, 16, 16), uint128),
    )

def _request_key(owner: address, unlock_delay: uint256, relevant_bits_mask: uint160, desired_bits: uint160) -> bytes32:
    return keccak256(abi_encode(
        RequestKey(
            owner = owner,
            unlock_delay = unlock_delay,
            relevant_bits_mask = relevant_bits_mask,
            desired_bits = desired_bits
        )
    ))


interface SubZeroVanityMarket:
    def mint(to: address, id: uint256, nonce: uint8): nonpayable

• please try running your example with the --verbose flag turned on

verbose output

cli specified: `Settings(compiler_version=None, optimize=None, evm_version=None, experimental_codegen=None, debug=None, enable_decimals=None)`
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/ast/pre_parser.py", line 165, in pre_parse
    token_list = list(tokenize(io.BytesIO(code_bytes).readline))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.9_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/tokenize.py", line 525, in _tokenize
    raise TokenError("EOF in multi-line statement", (lnum, 0))
tokenize.TokenError: ('EOF in multi-line statement', (56, 0))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/bin/vyper", line 8, in <module>
    sys.exit(_parse_cli_args())
             ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/cli/vyper_compile.py", line 65, in _parse_cli_args
    return _parse_args(sys.argv[1:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/cli/vyper_compile.py", line 238, in _parse_args
    compiled = compile_files(
               ^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/cli/vyper_compile.py", line 377, in compile_files
    output = vyper.compile_from_file_input(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/compiler/__init__.py", line 120, in compile_from_file_input
    with anchor_settings(compiler_data.settings):
                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.9_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/compiler/phases.py", line 140, in settings
    settings, _ = self._generate_ast
                  ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.9_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/compiler/phases.py", line 114, in _generate_ast
    settings, ast = vy_ast.parse_to_ast_with_settings(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/ast/parse.py", line 58, in parse_to_ast_with_settings
    settings, class_types, for_loop_annotations, python_source = pre_parse(vyper_source)
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/vyper/ast/pre_parser.py", line 271, in pre_parse
    raise SyntaxException(e.args[0], code, e.args[1][0], e.args[1][1]) from e
vyper.exceptions.SyntaxException: EOF in multi-line statement

  line 56:0 
       55     def mint(to: address, id: uint256, nonce: uint8): nonpayable
  ---> 56
  --------^

How can it be fixed?

Improve the error recovery to recognize that a missing paren at the end of a function (deindent) should be recovered instead of parsing further lines as part of some multi-line statement.

[cyberthirst]

this should be a dup of: #3857

[charles-cooper]

note that this is an issue with the tokenize library. if you try compiling the following file test.txt (i manually "translated" to python compatible syntax), we get something like

  File "/home/charles/test.txt", line 38
    return Request(
                  ^
SyntaxError: '(' was never closed

[Philogy]

this should be a dup of: #3857

@cyberthirst seems to be a different issue. In yours no useful error is shown at all. Here it shows an error but highlights the wrong piece