Strange output with terminal priorities

[vivien000]

Hi. I struggle to understand how terminal priorities work. With the Python grammar and the following code:

import lark
from lark.indenter import PythonIndenter

parser = lark.Lark.open_from_package(
    'lark',
    'python.lark',
     ['grammars'],
    parser='lalr',
    postlex=PythonIndenter(),
    start='file_input'
)

python_code = """_
"""
print(
    [(x.type, x.value) for x in parser.lex(python_code)],
    "\n\n",
    parser.parse(python_code)
)

... the output is:

[('UNDERSCORE', '_'), ('_NEWLINE', '\n')] 

 Tree(Token('RULE', 'file_input'), [Tree(Token('RULE', 'expr_stmt'), [Tree('var', [Tree(Token('RULE', 'name'), [Token('NAME', '_')])])])])

My understanding was that the lexer converts the string into a sequence of tokens which is then processed by the parser. But then how is it possible that the output syntax tree interprets _ as a NAME while the lexer interprets _ as an UNDERSCORE?

[erezsh]

The second line works because when parsing, Lark can use information from the parser to help the lexer choose the correct terminal, using the "contextual lexer".

This information isn't available when just lexing, so it chooses the more specific terminal, without regard for context.

[vivien000]

Thanks Erez. That's very clear.