We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Currently we store token in ASTNode.token and in Lexer.errors. But for most cases, these tokens are not used, which will waste a lot of memory.
ASTNode.token
Lexer.errors
For the ASTNode, a better way is to define some transformer to transform a token into a terminator ASTNode then we can drop the token.
ASTNode
For Lexer.errors, we can define a callback to receive those error tokens.
The text was updated successfully, but these errors were encountered:
But with re-lex, lexer callbacks might not accurate.
Sorry, something went wrong.
// add token transformer parserBuilder.lexer(lexer, { t1: token => nodeData })
feat(parser)!: add IParserBuilder.mapper, #38
bc048c1
tests(parser): add token-ASTData-mapper.test, #38
271c8eb
For lexer.error, maybe we can batch process them in parser.commit? This is to prevent the inconsistency caused by re-lex.
lexer.error
parser.commit
No branches or pull requests
Currently we store token in
ASTNode.token
and inLexer.errors
. But for most cases, these tokens are not used, which will waste a lot of memory.For the
ASTNode
, a better way is to define some transformer to transform a token into a terminator ASTNode then we can drop the token.For
Lexer.errors
, we can define a callback to receive those error tokens.The text was updated successfully, but these errors were encountered: