Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop tokens. #38

Open
DiscreteTom opened this issue Dec 4, 2023 · 3 comments
Open

Drop tokens. #38

DiscreteTom opened this issue Dec 4, 2023 · 3 comments
Labels
enhancement New feature or request parser

Comments

@DiscreteTom
Copy link
Owner

Currently we store token in ASTNode.token and in Lexer.errors. But for most cases, these tokens are not used, which will waste a lot of memory.

For the ASTNode, a better way is to define some transformer to transform a token into a terminator ASTNode then we can drop the token.

For Lexer.errors, we can define a callback to receive those error tokens.

@DiscreteTom DiscreteTom added enhancement New feature or request parser labels Dec 4, 2023
@DiscreteTom
Copy link
Owner Author

But with re-lex, lexer callbacks might not accurate.

@DiscreteTom
Copy link
Owner Author

// add token transformer
parserBuilder.lexer(lexer, {
	t1: token => nodeData
})

@DiscreteTom
Copy link
Owner Author

For lexer.error, maybe we can batch process them in parser.commit? This is to prevent the inconsistency caused by re-lex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request parser
Projects
None yet
Development

No branches or pull requests

1 participant