New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add "So many tokens, so little time" #6211

Merged

bennyvasquez merged 1 commit into rust-lang:master from look:look/tokens

Dec 19, 2024

Contributor

look commented Dec 19, 2024 •

edited

Loading

This is a blog post about a new byte-pair tokenizer written in Rust.

The source for the crates themselves is in the bpe and bpe-openai directories.


          Add "So many tokens, so little time"

This is a blog post about a new byte-pair tokenizer written in Rust.

Contributor

bennyvasquez commented Dec 19, 2024

Thank you!

bennyvasquez merged commit 6c1b379 into rust-lang:master

1 check passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet