Update tokenization section

TI-Toolkit · Dec 29, 2023 · f40f123 · f40f123
1 parent 2a6feb4
commit f40f123
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/README.md b/README.md
@@ -240,7 +240,7 @@ img.show()
 
 ### Tokenization
 
-Functions to decode and encode strings into tokens can be found in `tivars.tokenizer`. Support currently exists for all models in the 82/83/84 series as well as the TI-73; PR's concerning the sheets themselves should be directed upstream to [TI-Toolkit/tokens](https://github.com/TI-Toolkit/tokens).
+Functions to decode and encode strings into tokens can be found in `tivars.tokenizer`. These functions utilize the [TI-Toolkit token sheets](https://github.com/TI-Toolkit/tokens), which are kept as a submodule in `tivars.tokens`. Support currently exists for all models in the 82/83/84 series; PR's concerning the sheets themselves should be directed upstream.
 
 > [!IMPORTANT]
 > In contrast to some other tokenizers like SourceCoder, tokenization does _not_ depend on whether the content appears inside a BASIC string literal. Text is always assigned to the _longest_ permissible token.