v2.5

BBC-Esq · Oct 31, 2023 · c7582f0 · c7582f0
1 parent 0110a02
commit c7582f0
Show file tree

Hide file tree

Showing 4 changed files with 91,170 additions and 0 deletions.
diff --git a/src/Tokenizer/special_tokens_map.json b/src/Tokenizer/special_tokens_map.json
@@ -0,0 +1,5 @@
+{
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "unk_token": "<unk>"
+}