Skip to content

Commit

Permalink
OV Tokenizers
Browse files Browse the repository at this point in the history
  • Loading branch information
@iefode
iefode committed Jul 31, 2024
1 parent 5cf41eb commit dc4fff8
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Submodule openvino_tokenizers updated 22 files
+125 −12 .github/workflows/windows.yml
+1 −0 .gitignore
+80 −19 README.md
+17 −6 python/openvino_tokenizers/cli.py
+2 −0 python/openvino_tokenizers/convert_tokenizer.py
+47 −11 python/openvino_tokenizers/hf_parser.py
+6 −2 python/openvino_tokenizers/tiktoken_parser.py
+22 −4 python/openvino_tokenizers/tokenizer_pipeline.py
+34 −5 src/CMakeLists.txt
+8 −0 src/icu_filter_en.json
+71 −27 src/regex_normalization.cpp
+5 −1 src/regex_normalization.hpp
+93 −21 src/sentence_piece.cpp
+15 −2 src/sentence_piece.hpp
+71 −0 src/utils.cpp
+10 −1 src/utils.hpp
+4 −2 src/vocab_decoder.cpp
+0 −10 tests/conftest.py
+1 −1 tests/pass_rates.json
+2,298 −2,154 tests/stats.json
+16 −12 tests/tokenizers_test.py
+96 −0 third-party-programs.txt

0 comments on commit dc4fff8

Please sign in to comment.