Skip to content

Commit

Permalink
update submodule
Browse files Browse the repository at this point in the history
  • Loading branch information
@Wovchena
Wovchena committed Jul 15, 2024
1 parent 284cd03 commit dbb94da
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Submodule openvino_tokenizers updated 27 files
+2 −0 .github/workflows/labeler.yml
+16 −5 .github/workflows/linux.yml
+5 −1 .github/workflows/mac.yml
+5 −2 .github/workflows/windows.yml
+13 −4 CMakeLists.txt
+169 −126 README.md
+4 −3 pyproject.toml
+11 −0 python/openvino_tokenizers/constants.py
+9 −1 python/openvino_tokenizers/convert_tokenizer.py
+169 −50 python/openvino_tokenizers/hf_parser.py
+1 −17 python/openvino_tokenizers/tiktoken_parser.py
+91 −58 python/openvino_tokenizers/tokenizer_pipeline.py
+52 −7 python/openvino_tokenizers/utils.py
+12 −4 src/CMakeLists.txt
+52 −0 src/byte_fallback.cpp
+36 −0 src/byte_fallback.hpp
+1 −1 src/icu4c.patch
+1 −0 src/ov_extension.cpp
+3 −3 src/ragged_to_dense.cpp
+28 −21 src/sentence_piece.cpp
+2 −0 src/sentence_piece.hpp
+1 −0 src/tokenizer.hpp
+21 −20 src/vocab_decoder.cpp
+47 −1 tests/conftest.py
+1 −1 tests/pass_rates.json
+12,885 −0 tests/stats.json
+115 −69 tests/tokenizers_test.py

0 comments on commit dbb94da

Please sign in to comment.