Skip to content

Commit

Permalink
Update tokenizers
Browse files Browse the repository at this point in the history
  • Loading branch information
@as-suvorov
as-suvorov committed Apr 9, 2024
1 parent c86f607 commit 62ba58a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Submodule openvino_tokenizers updated 22 files
+2 −1 .github/workflows/linux.yml
+2 −1 .github/workflows/mac.yml
+1 −0 .github/workflows/sdl.yml
+2 −1 .github/workflows/windows.yml
+3 −0 Jenkinsfile
+1 −1 pyproject.toml
+7 −5 python/openvino_tokenizers/__init__.py
+76 −0 python/openvino_tokenizers/build_tokenizer.py
+3 −1 python/openvino_tokenizers/cli.py
+7 −7 python/openvino_tokenizers/hf_parser.py
+1 −1 python/openvino_tokenizers/str_pack.py
+54 −6 python/openvino_tokenizers/tokenizer_pipeline.py
+4 −2 python/openvino_tokenizers/utils.py
+1 −1 requirements-build.txt
+40 −0 src/fuze.cpp
+35 −0 src/fuze.hpp
+3 −1 src/ov_extension.cpp
+68 −31 src/tensorflow_translators.cpp
+1 −1 src/tensorflow_translators.hpp
+2 −0 src/tokenizer.hpp
+111 −0 src/trie_tokenizer.cpp
+54 −0 src/trie_tokenizer.hpp

0 comments on commit 62ba58a

Please sign in to comment.