Skip to content

Commit

Permalink
Update tokenizers (#1096)
Browse files Browse the repository at this point in the history 
To pick up
openvinotoolkit/openvino_tokenizers#305
  • Loading branch information
@ilya-lavrenov
ilya-lavrenov authored Oct 29, 2024
1 parent a038da2 commit 0eda83a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion thirdparty/openvino_tokenizers
Submodule openvino_tokenizers updated 12 files
+1 −1 .github/workflows/linux.yml
+3 −3 .github/workflows/mac.yml
+1 −1 .github/workflows/sdl.yml
+3 −3 .github/workflows/windows.yml
+6 −4 src/charsmap_normalization.cpp
+3 −3 src/ov_extension.cpp
+1 −1 src/regex_split.cpp
+4 −1 src/sentence_piece.cpp
+67 −67 src/sentence_piece.hpp
+150 −148 src/string_to_hash_bucket.cpp
+21 −20 src/tensorflow_translators.cpp
+1 −1 src/tensorflow_translators.hpp

0 comments on commit 0eda83a

Please sign in to comment.