[feature](inverted index) Add ICU library to support ICU tokenization. (

apache#47323) ### What problem does this PR solve? Problem Summary: 1. To enhance the inverted index tokenization functionality, it is necessary to incorporate the ICU tokenizer.
siriume · Jan 24, 2025 · 61fba92 · 61fba92
1 parent eb4056a
commit 61fba92
Show file tree

Hide file tree

Showing 5 changed files with 574 additions and 0 deletions.
diff --git a/dist/LICENSE-dist.txt b/dist/LICENSE-dist.txt
@@ -1536,3 +1536,4 @@ Other dependencies:
     * xxhash: 0.8.1  -- licenses/LICENSE-xxhash.txt
     * concurrentqueue: 1.0.3  -- licenses/LICENSE-concurrentqueue.txt
     * FlameGraph -- licenses/LICENSE-CDDL-1.0.txt
+    * icu 75.1 -- licenses/LICENSE-icu.txt