Support Unicode 15.1 for line segmenter #5218
Conversation
makotokato
requested review from
sffc,
robertbastian,
Manishearth,
aethanyc and
a team
as code owners
makotokato
requested review from
eggrobin,
aethanyc and
robertbastian
and removed request for
a team,
aethanyc,
sffc,
Manishearth and
robertbastian
| @@ -19,97 +19,111 @@ const UNKNOWN: u8 = 0; | |||
| #[allow(dead_code)] | |||
| const AI: u8 = 1; | |||
| #[allow(dead_code)] | |||
| const AL: u8 = 2; | |||
| const AK: u8 = 2; | |||
There was a problem hiding this comment.
This probably breaks compatibility with current data structs, so you need to bump the version.
There was a problem hiding this comment.
I have already bumped to V2 for 2.0. Should I bump to V3?
There was a problem hiding this comment.
V2 isn't released yet so it's fine as long as long as this only affects V2 structs (looks like dictionary and LSTM are still on V1)
|
r? @eggrobin for the algorithm |
robertbastian
added
the
waiting-on-author
|
I addressed the comments I had made above, moving the LB15a-specific logic into the state table (it is not exactly pretty, we should probably invest in some extensions to the state table description language; but it is nowhere near as bad as the ZWJ-CM situation was). |
Maybe we should be using something other than |
|
Now tested with 2 000 000 random strings. Before ICU 76 (specifically, before unicode-org/icu#3028) it doesn’t make sense to go further than that, since it turned out the PRNG was cycling around that length (see L2/24-162 §5.6). |
eggrobin
removed
the
waiting-on-author
This fix supports Unicode 15.1 for line segmenter. (a part of #3255)