Add support for canonical encoding

[Saibo-creator]

Currently we allow all encodings that satiisfy the constraints, but actually the tokenizer has a canonical encoding, which i generally the shortest and with priority.
For example,
"aaaa" can be tokenized as ["aa", "aa"] or ["aaa", "a"] or ["a", "aaa"] or more.
Currently our constraints allow all of them but actually only one is canonical.
One flag can be added to enfore strict mode