Code

Add BPE tokenizers#62

Code

Open

angeloskath wants to merge 7 commits intomainml-explore/mlx-data:mainfrom bpeml-explore/mlx-data:bpeCopy head branch name to clipboard

+901-85

Commits

Commits on Apr 16, 2024

• Add a replace operation

  angeloskath

  committed
  bea27c5

Commits on Apr 17, 2024

• Start standardizing the SPM tokenization

  angeloskath

  committed
  3df150d

Commits on Apr 18, 2024

• Add a seemingly working bpe tokenizer

  angeloskath

  committed
  06add11

• Write BPE reading from spm model and fix BPETokenizer

  angeloskath

  committed
  4ebe4f5

• Add some docs and a small test

  angeloskath

  committed
  596ba14

• Add the BPE tokenize op

  angeloskath

  committed
  4e61bbb

• Add pointers to left and right in symbol

  angeloskath

  committed
  17d4960