Code

VLM: special multimodal Tokenizer#34461

Code

Merged

zucchini-nlp merged 24 commits intohuggingface:mainhuggingface/transformers:mainfrom zucchini-nlp:vlm-tokenizerzucchini-nlp/transformers:vlm-tokenizerCopy head branch name to clipboardNov 4, 2024

+247-334

Commits

Commits on Oct 5, 2024

• kinda works

  zucchini-nlp

  committed
  d4778c9

  Copy full SHA for d4778c9

Commits on Oct 7, 2024

• update

  zucchini-nlp

  committed
  0ba73ec

  Copy full SHA for 0ba73ec

• add tests

  zucchini-nlp

  committed
  116a49e

  Copy full SHA for 116a49e

Commits on Oct 8, 2024

• update

  zucchini-nlp

  committed
  661b5df

  Copy full SHA for 661b5df

Commits on Oct 18, 2024

• use special tokens in processors

  zucchini-nlp

  committed
  5ed9379

  Copy full SHA for 5ed9379

Commits on Oct 28, 2024

• typo

  zucchini-nlp

  committed
  a284f31

  Copy full SHA for a284f31

• Merge remote-tracking branch 'upstream/main' into vlm-tokenizer

  zucchini-nlp

  committed
  9ad5362

  Copy full SHA for 9ad5362

• fix copies

  zucchini-nlp

  committed
  1def73c

  Copy full SHA for 1def73c

• fix

  zucchini-nlp

  committed
  8dfa536

  Copy full SHA for 8dfa536

• fix moshi after rebase

  zucchini-nlp

  committed
  200879b

  Copy full SHA for 200879b

• Merge branch 'main' into vlm-tokenizer

  zucchini-nlp

  authored
  e0bf53b

  Copy full SHA for e0bf53b

Commits on Oct 29, 2024

• update

  zucchini-nlp

  committed
  7e5c4ba

  Copy full SHA for 7e5c4ba

• Merge branch 'main' into vlm-tokenizer

  zucchini-nlp

  authored
  b77f3be

  Copy full SHA for b77f3be

• fix tests

  zucchini-nlp

  committed
  8b61969

  Copy full SHA for 8b61969

Commits on Oct 30, 2024

• update

  zucchini-nlp

  committed
  276c55e

  Copy full SHA for 276c55e

• Update docs/source/en/main_classes/tokenizer.md

  Show description for 3177519
  

  zucchini-nlp

  and

  ArthurZucker

  authored
  3177519

  Copy full SHA for 3177519

• update docs

  zucchini-nlp

  committed
  46a30e5

  Copy full SHA for 46a30e5

• test for load time adding tokens

  zucchini-nlp

  committed
  d7c4eb5

  Copy full SHA for d7c4eb5

Commits on Oct 31, 2024

• Merge branch 'main' into vlm-tokenizer

  zucchini-nlp

  authored
  f3b102e

  Copy full SHA for f3b102e

Commits on Nov 1, 2024

• Merge branch 'main' into vlm-tokenizer

  zucchini-nlp

  authored
  5240129

  Copy full SHA for 5240129

• fix some more tests which are now fetched better

  zucchini-nlp

  committed
  58bf9e7

  Copy full SHA for 58bf9e7

• one more fix

  zucchini-nlp

  committed
  76b39aa

  Copy full SHA for 76b39aa

• Merge branch 'main' into vlm-tokenizer

  zucchini-nlp

  authored
  9994aed

  Copy full SHA for 9994aed

• Merge branch 'main' into vlm-tokenizer

  zucchini-nlp

  authored
  c21997d

  Copy full SHA for c21997d