Code

Add ColPali to 🤗 transformers#33736

Code

Merged

ArthurZucker merged 137 commits intohuggingface:mainhuggingface/transformers:mainfrom tonywu71:add-colpalitonywu71/transformers:add-colpaliCopy head branch name to clipboardDec 17, 2024

+2,204-2

Commits

Commits on Nov 28, 2024

• feat: run add-new-model-like

  tonywu71

  committed
  0f5c6a7

  Copy full SHA for 0f5c6a7

• feat: add paligemma code with "copied from"

  tonywu71

  committed
  726f156

  Copy full SHA for 726f156

• feat: add ColPaliProcessor

  tonywu71

  committed
  9a88bf1

  Copy full SHA for 9a88bf1

• feat: add ColPaliModel

  tonywu71

  committed
  fab4e46

  Copy full SHA for fab4e46

• feat: add ColPaliConfig

  tonywu71

  committed
  66656f6

  Copy full SHA for 66656f6

• feat: rename ColPaliForConditionalGeneration to ColPaliModel

  tonywu71

  committed
  a377d60

  Copy full SHA for a377d60

• fixup modeling colpali

  tonywu71

  committed
  0addcab

  Copy full SHA for 0addcab

• fix: fix root import shortcuts

  tonywu71

  committed
  e8979b9

  Copy full SHA for e8979b9

• fix: fix modeling_auto dict

  tonywu71

  committed
  49fb8ba

  Copy full SHA for 49fb8ba

• feat: comment out ColPali test file

  tonywu71

  committed
  88b0212

  Copy full SHA for 88b0212

• fix: fix typos from add-new-model-like

  tonywu71

  committed
  cbd781b

  Copy full SHA for cbd781b

• feat: explicit the forward input args

  tonywu71

  committed
  44fcd04

  Copy full SHA for 44fcd04

• feat: move everything to modular_colpali.py

  tonywu71

  committed
  a6ca45a

  Copy full SHA for a6ca45a

• fix: put back ColPaliProcesor

  tonywu71

  committed
  af9ca36

  Copy full SHA for af9ca36

• feat: add auto-generated files

  tonywu71

  committed
  087870b

  Copy full SHA for 087870b

• fix: run fix-copies

  tonywu71

  committed
  cc11ef8

  Copy full SHA for cc11ef8

• fix: remove DOCStRING constants to make modular converter work

  tonywu71

  committed
  f69ee9b

  Copy full SHA for f69ee9b

• fix: fix typo + modular converter

  tonywu71

  committed
  fbe5665

  Copy full SHA for fbe5665

• fix: add missing imports

  tonywu71

  committed
  e58794c

  Copy full SHA for e58794c

• feat: no more errors when loading ColPaliModel

  tonywu71

  committed
  2dd5218

  Copy full SHA for 2dd5218

• fix: remove unused args in forward + tweak doc

  tonywu71

  committed
  e05ea43

  Copy full SHA for e05ea43

• feat: rename ColPaliModel to ColPaliForRetrieval

  tonywu71

  committed
  bda6916

  Copy full SHA for bda6916

• fix: apply fix-copies

  tonywu71

  committed
  bfff564

  Copy full SHA for bfff564

• feat: add ColPaliProcessor to modular_colpali

  tonywu71

  committed
  da4c566

  Copy full SHA for da4c566

• fix: run make quality + make style

  tonywu71

  committed
  ae37f18

  Copy full SHA for ae37f18

• fix: remove duplicate line in configuration_auto

  tonywu71

  committed
  38f0d8c

  Copy full SHA for 38f0d8c

• feat: make ColPaliModel inehrit from PaliGemmaForConditionalGeneration

  tonywu71

  committed
  c63a302

  Copy full SHA for c63a302

• fix: tweak and use ColPaliConfig

  tonywu71

  committed
  d66606e

  Copy full SHA for d66606e

• feat: rename score to post_process_retrieval

  tonywu71

  committed
  7f750d3

  Copy full SHA for 7f750d3

• build: run modular formatter + make style

  tonywu71

  committed
  41dbbb8

  Copy full SHA for 41dbbb8

• feat: convert colpali weights + fixes

  tonywu71

  committed
  28592c9

  Copy full SHA for 28592c9

• feat: remove old weight converter file

  tonywu71

  committed
  84763a3

  Copy full SHA for 84763a3

• feat: add and validate tests

  tonywu71

  committed
  672bdb2

  Copy full SHA for 672bdb2

• feat: replace harcoded path to "vidore/colpali-v1.2-hf" in tests

  tonywu71

  committed
  f7ce9b1

  Copy full SHA for f7ce9b1

• fix: add bfloat16 conversion in weight converter

  tonywu71

  committed
  3789a6e

  Copy full SHA for 3789a6e

• feat: replace pytest with unittest in modeling colpali test

  tonywu71

  committed
  5e09645

  Copy full SHA for 5e09645

• feat: add sanity check for weight conversion (doesn't work yet)

  tonywu71

  committed
  8ea8273

  Copy full SHA for 8ea8273

• feat: add shape sanity check in weigth converter

  tonywu71

  committed
  d100779

  Copy full SHA for d100779

• feat: make ColPaliProcessor args explicit

  tonywu71

  committed
  e6bdf40

  Copy full SHA for e6bdf40

• doc: add doc for ColPali

  tonywu71

  committed
  abe3232

  Copy full SHA for abe3232

• fix: trying to fix output mismatch

  tonywu71

  committed
  6ae178c

  Copy full SHA for 6ae178c

• feat: tweaks

  tonywu71

  committed
  6d35b27

  Copy full SHA for 6d35b27

• fix: ColPaliModelOutput inherits from ModelOutput instead of PaliGemmaCausalLMOutputWithPast

  tonywu71

  committed
  0653340

  Copy full SHA for 0653340

• fix: address comments on PR

  tonywu71

  committed
  97a6468

  Copy full SHA for 97a6468

• fix: adapt tests to the Hf norm

  tonywu71

  committed
  8212717

  Copy full SHA for 8212717

• wip: try things

  tonywu71

  committed
  a7b297a

  Copy full SHA for a7b297a

• feat: add __call__ method to ColPaliProcessor

  tonywu71

  committed
  592e716

  Copy full SHA for 592e716

• feat: remove need for dummy image in process_queries

  tonywu71

  committed
  f50a979

  Copy full SHA for f50a979

• build: run new modular converter

  tonywu71

  committed
  25eb21b

  Copy full SHA for 25eb21b

• fix: fix incorrect method override

  tonywu71

  committed
  3ed7627

  Copy full SHA for 3ed7627

• Fix tests, processing, modular, convert

  

  yonigozlan

  authored and

  tonywu71

  committed
  9038ead

  Copy full SHA for 9038ead

• fix tokenization auto

  

  yonigozlan

  authored and

  tonywu71

  committed
  cb7e301

  Copy full SHA for cb7e301

• hotfix: manually fix processor -> fixme once convert modular is fixed

  tonywu71

  committed
  3f118ca

  Copy full SHA for 3f118ca

• fix: convert weights working

  tonywu71

  committed
  3aa11a6

  Copy full SHA for 3aa11a6

• feat: rename and improve convert weight script

  tonywu71

  committed
  8ff8962

  Copy full SHA for 8ff8962

• feat: tweaks

  tonywu71

  committed
  7a54fec

  Copy full SHA for 7a54fec

• fest: remove device input for post_process_retrieval

  tonywu71

  committed
  2c94eaa

  Copy full SHA for 2c94eaa

• refactor: remove unused get_torch_device

  tonywu71

  committed
  2d7e96f

  Copy full SHA for 2d7e96f

• Fix all tests

  

  yonigozlan

  authored and

  tonywu71

  committed
  1189340

  Copy full SHA for 1189340

• docs: update ColPali model doc

  tonywu71

  committed
  246b67e

  Copy full SHA for 246b67e

• wip: fix convert weights to hf

  tonywu71

  committed
  4a5bc0c

  Copy full SHA for 4a5bc0c

• fix logging modular

  

  yonigozlan

  authored and

  tonywu71

  committed
  afbbc98

  Copy full SHA for afbbc98

• docs: add acknowledgements in model doc

  tonywu71

  committed
  9db013d

  Copy full SHA for 9db013d

• docs: add missing docstring to ColPaliProcessor

  tonywu71

  committed
  c4e156c

  Copy full SHA for c4e156c

• docs: tweak

  tonywu71

  committed
  0b4e089

  Copy full SHA for 0b4e089

• docs: add doc for ColPaliForRetrievalOutput.forward

  tonywu71

  committed
  d6a0bde

  Copy full SHA for d6a0bde

• feat: add modifications from colpali-engine v0.3.2 in ColPaliProcessor

  tonywu71

  committed
  1f115f9

  Copy full SHA for 1f115f9

• fix: fix and upload colapli hf weights

  tonywu71

  committed
  20d1927

  Copy full SHA for 20d1927

• refactor: rename post_process_retrieval to score_retrieval

  tonywu71

  committed
  5ef48fb

  Copy full SHA for 5ef48fb

• fix: fix wrong typing for score_retrieval

  tonywu71

  committed
  5ae2bac

  Copy full SHA for 5ae2bac

• test: add integration test for ColPali

  tonywu71

  committed
  ffe894a

  Copy full SHA for ffe894a

• chore: rerun convert modular

  tonywu71

  committed
  b0e33be

  Copy full SHA for b0e33be

• build: fix root imports

  tonywu71

  committed
  f052927

  Copy full SHA for f052927

• Update docs/source/en/index.md

  Show description for ad09d67
  

  tonywu71

  and

  yonigozlan

  committed
  ad09d67

  Copy full SHA for ad09d67

• fix: address PR comments

  tonywu71

  committed
  0dd1524

  Copy full SHA for 0dd1524

• wip: reduce the prediction gap in weight conversion

  tonywu71

  committed
  b647788

  Copy full SHA for b647788

• docs: add comment in weight conversion script

  tonywu71

  committed
  153f339

  Copy full SHA for 153f339

• docs: add example for ColPaliForRetrieval.forward

  tonywu71

  committed
  97b3a24

  Copy full SHA for 97b3a24

• tests: change dataset path to the new one in hf-internal

  tonywu71

  committed
  a711fa7

  Copy full SHA for a711fa7

• fix: colpali weight conversion works

  tonywu71

  committed
  e9035d9

  Copy full SHA for e9035d9

• test: add fine-grained check for ColPali integration test

  tonywu71

  committed
  9f7299b

  Copy full SHA for 9f7299b

• fix: fix typos in convert weight script

  tonywu71

  committed
  43274d2

  Copy full SHA for 43274d2

• docs: move input docstring in a variable

  tonywu71

  committed
  f6e3155

  Copy full SHA for f6e3155

• fix: remove hardcoded torch device in test

  tonywu71

  committed
  da03264

  Copy full SHA for da03264

• fix: run the new modular refactor

  tonywu71

  committed
  930f91a

  Copy full SHA for 930f91a

• docs: fix python example for ColPali

  tonywu71

  committed
  db37344

  Copy full SHA for db37344

• feat: add option to choose score_retrieval's output dtype and device

  tonywu71

  committed
  e72c379

  Copy full SHA for e72c379

• docs: update doc for score_retrieval

  tonywu71

  committed
  5b11870

  Copy full SHA for 5b11870

• feat: add patch_size property in ColPali model

  tonywu71

  committed
  c53ffcb

  Copy full SHA for c53ffcb

• chore: run make fix-copies

  tonywu71

  committed
  5346292

  Copy full SHA for 5346292

• docs: update description for ColPali cookbooks

  tonywu71

  committed
  b102738

  Copy full SHA for b102738

• fix: remove ignore_index methods

  tonywu71

  committed
  1d24773

  Copy full SHA for 1d24773

• feat: remove non-transformers specific methods

  tonywu71

  committed
  73d607a

  Copy full SHA for 73d607a

• feat: update __init__.py to new hf format

  tonywu71

  committed
  1db4c6c

  Copy full SHA for 1db4c6c

• fix: fix root imports in transformers

  tonywu71

  committed
  da05b70

  Copy full SHA for da05b70

• feat: remove ColPali's inheritance from PaliGemma

  tonywu71

  committed
  1b4f8f3

  Copy full SHA for 1b4f8f3

• Fix CI issues

  

  yonigozlan

  authored and

  tonywu71

  committed
  f100888

  Copy full SHA for f100888

• nit remove prints

  

  yonigozlan

  authored and

  tonywu71

  committed
  38210dc

  Copy full SHA for 38210dc

• feat: remove ColPali config and model from modular_colpali.py

  tonywu71

  committed
  aee8d7c

  Copy full SHA for aee8d7c

• feat: add ColPaliPreTrainedModel and update modeling and configuration code

  tonywu71

  committed
  f53ae20

  Copy full SHA for f53ae20

• fix: fix auto-removed imports in root __init__.py

  tonywu71

  committed
  b93c76b

  Copy full SHA for b93c76b

• fix: various fixes

  tonywu71

  committed
  87a16fd

  Copy full SHA for 87a16fd

• fix: fix _init_weight

  tonywu71

  committed
  fba3b77

  Copy full SHA for fba3b77

• temp: comment AutoModel.from_config for experiments

  tonywu71

  committed
  1e6c4ab

  Copy full SHA for 1e6c4ab

• fix: add missing output_attentions arg in ColPali's forward

  tonywu71

  committed
  6d20088

  Copy full SHA for 6d20088

• fix: fix resize_token_embeddings

  tonywu71

  committed
  be6a0bd

  Copy full SHA for be6a0bd

• fix: make input_ids optional in forward

  tonywu71

  committed
  ecc7982

  Copy full SHA for ecc7982

• feat: rename projection_layer to embedding_proj_layer

  tonywu71

  committed
  b1a25ce

  Copy full SHA for b1a25ce

• wip: fix convert colpali weight script

  tonywu71

  committed
  84fefad

  Copy full SHA for 84fefad

• fix tests and convert weights from original repo

  

  yonigozlan

  authored and

  tonywu71

  committed
  836dc97

  Copy full SHA for 836dc97

• fix unprotected import

  

  yonigozlan

  authored and

  tonywu71

  committed
  1eaa3d3

  Copy full SHA for 1eaa3d3

• fix unprotected torch import

  

  yonigozlan

  authored and

  tonywu71

  committed
  f187bc0

  Copy full SHA for f187bc0

• fix style

  

  yonigozlan

  authored and

  tonywu71

  committed
  3646790

  Copy full SHA for 3646790

• change vlm_backbone_config to vlm_config

  

  yonigozlan

  authored and

  tonywu71

  committed
  c8efb8a

  Copy full SHA for c8efb8a

• fix unprotected import in modular this time

  

  yonigozlan

  authored and

  tonywu71

  committed
  a30a74d

  Copy full SHA for a30a74d

• fix: load config from Hub + tweaks in convert weight script

  tonywu71

  committed
  c42c61b

  Copy full SHA for c42c61b

• docs: move example usage from model docstring to model markdown

  tonywu71

  committed
  e981b71

  Copy full SHA for e981b71

• docs: fix input docstring for ColPali's forward method

  tonywu71

  committed
  2ce28f5

  Copy full SHA for 2ce28f5

• fix: use sub_configs for ColPaliConfig

  tonywu71

  committed
  a582f48

  Copy full SHA for a582f48

• fix: remove non-needed sanity checks in weight conversion script + tweaks

  tonywu71

  committed
  9f34d80

  Copy full SHA for 9f34d80

• fix: fix issue with replace_return_docstrings in ColPali's forward

  tonywu71

  committed
  05c29da

  Copy full SHA for 05c29da

• docs: update docstring for ColPaliConfig

  tonywu71

  committed
  f67e217

  Copy full SHA for f67e217

• test: change model path in ColPali test

  tonywu71

  committed
  2ed868c

  Copy full SHA for 2ed868c

• fix: fix ColPaliConfig

  tonywu71

  committed
  2aa5e9d

  Copy full SHA for 2aa5e9d

• fix: fix weight conversion script

  tonywu71

  committed
  e6944ad

  Copy full SHA for e6944ad

• test: fix expected weights for ColPali model

  tonywu71

  committed
  337a0a0

  Copy full SHA for 337a0a0

• docs: update ColPali markdown

  tonywu71

  committed
  c10e760

  Copy full SHA for c10e760

• docs: fix minor typo in ColPaliProcessor

  tonywu71

  committed
  69d01fc

  Copy full SHA for 69d01fc

Commits on Nov 29, 2024

• Fix tests and add _no_split_modules

  yonigozlan

  committed
  8061469

  Copy full SHA for 8061469

Commits on Dec 5, 2024

• add text_config to colpali config

  yonigozlan

  committed
  7dce43f

  Copy full SHA for 7dce43f

• [run slow] colpali

  yonigozlan

  committed
  855f139

  Copy full SHA for 855f139

• Merge branch 'main' into add-colpali

  yonigozlan

  authored
  603e9e4

  Copy full SHA for 603e9e4

• move inputs to torch_device in integration test

  yonigozlan

  committed
  c41bad4

  Copy full SHA for c41bad4

• skip test_model_parallelism

  yonigozlan

  committed
  21c1309

  Copy full SHA for 21c1309

Commits on Dec 9, 2024

• docs: clarify quickstart snippet in ColPali's model card

  tonywu71

  committed
  505ad9e

  Copy full SHA for 505ad9e

Commits on Dec 10, 2024

• docs: update ColPali's model card

  tonywu71

  committed
  655bac7

  Copy full SHA for 655bac7

Commits on Dec 16, 2024

• Merge remote-tracking branch 'upstream/main' into add-colpali

  yonigozlan

  committed
  e9af3a5

  Copy full SHA for e9af3a5