Skip to content

Commit

Permalink
Update
Browse files Browse the repository at this point in the history
Signed-off-by: DarkLight1337 <[email protected]>
  • Loading branch information
DarkLight1337 committed Dec 2, 2024
1 parent aef7899 commit 5d3a629
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 5 deletions.
6 changes: 3 additions & 3 deletions docs/source/usage/compatibility_matrix.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Feature x Feature
- :abbr:`prmpt adptr (Prompt Adapter)`
- :ref:`SD <spec_decode>`
- CUDA graph
- :abbr:`emd (Embedding Models)`
- :abbr:`pooling (Pooling Models)`
- :abbr:`enc-dec (Encoder-Decoder Models)`
- :abbr:`logP (Logprobs)`
- :abbr:`prmpt logP (Prompt Logprobs)`
Expand Down Expand Up @@ -151,7 +151,7 @@ Feature x Feature
-
-
-
* - :abbr:`emd (Embedding Models)`
* - :abbr:`pooling (Pooling Models)`
- ✗
- ✗
- ✗
Expand Down Expand Up @@ -386,7 +386,7 @@ Feature x Hardware
- ✅
- ✗
- ✅
* - :abbr:`emd (Embedding Models)`
* - :abbr:`pooling (Pooling Models)`
- ✅
- ✅
- ✅
Expand Down
8 changes: 6 additions & 2 deletions docs/source/usage/pooling_models.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,19 @@
Using Pooling Models
====================

vLLM provides second-class support for pooling models, including embedding, reranking and reward models.
vLLM also supports pooling models, including embedding, reranking and reward models.

In vLLM, pooling models implement the :class:`~vllm.model_executor.models.VllmModelForPooling` interface.
These models use a :class:`~vllm.model_executor.layers.Pooler` to aggregate the final hidden states of the input
before returning them.

Technically, any :ref:`generative model <generative_models>` in vLLM can be converted into a pooling model
by aggregating and returning the hidden states directly, skipping the generation step.
Nevertheless, you should use those that are specifically trained as pooling models.
Nevertheless, to get the best results, you should use pooling models that are specifically trained as such.

We currently support pooling models primarily as a matter of convenience.
As shown in the :code:`Compatibility Matrix <compatibility_matrix>`, most vLLM features are not applicable to
pooling models as they only work on the generation or decode stage, so performance may not improve as much.

Offline Inference
-----------------
Expand Down

0 comments on commit 5d3a629

Please sign in to comment.