Skip to content

Commit

Permalink
document third party exllamav2 with logits processor
Browse files Browse the repository at this point in the history
  • Loading branch information
lapp0 committed Oct 5, 2024
1 parent 6a7eb90 commit a59c26f
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 2 deletions.
10 changes: 9 additions & 1 deletion docs/reference/models/exllamav2.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
# ExllamaV2

The `outlines.models.exllamav2` model requires a Logits Processor component for compatibility with Outlines structured generation. While ExLlamaV2 doesn't natively support this feature, a third-party fork provides the necessary functionality. You can install it with the following command:

```bash
pip install git+https://github.com/lapp0/exllamav2@sampler-logits-processor
```

Install other requirements:

```bash
pip install exllamav2 transformers torch
pip install transformers torch
```

*Coming soon*
4 changes: 3 additions & 1 deletion outlines/models/exllamav2.py
Original file line number Diff line number Diff line change
Expand Up @@ -300,7 +300,9 @@ def exl2(

except ImportError:
raise ImportError(
"The `exllamav2`, `transformers` and `torch` libraries needs to be installed in order to use `exllamav2` models."
"The `exllamav2`, `transformers` and `torch` libraries needs to be installed in order to use `exllamav2` models. "
"Please run `pip install transformers torch git+https://github.com/lapp0/exllamav2@sampler-logits-processor` "
"Documentation: https://dottxt-ai.github.io/outlines/reference/models/exllamav2/"
)
config = ExLlamaV2Config(model_path)
if max_chunk_size is not None:
Expand Down

0 comments on commit a59c26f

Please sign in to comment.