Skip to content

Commit 0da7039

Browse files
lapp0rlouf
authored andcommitted
document third party exllamav2 with logits processor
1 parent faa7c5c commit 0da7039

File tree

2 files changed

+12
-2
lines changed

2 files changed

+12
-2
lines changed

docs/reference/models/exllamav2.md

+9-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,15 @@
11
# ExllamaV2
22

3+
The `outlines.models.exllamav2` model requires a Logits Processor component for compatibility with Outlines structured generation. While ExLlamaV2 doesn't natively support this feature, a third-party fork provides the necessary functionality. You can install it with the following command:
4+
5+
```bash
6+
pip install git+https://github.com/lapp0/exllamav2@sampler-logits-processor
7+
```
8+
9+
Install other requirements:
10+
311
```bash
4-
pip install exllamav2 transformers torch
12+
pip install transformers torch
513
```
614

715
*Coming soon*

outlines/models/exllamav2.py

+3-1
Original file line numberDiff line numberDiff line change
@@ -300,7 +300,9 @@ def exl2(
300300

301301
except ImportError:
302302
raise ImportError(
303-
"The `exllamav2`, `transformers` and `torch` libraries needs to be installed in order to use `exllamav2` models."
303+
"The `exllamav2`, `transformers` and `torch` libraries needs to be installed in order to use `exllamav2` models. "
304+
"Please run `pip install transformers torch git+https://github.com/lapp0/exllamav2@sampler-logits-processor` "
305+
"Documentation: https://dottxt-ai.github.io/outlines/reference/models/exllamav2/"
304306
)
305307
config = ExLlamaV2Config(model_path)
306308
if max_chunk_size is not None:

0 commit comments

Comments
 (0)