Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backend] Add Llamacpp backend #2975

Merged
merged 63 commits into from
Feb 14, 2025
Merged

[Backend] Add Llamacpp backend #2975

merged 63 commits into from
Feb 14, 2025

Conversation

angt
Copy link
Member

@angt angt commented Jan 31, 2025 •
edited
Loading

This PR adds support for the llamacpp backend.
The Dockerfile_llamacpp enables native CPU execution as well as CUDA acceleration for GPUs.

For setup and usage details, you can check the doc.

@Cynn1989

This comment was marked as resolved.

Sign in to view
@mfuntowicz mfuntowicz changed the title Add Llamacpp backend [Backend] Add Llamacpp backend Feb 3, 2025
angt and others added 26 commits February 4, 2025 13:32
@angt
Add llamacpp backend
95e221e 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Get rid of llama_batch_get_one()
bd0cc99 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Use max_batch_total_tokens
3eb4823 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Handle max_batch_size
e7facf6 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add some input validation checks
a7b4b04 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Handle ctx args & fix sampling
8d2dfdf 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add GPU args
f388747 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add --defrag-threshold
e07835c 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add a stupid batch mechanism
d6ded89 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Cleanup
390f0ec 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add --numa
7a3ed41 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix args
3f19913 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Enable flash attention by default
ae5bb78 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add --offload-kqv
e88a527 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix batch_pos
f38c34a 
Signed-off-by: Adrien Gallouët <[email protected]>
@mfuntowicz @angt
backend(llama): add CUDA Dockerfile_llamacpp for now
960c12b
@angt
Only export the latest logits
161280f 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Output real logprobs
2a51e41 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix batching
96434a1 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix seq iterations
27534d8 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Auto-detect n_threads when not provided
c8505fb 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Clear request cache after completion
8ed362d 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Remove warmup
104a968 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Cleanup
ea28332 
Signed-off-by: Adrien Gallouët <[email protected]>
@mfuntowicz @angt
backend(llama): add CUDA architectures build argument for Dockerfile
e6a8d33
@angt
Add specific args for batch
bfb8e03 
Signed-off-by: Adrien Gallouët <[email protected]>
angt added 3 commits February 6, 2025 10:57
@angt
Thanks clippy
fb81c0d 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Thanks cargo fmt
2b0d99c 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Update docs
8bc10d3 
Signed-off-by: Adrien Gallouët <[email protected]>
@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@angt angt requested review from mfuntowicz and Hugoch February 6, 2025 10:40
@angt angt marked this pull request as ready for review February 6, 2025 10:43
angt added 4 commits February 6, 2025 13:17
@angt
Do not use HOSTNAME env
7bff88b 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Bump llama.cpp & cuda
df723e6 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix requirements.txt
5367d94 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix fmt
809e288 
Signed-off-by: Adrien Gallouët <[email protected]>
mfuntowicz
mfuntowicz previously approved these changes Feb 6, 2025
View reviewed changes
angt added 2 commits February 6, 2025 18:33
@angt
Enable KQV offload by default
3b1b049 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Remove Ngrok tunneling
acca9c3 
Signed-off-by: Adrien Gallouët <[email protected]>
angt added 10 commits February 7, 2025 08:51
@angt
Remove .cargo/config.toml
0d27ee7 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix Dockerfile
4841f71 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add missing cuda prefix
b6cfa0f 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Handle custom llama.cpp dir
6bdb644 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Cleanup
0702e0b 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add README.md
508d47f 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Add HF transfer
1401418 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Fix bool args
b77d05d 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Update doc
d96a777 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt
Update doc
5fb4afb 
Signed-off-by: Adrien Gallouët <[email protected]>
@angt angt requested a review from mfuntowicz February 14, 2025 10:28
@angt angt merged commit cfd4fbb into huggingface:main Feb 14, 2025
12 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hugoch Hugoch Hugoch approved these changes

@mfuntowicz mfuntowicz mfuntowicz approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@angt @Cynn1989 @HuggingFaceDocBuilderDev @Hugoch @mfuntowicz