Add Llama.cpp model capability #450
Open
+184
−10
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #449
Description
This pull request introduces the
LlamaCppModel
class to the smolagents library, enabling seamless integration withllama.cpp
models. This enhancement expands the library's versatility, allowing users to leverage optimized performance and efficiency offered byllama.cpp
for running large language models.Features Added:
LlamaCppModel Class:
A new class to interact with
llama.cpp
models, supporting both local model loading and Hugging Face repository integration.Parameter Handling:
Comprehensive parameter management, including GPU layers, context size, and maximum token generation.
Conditional Tool Integration:
Tools are integrated only when explicitly provided, ensuring optimized resource utilization.
Motivation and Context
Integrating
llama.cpp
models into smolagents addresses the growing demand for efficient and resource-optimized language model interactions. This addition allows users to benefit fromllama.cpp
's capabilities while maintaining the flexibility and functionality that smolagents offers.How Has This Been Tested?
Tested the
LlamaCppModel
within aCodeAgent
to ensure seamless interaction and tool usage.Example Usage from text_to_sql.py: