Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Llama.cpp model capability #450

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

ryantzr1
Copy link

@ryantzr1 ryantzr1 commented Jan 31, 2025

Closes #449

Description

This pull request introduces the LlamaCppModel class to the smolagents library, enabling seamless integration with llama.cpp models. This enhancement expands the library's versatility, allowing users to leverage optimized performance and efficiency offered by llama.cpp for running large language models.

Features Added:

  • LlamaCppModel Class:
    A new class to interact with llama.cpp models, supporting both local model loading and Hugging Face repository integration.

  • Parameter Handling:
    Comprehensive parameter management, including GPU layers, context size, and maximum token generation.

  • Conditional Tool Integration:
    Tools are integrated only when explicitly provided, ensuring optimized resource utilization.

Motivation and Context

Integrating llama.cpp models into smolagents addresses the growing demand for efficient and resource-optimized language model interactions. This addition allows users to benefit from llama.cpp's capabilities while maintaining the flexibility and functionality that smolagents offers.

How Has This Been Tested?

  • Integration Tests:
    Tested the LlamaCppModel within a CodeAgent to ensure seamless interaction and tool usage.

Example Usage from text_to_sql.py:

from smolagents import LlamaCppModel, CodeAgent
from smolagents.tools import SQLTool  # Assume SQLTool is predefined

# Initialize the SQL tool
sql_engine = SQLTool(...)

# Initialize the LlamaCppModel
model = LlamaCppModel(
    repo_id="bartowski/Qwen2.5-7B-Instruct-1M-GGUF",
    filename="Qwen2.5-7B-Instruct-1M-IQ2_M.gguf",
    n_ctx=8192,
    max_tokens=8192,
)

# Create the CodeAgent with the SQL tool and LlamaCppModel
agent = CodeAgent(
    tools=[sql_engine],
    model=model,
)

# Run the agent with a prompt
response = agent.run("Can you give me the name of the client who got the most expensive receipt?")
print(response.content)
# Output: "The client with the most expensive receipt is Woodrow Wilson."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature Request: Add LlamaCppModel Support to smolagents
1 participant