Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MLX model support #300

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Conversation

g-eoj
Copy link

@g-eoj g-eoj commented Jan 21, 2025

The goal of this PR is to enable users to run smolagents with models loaded onto Apple silicon with mlx-lm. The mlx-community has made available many models for experimentation. Personally I find running locally to be a convenient way to learn and experiment with the smolagents library, so I made this PR for a possible new feature.

Example usage:

from smolagents.models import MLXModel

mlx_model = MLXModel("mlx-community/Qwen2.5-Coder-32B-Instruct-4bit", max_tokens=10000)
messages = [{"role": "user", "content": "Explain quantum mechanics in simple terms."}]
print(mlx_model(messages))

Some questions:

  • tests won't work for CICD due to hardware requirements, what is the preferred way to handle that?
  • anything needed for docs and if so where should it go?

@g-eoj g-eoj force-pushed the g-eoj/mlx-model-support branch from b193f27 to 662a481 Compare January 22, 2025 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant