Muxing Support for Model Routing

[lukehinds]

Introduce support for "Muxing" in CodeGate, enabling users to route specific types of logic to different large language models (LLMs) based on file type, individual files, or the context of a Git codebase. This would provide cost savings , a common complaint from users is the expense of tokens and how to optimise those costs.

For example:

• Per File Type: Documentation files (e.g., .md, .rst) could be processed by a free, local model, while source code files (e.g., .py, .js) are handled by an advanced model like Claude Sonnet 3.5.
• Per File: Users could specify models for particular files within a project based on complexity or other criteria.
• Per Repository: For one Git codebase, users might choose a local model for privacy reasons, while another repository could leverage a different model for advanced coding tasks.

Why is this feature important?

1. Granular Control: Developers can tune model usage based on file type or project needs.
2. Resource Efficiency: Allocate lightweight or local models to simpler tasks, while reserving advanced models for more complex challenges, providing more cost efficiency.
3. Git Integration: Aligns with the workflow of managing multiple repositories, providing seamless configuration across projects. See

NOTE: As always, start small, simple and validate, the following acts as a guideline of where this could lead.

Possible Solution

Expand CodeGate’s functionality to include:

1. File-Type-Based Routing:

   • Route tasks based on file extensions or MIME types.
   • Example: .md -> local_model, .py -> claude_sonnet_3.5.

2. Per-File Routing:

   • Allow specific files in a project to be linked to a designated model.
   • Example: config.py -> local_model_v2.

3. Repository-Level Configuration:

   • Enable users to set model preferences for an entire repository.
   • Example: Repository A -> local_model, Repository B -> advanced_model.

Challenges and Considerations

• Git Context Awareness: Ensure the muxing system can recognize the active repository and apply the appropriate routing.
• Conflict Resolution: Define how conflicts between repository-level and file-level rules are handled.
• Latency and Performance: Optimize routing logic to prevent delays in large or multi-repository setups.
• User Experience: Provide clear documentation and possibly a UI component for managing these configurations.

[lukehinds]

Moving this to ready, as I think we can at least start prototyping and thinking about this, albeit we should land #454 first to really make it useful.