[Feature]Add Model-Level Rate Limiting

[SiYue-ZO]

🎯 The Goal / Use Case

In real-world usage, different models and providers (e.g., OpenAI, Anthropic, or self-hosted models) enforce different rate limits.

Currently, PicoClaw does not provide a built-in way to control request rates at the model level, which may lead to:

• Frequent rate limit exceeded errors
• Unstable gateway behavior under high concurrency
• Difficulty managing multi-model or multi-tenant workloads

A common use case is to limit a specific model to a fixed rate, such as 50 requests per minute (RPM).

💡 Proposed Solution

Introduce model-level rate limiting configuration, allowing users to define limits per model.

Suggested capabilities:

• Configure RPM (requests per minute) per model (e.g., 50 RPM)
• Optionally support TPM (tokens per minute)
• Apply rate limiting automatically in the gateway before forwarding requests to providers

Example configuration:

{
  "model": "gpt-4",
  "rate_limit": {
    "rpm": 50,
    "tpm": 10000
  }
}

🛠 Potential Implementation (Optional)

• Implement a token bucket or leaky bucket rate limiter per model
• Maintain a rate limiter map keyed by model name in the gateway layer
• Apply limiting before dispatching requests to the provider

Possible structure in Go:

• RateLimiterManager (map[model]limiter)
• Middleware in gateway request pipeline
• Config-driven initialization
  🚦 Impact & Roadmap Alignment
• This is a Core Feature
• This is a Nice-to-Have / Enhancement
• This aligns with the current Roadmap

🔄 Alternatives Considered

• External rate limiting (e.g., Nginx, API Gateway)
• Not flexible for per-model control
• Client-side throttling
• Hard to maintain and not centralized
  💬 Additional Context
• Many providers enforce strict rate limits, making this feature essential for stable operation
• This would greatly improve reliability in multi-model and production deployments
• Could be extended in the future to support:
  • Per-user or per-API-key rate limiting
  • Burst control and priority queues

[Alix-007]

Thanks for the proposal. We confirmed this is a real product gap, but it is not a narrow low-risk PR lane for the current fill batch.

Today, model_list[*].rpm is already exposed in config / API / UI, but the runtime request path still does not enforce it. A safe implementation needs coordinated changes across model resolution, fallback / multi-key routing, limiter lifecycle, and tests.

There is also active overlap risk with open PR #1697 in the same fallback/cooldown subsystem, so opening another behavioral PR here right now would raise integration risk.

The best next step is to reopen this as a scoped Phase 1 after #1697 lands: RPM-only enforcement, explicit model-scoped limiter keys, and targeted tests for fallback + multi-key + routing behavior. If you want to carry that narrower lane later, please do.