Skip to content
@llm-d

llm-d

llm-d is a Kubernetes-native high-performance distributed LLM inference framework
README.md

Welcome to llm-d: a Kubernetes-native high-performance distributed LLM inference framework

GitHub Org's stars Documentation License

Join Slack X (formerly Twitter) Follow LinkedIn Reddit

llm-d is a well-lit path for serving large language models at scale with the fastest time-to-value and competitive performance per dollar. Built on vLLM, Kubernetes, and Inference Gateway, llm-d provides modular solutions for distributed inference with features like KV-cache aware routing and disaggregated serving.

Key Resources

🤝 How to Contribute

Join the Community

  1. 💬 Slack: Join our development discussions at llm-d.slack.com
  2. 📧 Google Group: Subscribe to llm-d-contributors for architecture docs and meeting invites
  3. 🗓️ Weekly Standup: Wednesdays at 1230 ET - Public Calendar

Contributing Code

  1. Read Guidelines: Review our Code of Conduct and contribution process
  2. Sign Commits: All commits require DCO sign-off (git commit -s)

Ways to Contribute

  • 🐛 Bug fixes and small features - Submit PRs directly to component repos
  • 🚀 New features with APIs - Require project proposals
  • 📚 Documentation - Help improve guides and examples
  • 🧪 Testing & Benchmarking - Contribute to our test coverage
  • 💡 Experimental features - Start in llm-d-incubation org

License: Apache 2.0

Pinned Loading

  1. llm-d llm-d Public

    llm-d is a Kubernetes-native high-performance distributed LLM inference framework

    Makefile 1.1k 65

  2. llm-d-inference-scheduler llm-d-inference-scheduler Public

    Inference scheduler for llm-d

    Go 50 20

  3. llm-d-deployer llm-d-deployer Public

    Helm charts for llm-d

    Shell 37 22

  4. llm-d-kv-cache-manager llm-d-kv-cache-manager Public

    Distributed KV cache coordinator

    Go 31 5

  5. llm-d-model-service llm-d-model-service Public

    Simplified model deployment on llm-d

    Go 21 8

  6. llm-d-benchmark llm-d-benchmark Public

    llm-d benchmark scripts and tooling

    Shell 13 6

Repositories

Showing 10 of 12 repositories
View all repositories

Top languages

Loading…

Most used topics

Loading…