⚡ Indus LLM Gateway

Production-ready LLM proxy — unified API in front of every AI provider, packed into a single Windows EXE.

What Is It?

Indus LLM Gateway is a self-hosted, zero-dependency reverse proxy for Large Language Model APIs. Drop it in front of your applications and get:

One API endpoint that works with any OpenAI-compatible SDK
Traffic routing across providers / models with 5 strategies
Auth & quotas — issue API keys with per-key token budgets
Streaming SSE proxied end-to-end in real-time
Prompt templating — inject system prompts and few-shot examples centrally
Conversation memory scoped per session
Token accounting with cost tracking per key
Prometheus metrics at /metrics
Embedded web dashboard at /ui/ — no separate front-end server needed

Everything ships as a single ~24 MB Windows EXE. No Docker required. No Node.js. No GCC. No CGO.

Quick Start (60 seconds)

Option 1 — Download the binary

# 1. Download from GitHub Releases
Invoke-WebRequest -Uri "https://github.com/indus-ai/indus-llm-gateway/releases/latest/download/indus-llm-gateway.exe" -OutFile "gateway.exe"

# 2. Copy and edit the config
Copy-Item gateway.example.yaml gateway.yaml
# Edit gateway.yaml → set your provider API key

# 3. Set your OpenAI key (or any supported provider)
$env:OPENAI_API_KEY = "sk-proj-..."

# 4. Run
.\gateway.exe --config .\gateway.yaml

Option 2 — Build from source

git clone https://github.com/indus-ai/indus-llm-gateway.git
cd "indus-llm-gateway"

$env:CGO_ENABLED = "0"
$env:GOOS        = "windows"
$env:GOARCH      = "amd64"

go build -ldflags="-s -w" -o dist\indus-llm-gateway.exe .\cmd\gateway

First Run

Navigate to http://localhost:8080/ui/ — the setup wizard will appear and create your first admin API key:

Architecture

System Overview

Request Lifecycle

Every incoming request passes through 9 stages:

Stage	Component	What Happens
1	`api/openai`	Parse request, decode JSON body
2	`api/middleware`	Authenticate — validate Bearer token
3	`policy`	Check quota / rate-limit for this key
4	`memory`	Load prior conversation turns
5	`prompt`	Apply system prompt template
6	`routing`	Select provider + model via strategy
7	`providers`	Forward to LLM API (with circuit-breaker)
8	`accounting`	Record tokens consumed + cost
9	`telemetry`	Update Prometheus counters / histograms

Multi-Provider Support

Provider	Models	Streaming	Notes
OpenAI	gpt-4o, gpt-4o-mini, gpt-3.5-turbo, o1-…	✅	Default
Anthropic	claude-3-5-sonnet, claude-3-opus	✅
Google Gemini	gemini-1.5-pro, gemini-1.5-flash	✅
Azure OpenAI	Any deployed model	✅	Needs `deployment_name`
Groq	llama-3.1-70b, mixtral-8x7b	✅	Ultra-fast inference
Mistral	mistral-large, codestral	✅
Cohere	command-r-plus	✅
Ollama	Any local model	✅	Local inference

Routing Strategies

Configure per-route or per-model:

routing:
  strategy: latency        # latency | weighted | error_rate | round_robin | random
  providers:
    - name: openai
      weight: 70
    - name: anthropic
      weight: 30

Strategy	Description
`latency`	Routes to the provider with lowest P95 latency (default)
`weighted`	Probabilistic split by configured weight
`error_rate`	Avoids providers with high recent error rates
`round_robin`	Cycle through providers in order
`random`	Uniform random selection

Authentication & API Keys

Bootstrap (first run)

# Create the first admin key (only works when no keys exist)
curl -s -X POST http://localhost:8080/indus/v1/setup | jq .

# Response
{
  "key": "igw_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "message": "First admin key created. Store this securely — it is shown only once."
}

Create additional keys

curl -s -X POST http://localhost:8080/indus/v1/keys \
  -H "Authorization: Bearer igw_<admin-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "my-app",
    "daily_token_limit": 100000,
    "monthly_token_limit": 2000000,
    "allowed_models": ["gpt-4o-mini", "gpt-4o"]
  }' | jq .

API Reference

OpenAI-Compatible Endpoints

Point any OpenAI SDK at http://localhost:8080:

from openai import OpenAI

client = OpenAI(
    api_key="igw_your_gateway_key",
    base_url="http://localhost:8080/v1"
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True
)

Method	Path	Description
`POST`	`/v1/chat/completions`	Chat completion (stream supported)
`POST`	`/v1/completions`	Legacy text completion
`POST`	`/v1/embeddings`	Embeddings
`GET`	`/v1/models`	List available models

Native Management API

Method	Path	Auth	Description
`GET`	`/indus/v1/health`	❌ Public	Health check
`GET`	`/indus/v1/version`	❌ Public	Version info
`POST`	`/indus/v1/setup`	❌ Public (once)	Bootstrap first admin key
`GET`	`/indus/v1/keys`	✅	List API keys
`POST`	`/indus/v1/keys`	✅	Create API key
`DELETE`	`/indus/v1/keys/{id}`	✅	Revoke API key
`GET`	`/indus/v1/models`	✅	List configured models
`GET`	`/indus/v1/usage`	✅	Usage statistics
`GET`	`/metrics`	✅	Prometheus metrics

Configuration

Copy gateway.example.yaml → gateway.yaml and customise:

server:
  port: 8080
  read_timeout: 30s
  write_timeout: 60s

providers:
  openai:
    api_key: "${OPENAI_API_KEY}"       # use env var — never commit secrets
    default_model: gpt-4o-mini
    timeout: 45s
    max_retries: 3

routing:
  strategy: latency

policy:
  default_daily_limit: 100000          # tokens per key per day
  default_monthly_limit: 2000000

memory:
  enabled: true
  backend: sqlite                      # sqlite | memory
  max_turns: 20                        # turns to keep per session

storage:
  path: ./data/gateway.db

telemetry:
  prometheus_enabled: true

All values can be overridden with environment variables (GATEWAY_SERVER_PORT, OPENAI_API_KEY, etc.).

Web Dashboard

Indus Gateway ships a fully embedded web dashboard at /ui/ — no separate web server, no Node.js, no build step. It is compiled directly into the binary.

Features:

Overview — system status, token usage charts, quick links
Models — browse registered models and providers
API Keys — create, list, and revoke keys
Usage — per-key token and cost breakdown
Playground — interactive chat UI to test prompts
Prompts — manage system prompt templates
Routing — visualise current routing configuration

Deployment

Standalone EXE (Recommended for Windows)

.\indus-llm-gateway.exe --config .\gateway.yaml

Docker

docker run -p 8080:8080 \
  -e OPENAI_API_KEY=sk-... \
  -v ./gateway.yaml:/etc/gateway/gateway.yaml:ro \
  ghcr.io/indus-ai/indus-llm-gateway:latest

Windows Service (via NSSM)

# Install NSSM: https://nssm.cc
nssm install IndusGateway "C:\indus\indus-llm-gateway.exe"
nssm set IndusGateway AppParameters "--config C:\indus\gateway.yaml"
nssm set IndusGateway AppEnvironmentExtra "OPENAI_API_KEY=sk-..."
nssm start IndusGateway

UML Diagrams

Component Diagram

Chat Completion Sequence

Project Structure

indus-llm-gateway/
├── cmd/
│   └── gateway/
│       └── main.go              # Entry point — wires everything together
├── internal/
│   ├── api/
│   │   ├── middleware/          # Auth middleware, request logging
│   │   ├── native/              # Native management REST API (/indus/v1/*)
│   │   └── openai/              # OpenAI-compatible API (/v1/*)
│   ├── auth/                    # API key store, validation, scopes
│   ├── accounting/              # Token counting, cost calculation
│   ├── circuit/                 # Circuit-breaker per provider
│   ├── config/                  # YAML config loader + env var overrides
│   ├── eval/                    # Response quality evaluation hooks
│   ├── logger/                  # Structured zerolog wrapper
│   ├── memory/                  # Conversation session storage
│   ├── policy/                  # Quota enforcement, rate-limiting
│   ├── prompt/                  # Template engine, few-shot injection
│   ├── providers/               # LLMProvider interface + per-provider adapters
│   ├── routing/                 # 5 routing strategies
│   ├── storage/                 # SQLite + in-memory backends
│   ├── streaming/               # SSE proxy, delta buffering
│   ├── telemetry/               # Prometheus metrics registry
│   ├── types/                   # Shared domain types
│   └── ui/
│       ├── ui.go                # Serve embedded static files
│       └── static/
│           └── index.html       # Single-page dashboard app
├── docs/
│   ├── assets/                  # SVG diagrams
│   └── uml/                     # UML diagrams
├── gateway.example.yaml         # Config template
├── go.mod
├── go.sum
└── README.md

Observability

Prometheus Metrics

Exposed at GET /metrics:

Metric	Type	Labels
`indus_requests_total`	Counter	`method`, `path`, `status`
`indus_request_duration_seconds`	Histogram	`provider`, `model`
`indus_tokens_total`	Counter	`key_id`, `model`, `type` (prompt/completion)
`indus_provider_errors_total`	Counter	`provider`, `error_type`
`indus_circuit_open`	Gauge	`provider`

Grafana

Import the sample dashboard from docs/grafana/indus-gateway.json (coming soon).

Environment Variables

Variable	Default	Description
`GATEWAY_SERVER_PORT`	`8080`	HTTP listen port
`OPENAI_API_KEY`	—	OpenAI provider API key
`ANTHROPIC_API_KEY`	—	Anthropic API key
`GEMINI_API_KEY`	—	Google Gemini API key
`AZURE_OPENAI_API_KEY`	—	Azure OpenAI key
`GROQ_API_KEY`	—	Groq key
`MISTRAL_API_KEY`	—	Mistral key
`COHERE_API_KEY`	—	Cohere key
`GATEWAY_CONFIG`	`gateway.yaml`	Path to config file
`LOG_LEVEL`	`info`	`debug` \| `info` \| `warn` \| `error`

Security

Never commit gateway.yaml — it is in .gitignore. Use gateway.example.yaml as a template.
API keys are stored hashed (SHA-256) in SQLite.
The /indus/v1/setup bootstrap endpoint becomes a 403 after the first key is created.
TLS termination should be done upstream (nginx, Caddy, Cloudflare Tunnel).

Contributing

See CONTRIBUTING.md for the full guide. In short:

git checkout -b feat/my-feature
# make changes
go test ./...
git push origin feat/my-feature
# open a Pull Request

Changelog

See CHANGELOG.md.

License

Built with ❤️ in Go · No dependencies · Runs anywhere on Windows x64

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.github		.github
cmd/gateway		cmd/gateway
docs		docs
internal		internal
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
gateway.example.yaml		gateway.example.yaml
go.mod		go.mod
go.sum		go.sum

Folders and files

Latest commit

History

Repository files navigation

⚡ Indus LLM Gateway

What Is It?

Quick Start (60 seconds)

Option 1 — Download the binary

Option 2 — Build from source

First Run

Architecture

System Overview

Request Lifecycle

Multi-Provider Support

Routing Strategies

Authentication & API Keys

Bootstrap (first run)

Create additional keys

API Reference

OpenAI-Compatible Endpoints

Native Management API

Configuration

Web Dashboard

Deployment

Standalone EXE (Recommended for Windows)

Docker

Windows Service (via NSSM)

UML Diagrams

Component Diagram

Chat Completion Sequence

Project Structure

Observability

Prometheus Metrics

Grafana

Environment Variables

Security

Contributing

Changelog

License

About

Topics

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases 1

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages