Skip to content

architecture.md

Latest commit

 

History

History
158 lines (124 loc) · 8.4 KB

architecture.md

File metadata and controls

158 lines (124 loc) · 8.4 KB
Raw

Architecture

inferenced is intentionally small — its job is to take an existing, already-good piece of software (Apple's mlx_lm.server) and wrap it with the thin layer of operational glue that's missing for production-style use.

The big picture

              ┌───────────────────────────────────────────────────┐
              │                   client                          │
              │   (curl, OpenAI SDK, in-cluster pod, k8s          │
              │    Service forwarding via EndpointSlice)          │
              └─────────────────────────┬─────────────────────────┘
                                        │ HTTP, source-CIDR filtered
                                        ▼
              ┌───────────────────────────────────────────────────┐
              │                    inferenced                     │
              │ ┌──────────────┐  ┌────────────┐  ┌───────────┐  │
              │ │  axum router │  │  source    │  │ Prometheus│  │
              │ │ /v1/*, /, …  │──┤  filter    │──┤ /metrics  │  │
              │ └──────┬───────┘  └────────────┘  └───────────┘  │
              │        │ reqwest streaming proxy                  │
              │ ┌──────▼─────────┐  ┌─────────────────────┐      │
              │ │ Supervisor for │  │   /healthz upstream │      │
              │ │ mlx_lm.server  │──┤   probe             │      │
              │ └──────┬─────────┘  └─────────────────────┘      │
              └────────┼─────────────────────────────────────────┘
                       │ tokio::process; pipes stdout/stderr → tracing
                       ▼
              ┌───────────────────────────────────────────────────┐
              │            mlx_lm.server (Python)                 │
              │  127.0.0.1:18080 (loopback, never exposed)        │
              └─────────────────────────┬─────────────────────────┘
                                        ▼
              ┌───────────────────────────────────────────────────┐
              │         Apple Silicon GPU via Metal               │
              │              MLX framework                         │
              └───────────────────────────────────────────────────┘

Why a wrapper at all?

mlx_lm.server is great. It speaks OpenAI's API, it does proper KV-cache management, it streams. What it doesn't do:

Concern mlx_lm.server inferenced
Source-CIDR filtering ❌ ✅
launchd-friendly logging stdout/stderr only structured tracing events
Prometheus metrics ❌ ✅
Restart on crash depends on supervisor ✅ supervised w/ exponential backoff
Graceful shutdown partial ✅ propagates SIGTERM
Single deployable artifact Python+venv+pip a 3 MB Rust binary + python3.12 -m pip for the runtime
Auth model none source-CIDR allow-list

If you only need it on a workstation behind your firewall, you can run mlx_lm.server directly. If you want to expose it across a tailnet, drop it into a homelab cluster, or have an operator manage it, you want a daemon with these features.

Why not write inference in Rust?

mlx-rs (Rust bindings to MLX) exists, but the LLM-specific pieces (chat-template handling, KV-cache, structured sampling, OpenAI surface area) are still all in Python via mlx-lm. Rather than reimplement those, we treat mlx_lm.server as a black box and own everything around it.

If/when mlx-rs reaches feature parity with mlx-lm for the inference loop we care about, we'll fold it in directly and drop the Python subprocess.

Process model

A single inferenced process owns:

  1. The HTTP server (axum on the public bind port, default 0.0.0.0:11434).
  2. The supervisor task that spawns and restarts mlx_lm.server as a child process bound to a loopback port (default 127.0.0.1:18080).
  3. A stdio drain that forwards mlx_lm.server's stdout and stderr to tracing, so logs arrive at /var/log/inferenced.log (or wherever launchd is sending stderr) instead of disappearing.

When the parent exits — Ctrl-C, SIGTERM from launchd bootout, etc. — it sends SIGTERM to mlx_lm.server, waits up to 5 seconds, then SIGKILLs.

Wire protocol

inferenced is a transparent reverse proxy for /v1/*. The router has a fallback handler that forwards <method, headers, body> to the upstream and streams the response back. SSE is preserved by keeping the response body as a Body::from_stream(...). Hop-by-hop headers (Connection, Transfer-Encoding, etc.) are stripped both ways per RFC 9110.

This means anything mlx_lm.server adds to its OpenAI surface is automatically supported — we don't need to chase Apple's API additions.

Ports

Port Bound on Purpose Default
11434 0.0.0.0 (configurable) Public HTTP, source-CIDR filtered --bind 0.0.0.0:11434
18080 127.0.0.1 only mlx_lm.server upstream --backend-port 18080

11434 is the same default Ollama uses, so existing OpenAI clients pointed at "localhost Ollama" Just Work against inferenced too.

Source filtering

By default inferenced only accepts requests from:

  • 100.64.0.0/10 — Tailscale CGNAT IPv4 range
  • fd7a:115c:a1e0::/48 — Tailscale ULA IPv6 range
  • 127.0.0.0/8 and ::1/128 — loopback

Connecting from anywhere else returns 403 source not allowed. Disable with --allow-cidrs 0.0.0.0/0,::/0.

A common deployment is to bind inferenced to 0.0.0.0:11434 with the default Tailscale-only filter so it's reachable from cluster pods (whose Tailscale-in-VM IPs match) and your laptop, but invisible to the public internet even if the host's port 11434 is otherwise reachable.

Roadmap

v0.2 — admin API + multi-model

Add /admin/* routes for runtime model management:

  • POST /admin/models — load a new model into a new mlx_lm.server child on a unique loopback port. Updates an in-memory routing table so /v1/chat/completions with "model": "<id>" is dispatched to the right child.
  • DELETE /admin/models/{id} — graceful shutdown of one child.
  • GET /admin/models — list currently loaded models with RAM usage and hosting child PID.
  • GET /admin/info — host capabilities (RAM, GPU family, current load).

The Tailscale CIDR filter is intentionally insufficient for these admin routes — they need a stronger auth (mTLS or a bearer token) since they allow remote model loading. Both will be added in v0.2.

v0.3 — direct MLX (no Python subprocess)

Once mlx-rs covers the inference loop, drop the mlx_lm.server child process and link MLX directly. This gives us a truly single-binary deploy, no Python dependency, and lets us add features (continuous batching, paged attention) without waiting for upstream.

Pairing with inferenced-operator

inferenced-operator takes a fleet of inferenced daemons across multiple macOS hosts and exposes them as Kubernetes-managed inference services. The bridge stays unchanged for v0.1 — the operator just discovers them via InferenceHost CRDs and routes through K8s Service + EndpointSlice. For v0.2 the operator will use the admin API to load/unload models on demand.