Skip to content

Commit 25bcf49

Browse files
GrowlyXclaude
GrowlyX
and
claude
committed
init: inferenced — macOS-native MLX inference daemon
OpenAI-compatible HTTP daemon that supervises mlx_lm.server, adds: - Source-CIDR allow-list (Tailscale + loopback by default) - Prometheus /metrics - /healthz upstream-aware liveness - /admin/* API for runtime model management (load/unload/list/info) - launchd LaunchDaemon template - Kubernetes Service + EndpointSlice examples for clusters not running inferenced-operator Pairs with github.com/dormlab/inferenced-operator. Tests pass: cargo test test result: ok. 5 passed Comprehensive docs/ covering architecture, install, configuration, HTTP API, metrics, development, troubleshooting. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
0 parents  commit 25bcf49

20 files changed

Lines changed: 2003 additions & 0 deletions

‎.github/workflows/ci.yaml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
name: ci
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
jobs:
9+
build-test:
10+
runs-on: macos-latest
11+
steps:
12+
- uses: actions/checkout@v4
13+
- uses: dtolnay/rust-toolchain@stable
14+
with:
15+
components: rustfmt, clippy
16+
- uses: Swatinem/rust-cache@v2
17+
18+
- name: fmt
19+
run: cargo fmt --check
20+
21+
- name: clippy
22+
run: cargo clippy --all-targets -- -D warnings
23+
24+
- name: build
25+
run: cargo build --release --target aarch64-apple-darwin
26+
27+
- name: test
28+
run: cargo test

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
target/
2+
Cargo.lock
3+
.idea/
4+
.vscode/
5+
.DS_Store

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Changelog
2+
3+
## [Unreleased]
4+
5+
## [0.1.0] - initial release
6+
7+
### Added
8+
- Single-model supervisor wrapping `mlx_lm.server` with restart-on-crash and graceful shutdown.
9+
- OpenAI-compatible `/v1/*` proxy preserving SSE streaming.
10+
- Source-CIDR allow-list (defaults to Tailscale + loopback).
11+
- Admin API: `GET /admin/info`, `GET /admin/models`, `POST /admin/models`, `DELETE /admin/models/{id}`.
12+
- Prometheus `/metrics` with `inferenced_requests_total{route,status}`.
13+
- `/healthz` upstream-aware liveness check.
14+
- `launchd` LaunchDaemon plist template.
15+
- `tracing` structured logging.

‎CONTRIBUTING.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# Contributing
2+
3+
Thanks for your interest! `inferenced` is a small project; PRs are
4+
welcome but please open an issue first for larger changes so we can
5+
agree on direction.
6+
7+
## Quick start for contributors
8+
9+
```sh
10+
git clone https://github.com/dormlab/inferenced
11+
cd inferenced
12+
cargo test # unit tests
13+
cargo run -- --help # see all CLI options
14+
```
15+
16+
See [`docs/development.md`](./docs/development.md) for the full layout
17+
and conventions.
18+
19+
## Pull request checklist
20+
21+
- [ ] `cargo fmt` clean.
22+
- [ ] `cargo clippy --all-targets -- -D warnings` clean.
23+
- [ ] `cargo test` passes.
24+
- [ ] If you added a public route, [`docs/api.md`](./docs/api.md) is updated.
25+
- [ ] If you added a config flag, [`docs/configuration.md`](./docs/configuration.md) is updated.
26+
- [ ] If you added a metric, [`docs/metrics.md`](./docs/metrics.md) is updated.
27+
- [ ] [`CHANGELOG.md`](./CHANGELOG.md) has an entry under `[Unreleased]`.
28+
29+
## License
30+
31+
By contributing you agree that your work is licensed under the project's
32+
MIT License.

‎Cargo.toml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
[package]
2+
name = "inferenced"
3+
version = "0.1.0"
4+
edition = "2024"
5+
description = "Inference daemon: macOS-native MLX LLM serving with an OpenAI-compatible HTTP API and Tailscale-aware source filtering."
6+
license = "MIT"
7+
readme = "README.md"
8+
repository = "https://github.com/dormlab/inferenced"
9+
homepage = "https://github.com/dormlab/inferenced"
10+
keywords = ["mlx", "llm", "inference", "openai", "daemon"]
11+
categories = ["command-line-utilities"]
12+
13+
[[bin]]
14+
name = "inferenced"
15+
path = "src/main.rs"
16+
17+
[dependencies]
18+
anyhow = "1"
19+
axum = { version = "0.8", features = ["macros"] }
20+
bytes = "1"
21+
clap = { version = "4", features = ["derive", "env"] }
22+
futures-util = "0.3"
23+
prometheus = { version = "0.14", default-features = false }
24+
reqwest = { version = "0.12", default-features = false, features = ["stream", "rustls-tls", "json"] }
25+
serde = { version = "1", features = ["derive"] }
26+
serde_json = "1"
27+
tokio = { version = "1", features = [
28+
"rt-multi-thread",
29+
"macros",
30+
"process",
31+
"signal",
32+
"sync",
33+
"time",
34+
"io-util",
35+
] }
36+
tower-http = { version = "0.6", features = ["trace"] }
37+
tracing = "0.1"
38+
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt"] }
39+
40+
[profile.release]
41+
lto = true
42+
codegen-units = 1
43+
strip = true
44+
opt-level = 3
45+
panic = "abort"

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 DormLab
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
# inferenced
2+
3+
> *Inference daemon for Apple Silicon. macOS-native MLX serving with an
4+
> OpenAI-compatible HTTP API and Tailscale-aware source filtering.*
5+
6+
`inferenced` is a small Rust daemon that runs on a macOS host. It supervises
7+
[`mlx_lm.server`](https://github.com/ml-explore/mlx-lm) (Apple's reference
8+
LLM server, which uses Metal under the hood for GPU-accelerated inference),
9+
adds proper process supervision, source-CIDR filtering, Prometheus metrics,
10+
and a clean OpenAI-compatible HTTP surface that's safe to expose on a
11+
Tailscale-only port.
12+
13+
It exists because LLM inference on Apple Silicon **must** run as native
14+
macOS code — Apple's Virtualization framework doesn't expose Metal/MPS/ANE
15+
to Linux guests. So if your workloads (Kubernetes pods, scripts, agents)
16+
live in Linux land but you want to use the GPU your Mac mini already has,
17+
**you need a daemon on the host that serves inference and a clean way for
18+
clients to call it**. This is that daemon.
19+
20+
It pairs with [`inferenced-operator`](https://github.com/dormlab/inferenced-operator),
21+
a Kubernetes operator that orchestrates fleets of `inferenced` instances
22+
across a cluster of Apple Silicon hosts. You can also run `inferenced`
23+
standalone — `curl localhost:11434/v1/chat/completions` and you're done.
24+
25+
```text
26+
┌──────────────────┐ ┌────────────────────┐
27+
│ any client │ HTTP │ inferenced │
28+
│ (curl, kubectl, ├────────►│ (axum, supervisor,│
29+
│ pod, script) │ │ metrics, auth) │
30+
└──────────────────┘ └─────────┬──────────┘
31+
│ proxy /v1/*
32+
▼
33+
┌────────────────────┐
34+
│ mlx_lm.server │
35+
│ (Python, MLX) │
36+
└─────────┬──────────┘
37+
▼
38+
┌────────────────────┐
39+
│ Apple Silicon GPU │
40+
│ via Metal │
41+
└────────────────────┘
42+
```
43+
44+
## Quick start
45+
46+
```sh
47+
# Prereqs: Rust 1.75+, Python 3.10+, Apple Silicon Mac
48+
brew install python@3.12
49+
python3.12 -m pip install --user --break-system-packages mlx-lm
50+
51+
# Build
52+
cargo build --release --target aarch64-apple-darwin
53+
54+
# Run (defaults to Qwen2.5-3B-Instruct-4bit and binds 0.0.0.0:11434)
55+
./target/aarch64-apple-darwin/release/inferenced
56+
57+
# In another terminal
58+
curl http://localhost:11434/v1/chat/completions \
59+
-H 'Content-Type: application/json' \
60+
-d '{
61+
"model": "mlx-community/Qwen2.5-3B-Instruct-4bit",
62+
"messages": [{"role": "user", "content": "hello"}],
63+
"stream": false
64+
}'
65+
```
66+
67+
## Documentation
68+
69+
| | |
70+
|---|---|
71+
| [Architecture](./docs/architecture.md) | How `inferenced` fits between clients, MLX, and the rest of your infrastructure. |
72+
| [Installation](./docs/installation.md) | Install on a single Mac — Homebrew, Rust toolchain, `mlx-lm`, and a `launchd` LaunchDaemon for boot persistence. |
73+
| [Configuration](./docs/configuration.md) | Every CLI flag and env var. |
74+
| [HTTP API](./docs/api.md) | OpenAI-compatible `/v1/*`, plus `/healthz`, `/metrics`, `/`. |
75+
| [Metrics](./docs/metrics.md) | Prometheus metric reference. |
76+
| [Development](./docs/development.md) | Building from source, running tests, contributing. |
77+
| [Troubleshooting](./docs/troubleshooting.md) | "It's not starting", "I get `source not allowed`", "tokens/sec is bad". |
78+
79+
## Examples
80+
81+
- [`examples/launchd/dev.dormlab.inferenced.plist`](./examples/launchd/dev.dormlab.inferenced.plist) — system-level LaunchDaemon (runs as root for Metal access).
82+
- [`examples/kubernetes/`](./examples/kubernetes/) — Service + EndpointSlice manifests so cluster pods can call your fleet of macOS hosts as a single in-cluster Service.
83+
84+
## Features
85+
86+
- ✅ **Single static binary**, ~3 MB (`cargo build --release`).
87+
- ✅ **OpenAI-compatible** — every `/v1/*` route is transparently proxied; SSE streaming preserved end-to-end.
88+
- ✅ **Source-CIDR filtering** — defaults to Tailscale + loopback, configurable.
89+
- ✅ **Process supervision** — restarts `mlx_lm.server` with capped exponential backoff.
90+
- ✅ **Prometheus `/metrics`** — request counters by route + status class.
91+
- ✅ **Healthchecks** — `/healthz` validates the upstream Python process.
92+
- ✅ **Graceful shutdown** — SIGTERM propagates to `mlx_lm.server`.
93+
- ✅ **`launchd` LaunchDaemon** template for boot persistence.
94+
95+
## Status
96+
97+
`v0.1` — single-model per daemon, fixed at startup via `--model`. Multi-model
98+
hot-loading is the v0.2 goal (admin API for `POST /admin/models/{load,unload}`)
99+
which the operator can drive, see [the architecture doc](./docs/architecture.md#roadmap).
100+
101+
## License
102+
103+
MIT. See [LICENSE](./LICENSE).

‎docs/api.md‎

Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# HTTP API
2+
3+
`inferenced` exposes three groups of routes:
4+
5+
1. **OpenAI-compatible (`/v1/*`)** — transparently proxied to `mlx_lm.server`.
6+
2. **Admin (`/admin/*`)** — model management; intended for use by an operator like [`inferenced-operator`](https://github.com/dormlab/inferenced-operator).
7+
3. **Operational (`/healthz`, `/metrics`, `/`)** — for monitoring / liveness.
8+
9+
All routes are subject to the source-CIDR allow-list (default
10+
Tailscale + loopback). The bridge does not implement OpenAI-style
11+
`Authorization: Bearer` token validation in v0.1; the v0.2 admin-API
12+
work will add that for the `/admin/*` routes specifically.
13+
14+
## OpenAI surface
15+
16+
Anything `mlx_lm.server` accepts, `inferenced` forwards verbatim. The
17+
fallback handler streams the response body straight back, so SSE-based
18+
chat completions stream through the bridge without buffering.
19+
20+
```text
21+
GET /v1/models → list of currently-loaded models
22+
POST /v1/chat/completions → chat (supports stream=true)
23+
POST /v1/completions → legacy text completions
24+
```
25+
26+
Example (non-streaming):
27+
28+
```sh
29+
curl -s http://localhost:11434/v1/chat/completions \
30+
-H 'Content-Type: application/json' \
31+
-d '{
32+
"model": "mlx-community/Qwen2.5-3B-Instruct-4bit",
33+
"messages": [{"role":"user","content":"hello"}],
34+
"stream": false,
35+
"max_tokens": 100
36+
}' | jq .choices[0].message.content
37+
```
38+
39+
Streaming:
40+
41+
```sh
42+
curl -N http://localhost:11434/v1/chat/completions \
43+
-H 'Content-Type: application/json' \
44+
-d '{ ..., "stream": true }'
45+
```
46+
47+
## Admin API
48+
49+
All admin routes return JSON with `Content-Type: application/json` on
50+
success.
51+
52+
### `GET /admin/info`
53+
54+
Live host capabilities + currently loaded model.
55+
56+
```json
57+
{
58+
"hostname": "amelia",
59+
"memory_total_bytes": 17179869184,
60+
"memory_free_bytes": 4429185024,
61+
"cpu_brand": "Apple M4",
62+
"cpu_cores": 10,
63+
"current_model": "mlx-community/Qwen2.5-3B-Instruct-4bit",
64+
"version": "0.1.0",
65+
"timestamp": 1761617655
66+
}
67+
```
68+
69+
### `GET /admin/models`
70+
71+
List of currently-loaded models. v0.1 always returns 0 or 1 entries
72+
(single-model supervisor); v0.2 will return many.
73+
74+
```json
75+
[ { "id": "mlx-community/Qwen2.5-3B-Instruct-4bit", "backend_port": 18080 } ]
76+
```
77+
78+
### `POST /admin/models`
79+
80+
Body: `{ "id": "<huggingface-id>" }`. Swaps the running model for the
81+
given one — kills the existing `mlx_lm.server` child, spawns a new one
82+
with the new id, and updates `current_model`.
83+
84+
Returns the same `ModelEntry` shape as `GET /admin/models`.
85+
86+
```sh
87+
curl -X POST http://localhost:11434/admin/models \
88+
-H 'Content-Type: application/json' \
89+
-d '{"id": "mlx-community/Qwen2.5-7B-Instruct-4bit"}'
90+
```
91+
92+
### `DELETE /admin/models/{id}`
93+
94+
Unloads the named model. The supervisor enters an idle state; further
95+
`/v1/*` requests will 502 until a `POST /admin/models` re-loads
96+
something. Returns `204 No Content` on success, `404` if the named id
97+
isn't currently loaded.
98+
99+
## Operational
100+
101+
### `GET /healthz`
102+
103+
`200 ok` if `inferenced` itself is alive AND its supervised
104+
`mlx_lm.server` answers `/v1/models`. `503` otherwise.
105+
106+
### `GET /metrics`
107+
108+
Prometheus exposition. See [metrics.md](./metrics.md).
109+
110+
### `GET /`
111+
112+
Trivial HTML landing page with links to the routes above.
113+
114+
## Errors
115+
116+
| Code | When | Body |
117+
|---|---|---|
118+
| `403` | Source IP not in `--allow-cidrs` | `source not allowed\n` |
119+
| `404` | Path doesn't match `/v1/*`, `/admin/*`, etc., or trying to delete an unloaded model | `not found\n` or `loaded model is "X", not "Y"\n` |
120+
| `502` | Upstream `mlx_lm.server` returned a network error | `upstream error: ...\n` |
121+
| `500` | Local error encoding metrics, parsing host info, etc. | `error: ...\n` or `encode failure\n` |

0 commit comments

Comments
 (0)