Skip to content

SERVER.md

Latest commit

 

History

History
602 lines (438 loc) · 38 KB

SERVER.md

File metadata and controls

602 lines (438 loc) · 38 KB
Raw

SERVER.md

aictl-server is an HTTP LLM proxy. It exposes the same provider catalogue that the CLI ships behind one OpenAI-compatible endpoint so any client that already speaks the OpenAI SDK can transparently call Anthropic, Gemini, Grok, Mistral, DeepSeek, Kimi, Z.ai, Ollama, GGUF, or MLX models.

Pure proxy. No agent loop, no tool dispatch, no agents/skills/sessions, no slash commands. Those are CLI-only and stay CLI-only — see README.md and ARCH.md. For agent capabilities over HTTP, use the CLI.

Scope

Does

  • Expose POST /v1/chat/completions, POST /v1/completions, POST /v1/messages, GET /v1/models, GET /v1/stats, GET /healthz, GET /openapi.json.
  • Translate OpenAI-shaped requests into each provider's native format and back.
  • Proxy native Anthropic Messages API requests verbatim to api.anthropic.com so Claude Code (and any client speaking the Anthropic shape directly) can route through the server with tools, content blocks, and prompt caching intact.
  • Apply outbound redaction (run::redact_outbound) on every gateway request.
  • Apply the prompt-injection guard (security::detect_prompt_injection) on every user message.
  • Audit every gateway dispatch via audit::log_tool as gateway:<provider>.
  • Require a master API key on every authenticated request (auto-generated on first launch if not configured).
  • Stream responses over Server-Sent Events in OpenAI's data: {"choices":[{"delta":...}]} shape.
  • Apply a global in-flight concurrency cap (AICTL_SERVER_MAX_CONCURRENT_REQUESTS) and an optional per-client-IP token-bucket rate limit (AICTL_SERVER_RATE_LIMIT_RPM / AICTL_SERVER_RATE_LIMIT_BURST).

Does not

  • Run the agent loop or expose any endpoint that does.
  • Dispatch tools. Tool calls returned by upstream providers are not executed server-side; the security gate (security::validate_tool) is not wired up because there is nothing to validate.
  • Surface agents, skills, sessions, plugins, hooks, or slash commands.
  • Terminate TLS in v1. Run nginx/Caddy in front for HTTPS.

Install

One-liner

curl -fsSL https://aictl.app/server/install.sh | sh

The installer detects the platform (macOS arm64/x86_64, Linux x86_64/arm64), downloads the matching binary from the latest GitHub release, drops it on $PATH, and prints next-step guidance. It is idempotent — re-running upgrades in place.

From source

cargo install --git https://github.com/pwittchen/aictl.git --bin aictl-server

From a cloned repo

If you've already cloned the repo locally and want to build from your working tree (the usual workflow when developing or testing a patch):

# Run directly without installing — handy while iterating.
cargo run --release --bin aictl-server

# Build the release binary; lands at target/release/aictl-server.
cargo build --release --bin aictl-server

# Install the workspace binary to ~/.cargo/bin so it's on $PATH.
cargo install --path crates/aictl-server

# With optional features (mirrors the CLI's feature flags).
cargo install --path crates/aictl-server --features "gguf mlx redaction-ner"

cargo install --path puts the binary at ~/.cargo/bin/aictl-server. Make sure ~/.cargo/bin is on your $PATH (rustup adds it by default). To uninstall: aictl-server --uninstall (removes the binary from ~/.cargo/bin/, ~/.local/bin/, /usr/local/bin/, and $AICTL_INSTALL_DIR; leaves ~/.aictl/ untouched), or cargo uninstall aictl-server for a cargo-managed install.

First launch

aictl-server

On first launch the server generates a 32-byte master API key, persists it as AICTL_SERVER_MASTER_KEY (into the system keyring when available, otherwise plain ~/.aictl/config), and prints it once to stderr along with where it landed. Copy it — you'll need it on every request.

Launch via the CLI shortcut

If you have both binaries installed, aictl --serve is a convenience shortcut that locates aictl-server and execs it. Trailing args are forwarded to the server:

aictl --serve                                # default 127.0.0.1:7878
aictl --serve -- --bind 0.0.0.0:7878 --quiet # forward server flags after `--`

Resolution order for the server binary: a sibling of the current aictl executable, then $PATH, then ~/.cargo/bin/, then ~/.local/bin/, then $AICTL_INSTALL_DIR. If nothing is found, the CLI prints a clear "not installed" message with the install one-liner.

Usage

# Server runs on http://127.0.0.1:7878 by default.
aictl-server

# In another terminal:
curl http://127.0.0.1:7878/v1/chat/completions \
  -H "Authorization: Bearer $AICTL_SERVER_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

From the OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    api_key="<AICTL_SERVER_MASTER_KEY>",
    base_url="http://127.0.0.1:7878/v1",
)

reply = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

The same SDK call works against any model in GET /v1/models — the server picks the right upstream provider and substitutes its own configured key.

Connecting aictl-cli to aictl-server

The CLI can route every non-local LLM call through this server instead of holding upstream provider keys itself. Set two values in the CLI's ~/.aictl/config:

AICTL_CLIENT_HOST=http://127.0.0.1:7878
AICTL_CLIENT_MASTER_KEY=<value of AICTL_SERVER_MASTER_KEY from the server>

Or pass them per-launch without persisting:

aictl --client-url http://127.0.0.1:7878 --client-master-key sk-aictl-…

The AICTL_CLIENT_* vs AICTL_SERVER_* split is deliberate: a single host may run both roles, and the CLI side stores the connection key (what it presents to some server) under a name distinct from the server's own master key. Locking via /keys moves AICTL_CLIENT_MASTER_KEY into the OS keyring just like any provider key. Local providers (Ollama / GGUF / MLX) always bypass the server. /balance reads the server's /v1/stats aggregate when routing is active.

Configuration

aictl-server reads the same ~/.aictl/config file the CLI reads. Server-only knobs are prefixed AICTL_SERVER_* and sit alongside the existing CLI keys.

Key Default Description
AICTL_SERVER_BIND 127.0.0.1:7878 Bind address. Non-loopback values still require the master key; a startup warning is printed.
AICTL_SERVER_MASTER_KEY (auto-generated) Bearer token required on every authenticated request. Auto-generated on first launch into the system keyring when available (otherwise plain config). Participates in the CLI's /keys lock/unlock/clear flow alongside the provider keys, so a co-located CLI can move it between the keyring and plain config without restarting the server.
AICTL_SERVER_REQUEST_TIMEOUT 120 Per-request wall-clock timeout (seconds). 0 disables.
AICTL_SERVER_BODY_LIMIT_BYTES 2097152 Per-request body cap (2 MiB).
AICTL_SERVER_MAX_CONCURRENT_REQUESTS 32 Global concurrency semaphore. Returns 503 when saturated.
AICTL_SERVER_SHUTDOWN_TIMEOUT 20 Drain grace period on SIGTERM (seconds).
AICTL_SERVER_SSE_KEEPALIVE 15 SSE keepalive comment interval (seconds). 0 disables.
AICTL_SERVER_LOG_LEVEL info trace/debug/info/warn/error.
AICTL_SERVER_LOG_FILE ~/.aictl/server.log JSON-Lines log file. Empty disables the file sink (terminal sink stays on).
AICTL_SERVER_LOG_BODIES true Log redacted request/response bodies. false drops body lines at the source.
AICTL_SERVER_AUDIT_FILE ~/.aictl/server-audit.log Per-process audit log (JSON-Lines). Every gateway dispatch and redaction event lands here. Empty string disables disk audit even when the toggle is on. Suppressed entirely by AICTL_SERVER_SECURITY_AUDIT_LOG=false.
AICTL_SERVER_CORS_ORIGINS (empty) Comma-separated origin list. Empty = CORS off.
AICTL_SERVER_RATE_LIMIT_RPM 0 Per-client-IP requests per minute. 0 disables (only the global concurrency cap applies).
AICTL_SERVER_RATE_LIMIT_BURST 0 Token-bucket capacity (max consecutive requests). 0 falls back to the RPM value, so the bucket holds one minute of tokens.

Provider keys (LLM_OPENAI_API_KEY, LLM_ANTHROPIC_API_KEY, …) live under their existing CLI names — the server reads them via keys::get_secret, so keyring-stored keys work the same as plain-text fallback.

Server-scoped security and redaction

The server can run a different security / redaction posture than the CLI on the same host without forking ~/.aictl/config. For every flag that makes sense in a pure HTTP proxy, an AICTL_SERVER_* form takes precedence over the matching AICTL_* form when the engine is loaded inside aictl-server. Unset server overrides fall through to the shared key, so a single-host setup needs no duplication.

Tool-dispatch knobs (CWD jail, shell allow/block lists, blocked env vars, disabled tools, max-write byte cap, shell timeout) are intentionally not mirrored: the server does not run tools, so those flags have no meaning here.

Server key Falls back to Default Description
AICTL_SERVER_SECURITY AICTL_SECURITY true Master enable for the security subsystem (the prompt-injection guard + audit). false/0 turns it off entirely.
AICTL_SERVER_SECURITY_INJECTION_GUARD AICTL_SECURITY_INJECTION_GUARD true Run detect_prompt_injection on every user message before dispatch.
AICTL_SERVER_SECURITY_AUDIT_LOG AICTL_SECURITY_AUDIT_LOG true Append gateway:<provider> entries to ~/.aictl/audit/<request-id>.
AICTL_SERVER_SECURITY_REDACTION AICTL_SECURITY_REDACTION off off / redact / block. redact rewrites detected secrets in-place; block returns 400 redaction_blocked.
AICTL_SERVER_SECURITY_REDACTION_LOCAL AICTL_SECURITY_REDACTION_LOCAL false When false, local-provider dispatches (Ollama / GGUF / MLX from the server's host) skip the redaction pass. Set true to enforce redaction even on in-host traffic.
AICTL_SERVER_REDACTION_DETECTORS AICTL_REDACTION_DETECTORS (empty = all) Comma-separated subset of api_key, aws, aws_secret, jwt, private_key, connection_string, credit_card, iban, email, phone, url_secret, ssn, pesel, ip_address, mac_address, high_entropy.
AICTL_SERVER_REDACTION_EXTRA_PATTERNS AICTL_REDACTION_EXTRA_PATTERNS (empty) Semicolon-separated NAME=REGEX pairs → rewritten as [REDACTED:NAME].
AICTL_SERVER_REDACTION_ALLOW AICTL_REDACTION_ALLOW (empty) Semicolon-separated allowlist regexes — matches survive Layer-A/B redaction.
AICTL_SERVER_REDACTION_NER AICTL_REDACTION_NER false Enable Layer-C NER. Requires the redaction-ner cargo feature plus a pulled model.
AICTL_SERVER_REDACTION_NER_MODEL AICTL_REDACTION_NER_MODEL onnx-community/gliner_small-v2.1 NER model name (or owner/repo). The server can ship a different model from the CLI without forking config.

CLI flags

Flag Description
--bind <addr:port> Override AICTL_SERVER_BIND.
--master-key <value> Use this key for this launch only (not persisted).
--quiet Suppress startup banner.
--log-level <level> Override AICTL_SERVER_LOG_LEVEL.
--log-file <path> Override AICTL_SERVER_LOG_FILE.
--audit-file <path> Override AICTL_SERVER_AUDIT_FILE.

--unrestricted is intentionally absent — the server does not dispatch tools, so there is nothing to gate.

Master-key handling

  1. --master-key <value> wins for the current launch (not persisted).
  2. Otherwise the persisted AICTL_SERVER_MASTER_KEY is used. Resolution goes through keys::get_secret, so a value migrated into the system keyring via the CLI's /keys lock flow (or --lock-keys, or the desktop's Settings → Keys panel) resolves identically to a plain ~/.aictl/config entry.
  3. Otherwise 32 bytes of OS randomness are generated, base64url-encoded, and persisted via keys::set_secret: into the system keyring when the backend is available, falling back to plain ~/.aictl/config otherwise. The startup banner reports which store the key landed in.

Rotate by clearing the entry — /keys from the CLI, the Keys panel from the desktop, or by removing the line from ~/.aictl/config directly. The next launch regenerates. Comparison at the auth boundary is constant-time.

REST API

Every authenticated request must carry Authorization: Bearer <master-key>. Unauthenticated requests get a 401 with an OpenAI-shaped error envelope. GET /healthz and GET /openapi.json are the only auth-free routes.

POST /v1/chat/completions

OpenAI-shaped request schema. The model field selects the provider — exact match against the catalogue from GET /v1/models. stream: true returns SSE.

Request:

{
  "model": "claude-sonnet-4-6",
  "messages": [{"role": "user", "content": "Hello"}],
  "stream": false
}

Response (stream: false):

{
  "id": "chatcmpl-…",
  "object": "chat.completion",
  "created": 1714411200,
  "model": "claude-sonnet-4-6",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "Hi!"},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 8, "completion_tokens": 2, "total_tokens": 10}
}

Streaming response (stream: true):

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"}}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hi"}}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Tool-calling passthrough — not implemented in this phase. Requests with a non-empty tools array, a non-null non-"none" tool_choice, or any legacy functions field get a 400 tools_unsupported_for_provider.

POST /v1/messages

Native Anthropic Messages API endpoint. Used by clients that speak the Anthropic shape directly — most notably Claude Code via ANTHROPIC_BASE_URL. The body is forwarded verbatim to https://api.anthropic.com/v1/messages with the operator's stored LLM_ANTHROPIC_API_KEY substituted in, so tool use, content blocks, system prompts, and prompt caching all pass through unchanged.

By default the model field must resolve to an Anthropic model (GET /v1/models with owned_by: "Anthropic"); non-Anthropic models are rejected with 400 model_not_found. Set AICTL_SERVER_MESSAGES_CROSS_PROVIDER=true and the route translates the Anthropic Messages shape to/from any supported provider (OpenAI, Grok, Mistral, DeepSeek, Kimi, Z.ai, Gemini, Ollama) — see Cross-provider routing below for the full trade-off table. The OpenAI-shaped POST /v1/chat/completions remains the other route to non-Anthropic providers.

curl http://127.0.0.1:7878/v1/messages \
  -H "Authorization: Bearer $AICTL_SERVER_MASTER_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Streaming: "stream": true returns the native Anthropic SSE event stream (message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop) piped through unchanged — not the OpenAI data: {...} shape.

Headers: anthropic-version defaults to 2023-06-01; the client can override per request, and any anthropic-beta header the client sends is forwarded. The server's master-key gate is on Authorization: Bearer ..., distinct from Anthropic's x-api-key — the proxy fills in x-api-key itself.

Security passes: the prompt-injection guard runs on user-text content; the redactor runs on system and messages[*].content text surfaces. tool_use / tool_result blocks are deliberately left untouched (regex passes can't meaningfully redact opaque tool JSON without risking corruption).

Cross-provider routing

Warning

Experimental. The translator works for most chat-tuned models in the supported provider families, but it is best-effort — not every model survives the round-trip. Known gaps:

  • OpenAI reasoning models (o1, o3, o3-mini, gpt-5) — won't work. They use max_completion_tokens instead of max_tokens, reject the system role, ignore temperature, and need a reasoning_effort parameter the translator doesn't send.
  • Ollama models without the tools capability (plain Llama 2, older Phi, DeepSeek-V2) — basic chat works, but Ollama silently drops tool calls. Use Qwen 2.5/3, Llama 3.1+, Mistral Nemo, or Command-R for tool-using sessions.
  • Vision inputs need a vision-capable model on the target provider (GPT-4o, Gemini Flash/Pro, Llava, Qwen-VL); text-only models reject image content blocks upstream.
  • DeepSeek-R1 (deepseek-reasoner) — works for text but ignores tools; its reasoning_content is not surfaced.
  • Anthropic-only tools (memory, computer_use, code_execution_*) — Claude Code may declare these; on cross-provider routes they reach the upstream as plain function definitions and fail because no implementation exists.

Upstream rejections surface as 503 provider_unavailable (or 400 for shape errors) with the response body logged at warn level for diagnosis. Use Anthropic models for production reliability; treat cross-provider routes as best-effort.

By default the route only forwards to Anthropic — that's the byte-for-byte passthrough path described above, and it preserves every Anthropic-specific feature (prompt caching, extended thinking, anthropic-beta headers, fine-grained tool streaming, native PDF blocks). Flip the master flag and a second mode translates the Anthropic Messages shape into each provider's native shape and back, so Claude Code (or any native-Anthropic client) can run against OpenAI, Grok, Mistral, DeepSeek, Kimi, Z.ai, Gemini, or Ollama models. Anthropic models keep the passthrough path with zero behavioral drift; non-Anthropic models flow through a dedicated translator that owns the full provider HTTP round-trip (not aictl_core::llm::call_*, which uses the engine's internal XML tool format).

The full design — translation matrices for every Anthropic field, the streaming SSE state machine, feature-gate policy — lives in .claude/plans/done/messages-cross-provider.md.

Configuration:

Key Default Effect
AICTL_SERVER_MESSAGES_CROSS_PROVIDER false Master switch. When false, non-Anthropic models return 400 model_not_found.
AICTL_SERVER_MESSAGES_FEATURE_GATE strip strip / warn / reject for unsupported Anthropic features.
AICTL_SERVER_MESSAGES_TRANSLATE_PROVIDERS * Comma-separated allow-list (openai,gemini,ollama). * = any non-Anthropic provider.

Usage — Claude Code against a non-Anthropic model:

# 1. Server host — enable the translator and make sure the upstream key
#    for the target provider is configured (e.g. LLM_OPENAI_API_KEY for
#    OpenAI, LLM_GEMINI_API_KEY for Gemini, none needed for Ollama).
export AICTL_SERVER_MESSAGES_CROSS_PROVIDER=true
# Optional — restrict which providers the translator will dispatch to:
# export AICTL_SERVER_MESSAGES_TRANSLATE_PROVIDERS=openai,gemini
# Optional — fail loud instead of stripping unsupported Anthropic-only
# fields like cache_control or thinking:
# export AICTL_SERVER_MESSAGES_FEATURE_GATE=reject
aictl-server
# 2. Client laptop — point Claude Code at the server and pick any model
#    GET /v1/models exposes. Same Authorization scheme as the Anthropic
#    passthrough; only the model id changes.
export ANTHROPIC_BASE_URL="http://127.0.0.1:7878"
export ANTHROPIC_AUTH_TOKEN="$AICTL_SERVER_MASTER_KEY"

export ANTHROPIC_MODEL="gpt-4o-mini"                   # OpenAI
# export ANTHROPIC_MODEL="gemini-2.5-flash"            # Gemini
# export ANTHROPIC_MODEL="qwen2.5-coder:14b"           # Ollama (must declare `tools` capability)
# export ANTHROPIC_MODEL="deepseek-chat"               # DeepSeek
# export ANTHROPIC_MODEL="grok-2-1212"                 # Grok
# export ANTHROPIC_MODEL="mistral-large-latest"        # Mistral
# export ANTHROPIC_MODEL="kimi-k2-0905-preview"        # Kimi
# export ANTHROPIC_MODEL="glm-4.6"                     # Z.ai
export ANTHROPIC_SMALL_FAST_MODEL="$ANTHROPIC_MODEL"

claude

Anthropic models continue to use the byte-for-byte passthrough path even with the flag on — the cross-provider translator only activates when the resolved provider is non-Anthropic. Flipping the flag on a host that already serves Claude Code against Anthropic models is therefore safe (no behavioral drift on Anthropic dispatches; the translator path is only reached when a client picks a non-Anthropic ANTHROPIC_MODEL).

What survives on the cross-provider path:

Feature Anthropic passthrough OpenAI-family Gemini Ollama (tool-capable model)
Text content blocks yes yes yes yes
Image blocks (base64) yes yes (data URL) yes (inlineData) yes (images[])
Image blocks (URL) yes yes rejected rejected
Tool use (tool_use / tool_result) yes yes yes yes (capability-gated)
Streaming SSE event sequence native (verbatim) translated translated translated (from NDJSON)
Prompt caching (cache_control) yes stripped stripped stripped
Extended thinking (thinking) yes stripped stripped stripped
anthropic-beta headers yes (forwarded) ignored ignored ignored
Memory tool / computer use yes n/a n/a n/a
PDF document blocks yes rejected rejected rejected
top_k yes stripped yes yes
metadata.user_id yes yes (user) stripped stripped
Reasoning models (o1, o3, …) n/a not mapped (use /v1/chat/completions) n/a n/a

What's preserved on every path (passthrough and translation alike):

  • Master-key gate on every request.
  • Prompt-injection guard on every user-role text surface.
  • Redaction on every text surface in the request.
  • Per-request UUID + audit log entry (gateway:anthropic for passthrough, gateway:messages:<provider> for translation).

Trade-offs to weigh before enabling:

  • Cost. Cross-provider routes lose prompt caching. On long agent conversations (Claude Code typical: 20k-token system prompts cached on Anthropic) the same workflow can cost meaningfully more on OpenAI/Gemini. Run a budget probe first.
  • Tool fidelity. tool_use ↔ tool_calls translation is mechanical; subtle JSON-schema differences (Anthropic accepts looser schemas than OpenAI strict mode) may cause provider-side rejections. Set AICTL_SERVER_MESSAGES_FEATURE_GATE=warn initially and watch the response headers.
  • Streaming feel. Anthropic's event sequence is more structured than OpenAI's flat deltas. The translator approximates the rhythm but can't perfectly replicate latency profiles.
  • Local-only inference scope. GGUF and MLX are rejected on the cross-provider path — the in-process backends don't expose native tool calling. Use Ollama with a tool-capable model (Qwen 2.5, Llama 3.1+, Mistral Nemo) if you want local + tools.
  • Stop reasons. Best-effort mapping; content_filter collapses to end_turn, length → max_tokens, tool_calls → tool_use. Clients that key off granular stop reasons may behave differently.

Connecting Claude Code

Claude Code ships with first-class support for third-party Anthropic-compatible inference via two environment variables. Point them at this server:

export ANTHROPIC_BASE_URL="http://127.0.0.1:7878"
export ANTHROPIC_AUTH_TOKEN="<value of AICTL_SERVER_MASTER_KEY>"

# Optional — pin the models Claude Code should use. Any model from
# `GET /v1/models` owned by Anthropic works.
export ANTHROPIC_MODEL="claude-sonnet-4-6"
export ANTHROPIC_SMALL_FAST_MODEL="claude-haiku-4-5-20251001"

claude

ANTHROPIC_AUTH_TOKEN (vs ANTHROPIC_API_KEY) is the right variable — Claude Code sends the value as Authorization: Bearer <token>, which is what this server's master-key gate expects. ANTHROPIC_API_KEY sends it as x-api-key instead, which the server's auth layer doesn't read.

What you get out of routing Claude Code through aictl-server:

  • Centralized Anthropic key — the actual LLM_ANTHROPIC_API_KEY lives only on the server host. Every laptop running Claude Code talks to the server with the master key instead.
  • Audit trail — every Claude Code dispatch is logged as gateway:anthropic in AICTL_SERVER_AUDIT_FILE with the per-request UUID.
  • Outbound redaction — secrets in user messages get rewritten as [REDACTED:<KIND>] before they leave the network (set AICTL_SERVER_SECURITY_REDACTION=redact).
  • Prompt-injection guard — poisoned content surfaces as 400 prompt_injection so it cannot burn the operator's tokens.
  • Concurrency cap + per-IP rate limit — same backpressure controls as the OpenAI gateway routes.

For a non-loopback deployment, terminate TLS in front (see the nginx snippet below) and point ANTHROPIC_BASE_URL at the HTTPS hostname.

POST /v1/completions

Legacy text-completion API. The prompt is wrapped into a single user message and routed through the same provider-selection logic.

{
  "model": "gpt-4o-mini",
  "prompt": "Once upon a time"
}

GET /v1/models

Lists every model from aictl_core::llm::MODELS plus locally available Ollama / GGUF / MLX models. The available field is true when the upstream API key is configured (or, for local providers, when the model file is present).

GET /healthz

No auth. Returns {"status":"ok","version":"…","uptime_secs":…,"active_requests":N}.

GET /v1/stats

Authenticated. Returns the aictl_core::stats aggregates (today / month / overall).

GET /openapi.json

No auth. Serves the OpenAPI 3.1 description of every route above (request/response schemas, error envelope, security scheme, status-code table). Point Swagger UI, Redoc, openapi-generator, or any SDK generator at http://127.0.0.1:7878/openapi.json to introspect the surface or generate a typed client. The spec embeds the running server version under info.version.

Render it locally with one Docker command:

docker run --rm -p 8080:8080 \
  -e SWAGGER_JSON_URL=http://host.docker.internal:7878/openapi.json \
  swaggerapi/swagger-ui

Errors

Every error response is {"error": {"code": "…", "message": "…"}} — the same shape OpenAI uses, so SDK error handlers keep working.

HTTP Error code Cause
400 prompt_injection The prompt-injection guard tripped.
400 redaction_blocked Outbound message contained sensitive data (block mode).
400 model_not_found No provider knows how to serve the requested model.
400 body_malformed Request body did not match the expected schema.
400 tools_unsupported_for_provider Tool-calling fields are not supported in this phase.
401 auth_invalid Missing or wrong Authorization: Bearer header.
403 provider_auth_failed Upstream provider rejected the substituted key.
413 body_too_large Request body exceeded AICTL_SERVER_BODY_LIMIT_BYTES.
429 rate_limited Per-IP token bucket exhausted (set via AICTL_SERVER_RATE_LIMIT_RPM). Response carries Retry-After: <seconds>.
503 provider_unavailable Upstream provider failed (5xx, empty response, stream error).
503 provider_key_not_configured No API key for the resolved provider in ~/.aictl/config.
503 concurrency_cap_reached Global semaphore saturated.
504 gateway_timeout Per-request timeout expired.

Security model

  • Master-key gate: every authenticated request must present Authorization: Bearer <master-key>. Comparison is constant-time; both wrong-token and missing-token map to the same 401 body.
  • Network bind: defaults to 127.0.0.1. Non-loopback binds emit a startup warning. Operators are responsible for putting TLS in front when exposing beyond localhost.
  • CORS: off by default. Set AICTL_SERVER_CORS_ORIGINS to opt in.
  • Body cap: 2 MiB by default; oversized bodies get 413.
  • Concurrency cap: 32 in-flight requests by default; saturated cap returns 503 immediately rather than queueing.
  • Rate limit (optional): per-client-IP token bucket via AICTL_SERVER_RATE_LIMIT_RPM and AICTL_SERVER_RATE_LIMIT_BURST. Off by default. Saturation returns 429 with a Retry-After: <seconds> header. Buckets are keyed by the request's source IP (read from the socket — X-Forwarded-* is not trusted), so the limiter only behaves as expected when the server is reached directly. Behind a reverse proxy every request appears to come from the proxy's IP — terminate the limit at the proxy or trust the proxy to set its own.
  • Redaction: aictl_core::run::redact_outbound runs on every gateway request, with the same regex bank, entropy pass, and optional NER as the CLI. Local providers (Ollama/GGUF/MLX) skip unless AICTL_SECURITY_REDACTION_LOCAL=true.
  • Prompt-injection guard: aictl_core::security::detect_prompt_injection runs on every user message; matches surface as 400 prompt_injection so poisoned prompts can't burn the operator's tokens.
  • Audit: every successful gateway dispatch is logged as gateway:<provider> (with the per-request UUID as the result tag) and every redaction event as redaction to a per-process JSON-Lines file at AICTL_SERVER_AUDIT_FILE (default ~/.aictl/server-audit.log). Toggle via AICTL_SERVER_SECURITY_AUDIT_LOG; override the path with --audit-file <path>. Unlike the CLI's session-keyed audit scheme, the server uses one file for the whole process — there is no session id.

The master key grants full proxy access — there is no second tier of credentials. Rotate by editing the config file.

Rate limiting

Two layers cooperate:

  1. Global concurrency cap — AICTL_SERVER_MAX_CONCURRENT_REQUESTS (default 32). A tokio::Semaphore bounds in-flight requests. Saturation returns 503 immediately rather than queueing. Always on.
  2. Per-client-IP token bucket — opt-in via AICTL_SERVER_RATE_LIMIT_RPM (0 disables, the default). Each unique client IP gets its own bucket; saturation returns 429 with a Retry-After: <seconds> header.

Configuration

Knob Meaning
AICTL_SERVER_RATE_LIMIT_RPM Steady-state requests per minute per IP. 0 disables.
AICTL_SERVER_RATE_LIMIT_BURST Bucket capacity (max consecutive requests). 0 falls back to RPM, so the bucket holds one minute of tokens.

Tokens refill linearly at RPM / 60 per second. The bucket starts full so the first burst is allowed up to the configured capacity.

Examples

# 60 requests/min sustained; bucket capacity defaults to RPM (60 tokens),
# so an idle client can fire 60 immediate requests, then drips at 1/s.
AICTL_SERVER_RATE_LIMIT_RPM=60

# 600 requests/min sustained, but limit any single burst to 50 consecutive
# requests (tighter than the default 600-token capacity for this RPM).
AICTL_SERVER_RATE_LIMIT_RPM=600
AICTL_SERVER_RATE_LIMIT_BURST=50

Behavior

  • The limiter sits behind the auth gate, so unauthenticated traffic gets 401 without burning a bucket entry.
  • GET /healthz is exempt entirely — it sits outside both auth and rate-limit middleware so liveness probes stay free.
  • Buckets are keyed by the request's source IP (read from the socket — X-Forwarded-* is not trusted). Behind a reverse proxy every request appears to come from the proxy's IP. Terminate the limit at the proxy, or accept that the per-IP bucket is effectively a global cap behind a single proxy.
  • The internal map is bounded: when more than 10,000 distinct buckets accumulate, idle buckets older than two minutes are evicted on the next request.
  • Startup logs event=rate_limit_enabled with the resolved RPM and burst when the limiter is active.
  • 429 events log event=rate_limited with client_ip and retry_after_secs.

Server log

Two sinks fan out from one event source:

  1. File sink — JSON-Lines at AICTL_SERVER_LOG_FILE (default ~/.aictl/server.log).
  2. Terminal sink — human-readable, ANSI-colored on TTY, written to stderr. Auto-disables colors on non-TTY or when NO_COLOR is set.

Levels: trace/debug/info/warn/error. Body lines are gated by AICTL_SERVER_LOG_BODIES (default true); turning it off drops body lines at the source.

Rotation is the operator's responsibility (logrotate, journald). The file is opened append-only; SIGTERM flushes via the buffered writer's Drop.

Deployment

systemd

[Unit]
Description=aictl-server
After=network.target

[Service]
Type=simple
User=aictl
ExecStart=/usr/local/bin/aictl-server
Restart=on-failure
NoNewPrivileges=yes
ProtectSystem=strict
ReadWritePaths=/var/lib/aictl

[Install]
WantedBy=multi-user.target

nginx reverse proxy (TLS + SSE)

server {
  listen 443 ssl http2;
  server_name aictl.example.com;

  location /v1/ {
    proxy_pass http://127.0.0.1:7878;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_read_timeout 600;
  }
}

proxy_buffering off is required for SSE streaming to flush deltas live instead of being buffered until the connection closes.

Docker

A multi-stage Dockerfile lives at docker/server.Dockerfile. The build context expects to be the workspace root so cargo can see every crate.

# Build the image (Rust → debian:bookworm-slim, ~25 MB final layer).
docker build -f docker/server.Dockerfile -t aictl-server .

# Run with a persistent named volume so the auto-generated master key
# and any provider keys you set inside the container survive restarts.
docker run --rm -d \
  --name aictl-server \
  -p 127.0.0.1:7878:7878 \
  -v aictl-data:/home/aictl/.aictl \
  aictl-server

# Capture the master key from the first-launch banner.
docker logs aictl-server

Knobs to know:

  • AICTL_SERVER_BIND is set to 0.0.0.0:7878 inside the image so the published port is reachable. Override with -e AICTL_SERVER_BIND=… to bind a different interface or port.
  • The aictl-data volume holds ~/.aictl/ for the in-container aictl user (UID 1000). The auto-generated AICTL_SERVER_MASTER_KEY lands in ~/.aictl/config (no Secret Service in the container, so the keyring backend silently falls back).
  • A HEALTHCHECK runs curl -fsS http://127.0.0.1:7878/healthz every 30s — Docker marks the container unhealthy if it stops responding.
  • Optional cargo features (gguf, redaction-ner) are off by default; enable per-build with --build-arg FEATURES="redaction-ner". MLX is Apple-Silicon-only and never built in the Linux image.
  • Provider keys: bake them into the volume's ~/.aictl/config or pass them at run time (-e LLM_OPENAI_API_KEY=…). The CLI's /keys lock flow does not apply inside the container — there is no keyring backend to migrate into.
  • For TLS, run nginx/Caddy in front (see the nginx snippet above) or terminate at the platform's load balancer; the server itself does not speak TLS.

FAQ

Is rate limiting available? Yes. A per-client-IP token bucket is configurable via AICTL_SERVER_RATE_LIMIT_RPM and AICTL_SERVER_RATE_LIMIT_BURST; off by default. Saturation surfaces as 429 with a Retry-After header. The global concurrency cap (AICTL_SERVER_MAX_CONCURRENT_REQUESTS) operates independently and remains the primary backpressure mechanism.

Can I forward a per-request provider key? Not in this phase. Phase 3 revisits whether to support a X-Provider-Authorization header (or a provider_key body field) for deployments that want the server to hold no provider keys.

Why no agent endpoints? Pure-proxy is the entire point. Agent loops over HTTP would have a different auth model, session story, and tool-approval protocol; that becomes a separate plan if a concrete demand surfaces.

Can I run two servers on one host? Yes — distinct --bind values plus distinct config trees (e.g. HOME=/var/lib/aictl-prod aictl-server). Native --config <path> support depends on the modular-architecture loader change landing.

Verification (developers)

The server enforces a hard separation from the CLI's interactive surface. CI greps:

grep -rE 'rustyline::|termimad::|indicatif::|crossterm::|dialoguer::' crates/aictl-server/src/
grep -rE 'run_agent_turn|run_agent_single|AgentUI|ToolApproval' crates/aictl-server/src/

Both must return empty. Any future change that pulls a REPL dep or reaches into the agent loop violates the proxy-only contract — fix the change, not the grep.