Skip to content

README.md

Latest commit

 

History

History
654 lines (509 loc) · 52.5 KB

README.md

File metadata and controls

654 lines (509 loc) · 52.5 KB
Raw

AuditTrace-AI

Sovereign Architecture Overview

Production-grade sovereign AI memory server with 4-layer memory architecture, full observability, and Zero Trust deployment on Kubernetes (Istio mTLS, SPIFFE/SVID workload identity, deny-all AuthorizationPolicies, Vault-backed secrets). A self-contained Docker Compose stack covers laptop/sandbox development environments that don't have a Kubernetes cluster available.

Research & Governance

This implementation is based on the technical and regulatory framework defined in: "Sovereign Local AI: Why On-Device LLM Inference on Unified Memory Hardware Outperforms Commercial API Stacks for Regulated Industries" (March 2026).

Important

Read the Full Technical Analysis (PDF) Digitally signed via SwissSign to certify authenticity and research priority.

This work formalises the Sovereignty-Reconstructibility Gap and provides an auditable architecture compliant with EU AI Act Article 12 and GDPR Article 44. The PDF is digitally signed, providing authorship non-repudiation, temporal anchoring, and tamper-evidence -- consistent with the reconstructibility principles defined in Section 7 of the paper.

Academic trajectory: This research is under discussion with the University of Liverpool Department of Computer Science as a potential PhD programme, building on the author's MSc in Big Data Analytics (Liverpool, 2022).

CI CodeQL License: AGPL v3 Python 3.12+ Code style: black Tests

Features

  • 🧠 4-Layer Memory Architecture -- Episodic (ADRs), procedural (skills), conversational (PostgreSQL), semantic (ChromaDB). Layers 1+2 are always MinIO-backed (S3) — no filesystem fallback; tests use Mock* services.
  • 🛠️ Memory-as-Tools -- LLM calls recall_decisions, recall_skills, recall_recent_sessions, recall_semantic (snippet discovery) plus read_decision, read_skill (full-content fetch by filename) on demand instead of paying the 4-layer cost on every prompt (ADR-025). Dynamic registry, Redis-backed result cache, configurable iteration cap.
  • 📄 Audit-Grade PDF Ingestion -- Tier-A PAdES signature validation with the 10-class taxonomy (signed_valid / signed_invalid / signed_untrusted / signed_expired / signed_tampered / none / check_skipped / check_unavailable / check_failed / check_error); tier-B robustness (encrypted-PDF reject, attachment quarantine, AcroForm extraction, OCR coverage tracking, document-level manifest with SHA-256 + GIN-indexed extraction_warnings); tier-C document metadata + per-document audit granularity + corrupted-file taxonomy + dry-run + LTV + PDF/A + TOC-aware chunking + surgical reindex. ADR-050 / ADR-052 / ADR-054 / ADR-056.
  • 🔐 EU + Swiss Trust Store -- Pluggable TrustStoreProvider / TrustStoreBuilder ABCs (S3-backed by default, Vault-future). EU LOTL walker via pyhanko[etsi] (ADR-052) + Swiss federal TSL fetcher (OFCOM/BAKOM, ADR-053) → composite ~897 EU + CH qualified-signature CAs. Refresh via POST /system/trust-store/refresh (audittrace:admin) or Helm post-install hook. As-of-signing-time retry with 5-min clock-skew tolerance (ADR-054) distinguishes signed_expired from signed_untrusted.
  • 👥 Multi-user Identity -- Keycloak-delegated OAuth2 + per-user UserContext plumbing + Postgres Row-Level Security + ChromaDB scoped wrapper. Non-superuser audittrace_app role means RLS actually bites at the DB layer (ADR-026). RLS-aware app.current_user_id ContextVar plumbed through background workers + CLIs (ADR-046).
  • 🔐 External IdP Federation -- Keycloak brokering to Google Workspace / Okta / EntraID for multi-tenant SSO (ADR-044). Multi-issuer JWT validation already in place (ADR-032 §2).
  • 🗄️ Server-Mode Databases -- PostgreSQL 18 + ChromaDB HTTP server + Redis 8, all with authentication (ADR-019, ADR-020). (Bitnami subchart versions: postgresql ~16, redis ~19; container binaries are PG 18.3 + Redis 8.6.2 — verified live 2026-05-13.)
  • 🔒 TLS Everywhere -- Istio Gateway terminates TLS in production (NodePort 30952). Traefik v3 + mkcert in dev (ADR-021).
  • 🔑 Secrets via Vault -- HashiCorp Vault is the sole production secret store (ADR-043). Auto-unseal at boot via systemd unit. .env files only used in dev.
  • 🔍 Reconstructible by Design -- Every interaction + memory tool call traced via Langfuse + OpenTelemetry; one tool_calls audit row per memory invocation with interaction_id FK. Test + Evidence + Reconstructibility gate (ADR-049) enforces unit + live-system evidence on every meaningful change; CI parses PR bodies for the three required sections.
  • 🪞 Recursive Self-Audit -- The recorder records its own security review. A dedicated least-privilege ingest (POST /assessments, scope audittrace:assessment:ingest) writes each assessment run as owner-scoped, trace-linked assessment audit rows with the raw payload content-addressed to object storage — so the system's governability becomes reconstructible through its own front door (ADR-058).
  • 📊 Full-Stack Observability -- @log_call aspect emits logs, OTel spans, and histogram metrics from a single decorator; stdout-only logging (12-factor). service.version templated from Chart.AppVersion so OTel resource attributes never lie about deployed code (ADR-055).
  • 🔄 Transparent LLM Proxy -- Raw dict pass-through for OpenAI tool-calling protocol (ADR-024); memory context injected into the system message without stripping tools, tool_calls, tool_call_id.
  • 🇪🇺 GDPR-Compliant -- Data never leaves your infrastructure.
  • 🔌 OpenAI-Compatible -- /v1/chat/completions API.
  • 🐳 Two runtimes, one image -- k3s + Istio + Helm chart for production ZTA; Docker Compose for development. Same image, same env-var contract.
  • ✅ Comprehensive Test Suite -- 98% coverage, 90% per-file gate enforced in CI (live suite size in the tests badge above; latest run).

Quick Start

Two supported runtimes

Mode Stack Purpose Bring-up
Production k3s + Istio + Helm + Vault + Keycloak Default. ZTA mTLS, SPIFFE/SVID workload identity, deny-all AuthorizationPolicies, secrets in Vault. make k8s-build && make k8s-install
Development Docker Compose + Traefik For laptops and sandboxes without a Kubernetes cluster. Self-contained, hot-reload via overlay. ./scripts/start-full-stack.sh

Both runtimes deploy the same memory-server image, share the OpenAI-compatible API surface, and use identical AUDITTRACE_* env-var names. Defaults differ only on in-cluster vs container-network DNS.

Deploy with Kubernetes (production)

# Clone repository
git clone https://github.com/lfdesousa/AuditTrace-AI
cd AuditTrace-AI

# Build container images and push to the local k3s registry on :5000
make k8s-build

# Install (Helm chart + Istio Gateway + Vault + Keycloak realm + observability shippers)
make k8s-install

The Helm chart wires the Istio Gateway on NodePort 30952 (HTTPS), the Vault Agent injector renders runtime secrets into the memory-server pod, and the Keycloak realm is imported on first boot. See docs/guides/deployment-runbook.md for the full bring-up — TLS cert provisioning, SPIFFE/SVID workload identity, deny-all AuthorizationPolicies, Vault unseal-key handling.

The stack exposes:

  • https://audittrace.local:30952/v1/... -- chat completions and audit API (TLS via Istio Gateway)
  • https://audittrace.local:30952/auth/... -- Keycloak realm endpoint (delegated by the same Gateway)
  • kubectl -n audittrace ... -- direct in-cluster access during operations

Map audittrace.local to your k3s node IP in /etc/hosts if your environment doesn't already do this.

Deploy with Docker Compose (development)

# Clone repository
git clone https://github.com/lfdesousa/AuditTrace-AI
cd AuditTrace-AI

# Generate dev secrets (postgres password + chroma token + minio KMS key)
./scripts/setup-secrets.sh

# Generate TLS certificates (requires mkcert)
./certs/generate-certs.sh

# Create the shared Docker network
docker network create audittrace-net

# Configure environment from setup-secrets output
cp .env.example .env
# Edit .env with the secrets

# Bring up the full stack (memory-server + postgres + chromadb + keycloak + redis + minio + Traefik)
./scripts/start-full-stack.sh

# For hot-reload during development:
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d memory-server

The dev stack exposes:

  • https://localhost/v1/... -- memory-server API (TLS via Traefik)
  • http://localhost:8080 -- Traefik dashboard

Warning

The Docker Compose configuration is for development only. It uses postgres superuser credentials (which bypass RLS), .env-based secrets (no Vault), and Traefik in place of Istio (no mTLS or SPIFFE/SVID). Run k3s for any deployment touching real users or real data.

Optional: Langfuse + observability stack (both runtimes)

Observability backends — Langfuse, Tempo, Loki, Prometheus, Grafana — live in the sibling repo at lfdesousa/AiSovereignObservability. Clone alongside this repo and run:

./scripts/setup-observability.sh        # bootstrap the sibling backends stack
./scripts/setup-observability.sh --down # tear down

The same script works for both runtimes — only the in-runtime shippers (OTel Collector + Promtail) differ. Backends are shared.

Local development

# Setup isolated development environment
./scripts/setup.sh
# or
make install

# Run tests (90% per-file coverage enforced; see the tests badge for the live suite size)
make test

# Run RLS integration tests against an ephemeral docker postgres on :15432
make test-rls-local

# Lint + format
make lint
make format

# Run the development server directly (against an external dev stack)
source .venv/bin/activate
uvicorn audittrace.server:app --reload

Drop-in OpenAI compatibility

Strict OpenAI /v1/chat/completions compatibility is AuditTrace-AI's North Star. Point any OpenAI SDK (openai-python, @ai-sdk/openai-compatible, langchain, llamaindex) or IDE integration (OpenCode, Continue, Cursor, Zed) at https://<your-audittrace-host>/v1 and it works. Every request field the SDK sends passes through unchanged (ADR-024 dict pass-through); every response shape — success, streaming chunk, error — is a strict superset of what the OpenAI OpenAPI spec defines.

All AuditTrace-specific features are additive and opt-in:

Extension Where Opt-in mechanism
Project tagging request X-Project: … header (ADR-029)
Memory-mode routing request X-Memory-Mode: inject | tools header (ADR-031, scoped)
Depth-of-thinking request X-Thinking: deep | fast | auto header (ADR-034, scoped)
Audit-row forensics response error body net-new keys status, operator_hint, trace_id, user_facing_message alongside OpenAI's {message,type,param,code}
Async job pattern request X-Async: true header OR separate endpoint — default POST stays strictly OpenAI-shaped (ADR-035, scoped)

No custom header is ever required for the default path to work. The guardrail is codified in docs/reference/openai/ — we vendor the current upstream OpenAI OpenAPI spec and compare our response shapes against it directly. Regression tests in tests/test_openai_compatibility.py lock the key contracts (ChatCompletion, ChatCompletionChunk, Error/ErrorResponse) so any future change that would break compatibility fails CI before it reaches main.

Refresh the vendored spec with ./scripts/refresh-openai-spec.sh and review the diff like any other dependency bump.

API Endpoints

Endpoint Method Description
/v1/chat/completions POST OpenAI-compatible chat with memory augmentation
/v1/models GET OpenAI-compatible model list, proxied from llama-server's /v1/models — lets any OpenAI SDK discover available models before sending completions. Scope: audittrace:query.
/context POST 4-layer memory retrieval
/interactions GET Human-facing audit browser — filters project, user_id, session_id, source, since; pagination via limit (≤1000) + offset. RLS scopes every caller to their own rows. See Auditing conversations.
/assessments POST Recursive self-audit ingest (ADR-058) — records the recorder's own security-assessment run as first-class owner-scoped, trace-linked assessment audit rows; the raw payload is content-addressed to assessments/{assessment_id}/{sha}.json in object storage. Dedicated least-privilege scope: audittrace:assessment:ingest.
/session/save POST Persist session data
/session/summary POST Operator-driven session summary save (ADR-030)
/memory/upload POST Two-path: PDF uploads (content-type application/pdf + magic-byte sniff) land in quarantine/<user>/<scan_id>/ and return HTTP 202 + {scan_id, status:"pending_scan", poll_url}; content-control scans + promotes to episodic/papers/ asynchronously (ADR-048 PR-B3). Non-PDF (Markdown skills, ADRs, plain text) keeps the synchronous PUT-to-{layer}/ path returning 200.
/memory/upload/status GET Poll a PDF upload's scan status by ?scan_id=... — returns one of pending_scan / scanning / scanned_clean / rejected_malware / scan_failed / scan_unrecoverable (ADR-048 closed-set per migration 012).
/memory/index POST Re-index a layer (or a single file via ?file=<key>); ?details=true adds per-document outcomes (chunks, signature_status, page_count, pdf_*, ok/error) per ADR-056 §3; ?dry_run=true reports what would happen without writing to ChromaDB
/memory/{layer} GET / POST / PUT / DELETE Backoffice CRUD over episodic / procedural / semantic items; per-layer memory:<layer>:write scope
/system/trust-store GET Read PEM-bundle metadata (sha256, builder_id, cert_count, source_url, built_at) — audittrace:admin
/system/trust-store/refresh POST Walk EU LOTL + Swiss TSL → composite ~897 CAs → persist to S3 — audittrace:admin
/health GET Health check
/metrics GET Monitoring metrics (scope: audittrace:admin)

Base URL by runtime:

  • k3s: https://audittrace.local:30952
  • docker-compose: https://localhost

Architecture

Service Topology (Kubernetes)

Client (HTTPS :30952, Bearer JWT)
    |
    v
+---------------------+
| Istio Gateway       |  istio-ingress · TLS (SIMPLE) · NodePort 30952
| + VirtualService    |  hosts: "*"  (audittrace namespace)
+----------+----------+
           |
===========|=== namespace: audittrace ====================================
           |    PeerAuthentication  mTLS STRICT
           |    Workload identity   SPIFFE/SVID
           |    AuthorizationPolicy deny-all + 6 per-flow allow rules
           v
+---------------------+    JWKS    +------------------+
| memory-server       +----------->| Keycloak         |
| FastAPI +           |            | OAuth2 Device    |
| require_user +      |            | Flow (ADR-032)   |
| tool-call loop      |            | + IdP brokering  |
+-+--+--+--+--+--+----+            | (ADR-044)        |
  |  |  |  |  |  |                 +------------------+
  |  |  |  |  |  v
  |  |  |  |  |  +----------------+
  |  |  |  |  |  | PostgreSQL 18  |  RLS + audittrace_app
  |  |  |  |  |  +----------------+  (NOSUPERUSER + NOBYPASSRLS)
  |  |  |  |  v
  |  |  |  |  +----------------+
  |  |  |  |  | ChromaDB       |  token auth + UserScopedSemanticService
  |  |  |  |  +----------------+
  |  |  |  v
  |  |  |  +----------------+
  |  |  |  | Redis 8        |  token:* + tool-result:* (disjoint prefixes)
  |  |  |  +----------------+
  |  |  v
  |  |  +----------------+
  |  |  | MinIO          |  SSE-S3 KMS · episodic + procedural (ADR-027)
  |  |  +----------------+
  |  |
  |  |  +----------------+
  |  |  | Vault          |  sole secret store (ADR-043) · auto-unseal
  |  |  +----------------+  via systemd unit at boot
  |  |
  |  |  OTLP :4318
  |  v
  |  +--------------------+   traces  --> Tempo  (egress)
  |  | OTel Collector     |
  |  | (DaemonSet)        |   metrics --> Prometheus scrape :8889
  |  +--------------------+
  |
  |  Langfuse SDK (LLM traces, separate from OTLP path)
  +---> Langfuse                                            (egress)

=== mesh egress: ServiceEntry + no-mTLS DestinationRule ==================

External LLMs (host-resident, systemd-managed):
  qwen-chat-llm      :11435   Qwen 3.6-35B-A3B     chat + tools
  nomic-embed-text   :11436   nomic v1.5           embeddings
  mistral-summariser :11437   Mistral 7B v0.3      ADR-030 summariser

Observability sinks (sibling repo AiSovereignObservability, off-mesh):
  Tempo (OTLP :14318)  ·  Prometheus  ·  Loki  ·  Grafana  ·  Langfuse

The Istio Gateway terminates TLS on NodePort 30952 and fans out via a VirtualService. Inside the mesh, PeerAuthentication enforces STRICT mTLS and every flow has an explicit SPIFFE-identity allow in its AuthorizationPolicy — everything else is deny-all. External egress (LLMs + observability sinks) traverses a ServiceEntry paired with a no-mTLS DestinationRule, since the upstreams are systemd processes on the host, not mesh workloads. The OTel Collector runs in-mesh as a DaemonSet: it receives OTLP from memory-server and fans out traces to Tempo and metrics to a Prometheus scrape endpoint on :8889. Langfuse receives LLM traces via the Langfuse SDK (separate from OTLP). Redis serves both the TokenCache (JWT hot path) and the ToolResultCache (memory-as-tools result cache) under disjoint key prefixes.

Development uses the Docker Compose topology (docker-compose.yml + Traefik + sibling Langfuse stack). The same memory-server image runs in both modes; only the periphery differs. See docs/guides/deployment-runbook.md for the full dev-mode topology and the production k8s bring-up.

4-Layer Memory

Layer Implementation Test backend Purpose
1. Episodic S3EpisodicService (MinIO bucket prefix episodic/) MockEpisodicService Architecture decisions
2. Procedural S3ProceduralService (MinIO bucket prefix procedural/) MockProceduralService Reusable skill documents
3. Conversational PostgresConversationalService (same) Session history + continuity
4. Semantic ChromaSemanticService (same) Vector search / RAG

Layers 1 + 2 are S3-only — AUDITTRACE_MINIO_SECRET_KEY is mandatory at startup; missing config raises a clear RuntimeError instead of silently falling back to filesystem. This applies to all environments including local dev (the file-backed services were removed in the 2026-05-03 stabilization sweep). Unit tests use the Mock* services. All services follow the ABC + implementation + mock pattern with @log_call observability.

Each layer's read paths support two access shapes: snippet-discovery (recall_*, returns truncated previews of matches) and full-content fetch (read_decision, read_skill, returns the entire document by exact filename — used after a recall_* narrows the candidate). See Memory-as-Tools for the env-var surface.

Memory Access Modes (ADR-025)

The 4-layer memory can be exposed to the LLM in two different ways. The choice is a runtime flag (AUDITTRACE_MEMORY_MODE) and does not change the underlying storage layout.

Mode What happens on every /v1/chat/completions Trade-off
tools (live default — .env ships with this set) The proxy advertises six recall tools to the LLM and runs a tool-call loop. The model decides which layer it needs and calls recall_decisions, recall_skills, recall_recent_sessions, recall_semantic (snippet-level discovery) or read_decision / read_skill (full-content fetch by exact filename) on demand — at most once per question in practice. Results are cached in Redis (TTL 900s) and audit-logged to tool_calls. Pay memory cost only when the model asks. Extra round-trips to llama-server (one per tool iteration), bounded by AUDITTRACE_MEMORY_TOOL_LOOP_MAX_ITERATIONS.
inject (legacy — v0.2.x default) The proxy retrieves all 4 layers up front and injects them into the system message. The model sees a preassembled context block and never issues tool calls for memory. Zero tool-loop latency but every prompt pays the full 4-layer retrieval + token cost, even when the model would have ignored memory.

The six tools in tools mode each map to one layer and carry their own Keycloak scope (memory:episodic:read for recall_decisions + read_decision, memory:procedural:read for recall_skills + read_skill, memory:conversational:read-own for recall_recent_sessions, memory:semantic:read for recall_semantic). The read_* tools fetch a single document by exact filename and return the full untruncated content — use them after a recall_* snippet hints at a candidate but the question requires the whole ADR / SKILL. Per-user isolation, RLS, and the ChromaDB scoped wrapper apply unchanged to both modes — the difference is purely when the layers are queried, not what the user sees.

Switch modes by setting AUDITTRACE_MEMORY_MODE=inject|tools in .env (dev) or chart values (k3s) and rolling the memory-server pod. See ADR-025 for the full design and the acceptance evidence.

Security

  • OAuth2 -- Keycloak-delegated identity (ADR-022, ADR-023, ADR-026). Every request is a JWT validated against the Keycloak JWKS endpoint. Hot path hits a Redis-backed TokenCache for sub-millisecond lookups. External IdP federation (Google/Okta/EntraID) brokered through Keycloak realms (ADR-044). Browser UIs use OIDC Authorization Code + PKCE via a BFF (ADR-042).
  • Per-user isolation, defense in depth:
    1. Service layer -- every memory service method takes a UserContext first arg; conversational load_sessions applies unconditional WHERE user_id =.
    2. Postgres RLS -- interactions, sessions, tool_calls all carry ENABLE ROW LEVEL SECURITY + FORCE ROW LEVEL SECURITY + one policy per table gated on current_setting('app.current_user_id', true).
    3. Non-superuser app role -- memory-server connects as audittrace_app (NOSUPERUSER + NOBYPASSRLS). Postgres superusers always bypass RLS, so this is not optional for real enforcement.
    4. ChromaDB wrapper -- UserScopedSemanticService binds a UserContext at construction and overrides any per-call context; isolation is true by construction.
    5. Session-id uniqueness -- _compute_session_id includes user_id in the sha256 so two users with the same first message never produce a colliding session id.
  • TLS -- Istio Gateway terminates HTTPS on NodePort 30952 in production. Traefik v3 + mkcert handles TLS in docker-compose dev (port 443).
  • Service-mesh ZTA (k3s only) -- PeerAuthentication mTLS STRICT, deny-all AuthorizationPolicies with explicit per-flow SPIFFE-identity allows.
  • Secrets -- Vault is the sole production secret store (ADR-043), Vault Agent injector renders runtime secrets into pod filesystems, auto-unseal at boot via the audittrace-vault-auto-unseal.service systemd unit. Dev mode uses .env and Docker secrets.
  • ChromaDB -- Token-based authentication via Authorization: Bearer header (ADR-019, ADR-020).
  • Network isolation -- Database services are not exposed outside the mesh / docker network.

Authentication (ADR-022, ADR-023, ADR-026, ADR-032, ADR-042, ADR-044)

All endpoints except /health require a valid JWT (AUDITTRACE_AUTH_REQUIRED=true is the default).

Human agents (OpenCode / Continue / Roo Code) — OAuth2 Device Flow (ADR-032):

# One-time on a fresh Keycloak (realm import handles it): nothing to do.
# On an already-running Keycloak: provision the public client + realm user once.
KEYCLOAK_ADMIN_PASSWORD=admin scripts/setup-human-user.sh

# Interactive login (opens a browser login on any device):
scripts/audittrace-login
#   → prints a short verification URL + user code; log in as `luis`,
#     tokens land in ~/.config/audittrace/tokens.json (mode 0600).

# Launch OpenCode with a fresh token wired into ~/.config/opencode/config.json:
scripts/opencode-wrapper.sh

# In scripts / ad-hoc curl calls:
BEARER=$(scripts/audittrace-login --show)   # auto-refreshes if near expiry
curl -H "Authorization: Bearer $BEARER" https://audittrace.local:30952/v1/chat/completions ...

Silent refresh is automatic within the 30-day SSO session lifetime. After that, scripts/audittrace-login re-logs in interactively.

Browser UIs — OIDC Authorization Code + PKCE (ADR-042): For first-party web UIs (LibreChat, future audit dashboards) we use OIDC Authorization Code with PKCE through a Backend-for-Frontend pattern. The BFF holds the refresh token; the SPA only sees short-lived access tokens.

External IdP — Keycloak brokering (ADR-044): Google Workspace / Okta / EntraID identities federate through Keycloak realm brokering. The memory-server's JWT validation is multi-issuer (ADR-032 §2) and accepts both the local realm issuer and any configured federated issuers via AUDITTRACE_KEYCLOAK_EXTRA_ISSUERS. No code changes were needed in the validation layer.

→ See docs/guides/idp-federation-setup.md for the operator-side procedure (uses scripts/setup-idp-federation.sh, post-Phase-B.3 auto-populates all OIDC discovery endpoints).

Service accounts (CI, smoke tests) -- client_credentials: The audittrace-dev client is the right choice for non-interactive scripts; see scripts/mint-dev-jwt.sh.

Scopes exposed by the audittrace realm:

Scope Endpoints / Tools
audittrace:query /v1/chat/completions, /session/save
audittrace:context /context
audittrace:audit /interactions
audittrace:admin /metrics
memory:episodic:read recall_decisions (snippet discovery) + read_decision (full content) tools (ADR-025) + GET /memory/episodic
memory:procedural:read recall_skills + read_skill tools + GET /memory/procedural
memory:conversational:read-own recall_recent_sessions tool
memory:semantic:read recall_semantic tool + GET /memory/semantic
memory:episodic:write POST/PUT/DELETE /memory/episodic, POST /memory/upload (episodic/), POST /memory/index?file=episodic/...
memory:procedural:write POST/PUT/DELETE /memory/procedural, POST /memory/upload (procedural/), POST /memory/index?file=procedural/...
memory:semantic:write POST/PUT/DELETE /memory/semantic
audittrace:admin /metrics, GET/POST /system/trust-store/*, bulk POST /memory/index (whole-collection delete-and-recreate)

→ See docs/guides/memory-backoffice.md for the operator workflow + endpoint matrix + the memory_items manifest table that backs every CRUD response (created_at_ms / modified_at_ms / deleted_at_ms in Unix epoch milliseconds UTC).

Roadmap

See docs/roadmap.md for the full dated master roadmap — phased through 2026-10-31, with honest risk renegotiation policy. The table below captures the completed-to-date subset; the master doc covers Phases 1–4 + research track.

Priority Item Description
✅ Done Intelligent Memory Tool Routing Ambient context instructs the LLM to pick ONE tool per question. Validated with Qwen 3.6-35B-A3B. ADR-025 Accepted.
✅ Done MinIO Object Storage Stateless 12-factor containers — host filesystem mounts replaced with MinIO S3 (ADR-027). SSE-S3 encryption at rest.
✅ Done Observability Aggregation Stack Prometheus + Grafana + Loki + OTel Collector as sibling compose stack (ADR-028). Per-LLM peer.service edges on Tempo service graph.
✅ Done mypy --strict Clean 0 errors across all source files.
✅ Done Audit Trail Completeness Project tagging via X-Project header (ADR-029). /interactions endpoint with RLS-scoped pagination.
✅ Done Session Summariser Background asyncio loop via Mistral 7B Instruct v0.3 (ADR-030). Three-model topology.
✅ Done OAuth2 Device Flow RFC 8628 Device Flow for human agents (ADR-032). Multi-issuer JWT validation. Per-user RLS bites at user granularity.
✅ Done Three-Audience Error Envelope Classify + persist chat-path failures (ADR-033). OpenAI strict-superset error shape. Migration 007.
✅ Done Long-Running Generation Per-chunk idle timeout, SSE keep-alive, X-Thinking header (ADR-034). Qwen <think> reasoning runs indefinitely as long as tokens flow.
✅ Done Qwen 3.6-35B-A3B Upgrade DeltaNet linear attention, ~2x throughput vs 3.5, native thinking mode on/off.
✅ Done Auth Fully Enabled Both AUDITTRACE_AUTH_ENABLED and AUDITTRACE_AUTH_REQUIRED active. Per-route scope enforcement.
✅ Done Backlog Cleared All 7 tech-debt items resolved: streaming generator decomposed, step counter scoped per-trace, session ID hash widened, empty stubs deleted, dev compose overlay added.
✅ Done Package Rename sovereign_memory → audittrace, SOVEREIGN_* → AUDITTRACE_*, all containers/DB/realm/scopes aligned (ADR-035).
✅ Done N=100 Eval Sweep Full 100 probes × 2 modes with 30-min client timeout. Validates Qwen 3.6 + ADR-034 per-chunk idle timeout.
✅ Done Kubernetes + Istio ZTA k3s + Istio mTLS + SPIFFE/SVID workload identity. Helm chart + deny-all AuthorizationPolicies. The v1.0 milestone.
✅ Done Vault as sole secret store HashiCorp Vault production-only secret store (ADR-043). Vault Agent injector renders pod-local secrets. Auto-unseal at boot.
✅ Done OIDC Auth Code + PKCE for browser UIs BFF-first first-party auth flow (ADR-042). Confidential clients only.
✅ Done Laptop-first deploy Chart deploys on a single-node k3s without LAN-IP hardcodes (ADR-045). Demo-ready off-LAN.
✅ Done External IdP federation (ADR-044) Keycloak brokering to Google Workspace / Okta / EntraID. Realm + provisioner + diagrams shipped 2026-04-26. M2 live evidence captured 2026-05-02 against a real Google Workspace tenant — federated user → Keycloak shadow → JWT (iss=audittrace.allaboutdata.eu:30952) → /v1/chat/completions 200 → audit row interactions.user_id = federated sub. ADR-044 status flipped to Accepted.
✅ Done Async Persistence (ADR-046) Background asyncio worker + scoped Postgres ContextVar so RLS bites on background writes. Caught by live test 2026-05-04 — InMemoryPostgresFactory had no RLS.
✅ Done Server-Side Embedding (ADR-047) Embeddings computed in-process via the bundled ONNX model; ChromaDB never sees the raw text.
✅ Done Test + Evidence + Reconstructibility Gate (ADR-049) Hard four-rule gate: unit + per-file ≥90% + E2E through public API on the deployed image + reconstruction artefacts (trace ID, audit row, ChromaDB / MinIO / Postgres query). Mechanical enforcement via pre-commit hook + CI evidence-check job + PR template.
✅ Done PDF tier-A signature validation PAdES audit-grade signature checks via pyhanko; 10-class taxonomy (signed_valid/invalid/untrusted/expired/tampered, none, check_*); per-chunk + manifest signature_status column.
✅ Done PDF tier-B robustness (ADR-050) Encrypted-PDF reject (no password endpoint), embedded-attachment quarantine (PDF/A-3, ZUGFeRD), AcroForm field extraction, OCR-coverage tracking, document-level manifest (page_count, ocr_coverage_pct, attachment_count, form_field_count, extraction_warnings JSONB with GIN index, document_sha256). 16-code closed-set warnings enum pinned by TestExtractionWarningCodes.
✅ Done PAdES trust store + Swiss TSL (ADR-052 + ADR-053) Pluggable TrustStoreProvider / TrustStoreBuilder ABCs (S3 + Mock + composite). EU LOTL walker via pyhanko[etsi] 830 + Swiss federal TSL fetcher (OFCOM/BAKOM TSLO) → composite ~897 EU + CH qualified-signature CAs. POST /system/trust-store/refresh admin endpoint + Helm post-install hook. Live evidence: main_signed.pdf flipped from check_unavailable → signed_invalid → signed_valid after refresh ran.
✅ Done As-of-signing-time validation (ADR-054) 9th class signed_expired: signature chain validates as-of self-reported signing time but the cert is no longer valid at present. 5-min Kerberos-style clock-skew tolerance handles cert NotBefore that lags signing time by seconds.
✅ Done Version-pin litany eliminated (ADR-055) Single chart-side source of truth (Chart.AppVersion == pyproject.toml::version). make release VERSION=… bumps both + regenerates OpenAPI snapshot + runs drift gate. service.version OTel attribute templated from Chart.AppVersion so it never drifts again. Dockerfile pip install --no-deps . closes the v1.0.13 stale-egg-info bug; .dockerignore rejects .venv / dist-info / __pycache__.
✅ Done Recursive self-audit (ADR-058) The recorder records its own security-assessment runs as first-class, owner-scoped, trace-linked assessment audit rows via POST /assessments under a dedicated least-privilege scope (audittrace:assessment:ingest); raw payload content-addressed to assessments/{assessment_id}/{sha}.json. Governability of the recorder becomes reconstructible through its own front door.
✅ Done PDF tier-C batch (ADR-056) 9/16 tier-C items shipped: document metadata extraction (pdf_title / pdf_author / pdf_creator / pdf_creation_date from pymupdf doc.metadata, Alembic 011); closed-set corruption taxonomy (pdf_corrupted_xref / pdf_corrupted_structure / pdf_metadata_parse_error); per-document audit granularity via ?details=true on /memory/index (per-doc ok/error/chunks/signature_status/page_count/extraction_warnings/document_sha256/pdf_*/pdfa_*/ltv_data); dry-run mode (?dry_run=true); LTV summary (DSS dictionary counts); PDF/A conformance (pdfa_part + pdfa_conformance from XMP); TOC-aware chunking (toc_section per-chunk metadata); file-level surgical re-index (?file= path); #11 closed as subsumed by ADR-052 9-class taxonomy. Live on v1.0.17, evidence captured in 2026-05-09-adr-056-tier-c/.
⏸ On hold ADR-031 Per-Request Memory-Mode Routing N=100 eval (2026-04-17) showed tools wins every category including ambiguous. Routing complexity not justified. Revisit only if a future model reintroduces a category gap.
✅ Done ADR-048 Content-Control External-PDF parser-exploit gap closed via a sibling pod (audittrace-content-control, separate identity, separate MinIO IAM role). Memory-server side: PR-B1 (contract scaffolding) + PR-B2 (Vault / bucket-init / quarantine denylist) + PR-B2.5 (RabbitMQ subchart + AMQP topology + ADR-057 broker decision) + PR-B3 (/memory/upload PDF rewrite to 202 + Hohpe outbox + asyncio.Queue producer + janitor) + PR-B4 (verdict + audit consumers — memory_items.scan_status transitions to terminal state, interactions row with event_class='security' written for SOC alerting) + PR-B5 (WebUI chips + Bruno + reconstructibility walkthrough) + PR-B7 (MinIO IAM split — audittrace_app policy with explicit Deny on s3:GetObject against quarantine/* enforces Decision rule §1 at the bucket-policy layer). Content-control side: PR-A1 skeleton + PR-A2 ClamAV adapter + PR-A3 async pipeline + PR-A4 operator surfaces (/v1/scan/stats, /v1/scan/status, /v1/scan/retrigger). ADR-048 flipped to Accepted 2026-05-10. Live cluster validation captured in a separate chart-flip PR (scan_pipeline_enabled=true + EICAR end-to-end).
🟡 Planned Tier-C deferred #2 mixed text/image, #3 tables, #4 multi-column, #5 RTL, #17 hybrid/linearised, #19 per-page memory, #20 embedding throughput. None blocks the M5 customer demo (2026-05-15) — each will land as its own small ADR.

Observability Stack (ADR-028)

The observability backends (Langfuse, Tempo, Loki, Prometheus, Grafana) are a sibling Docker Compose stack — shared across deployment profiles and living in its own repository at lfdesousa/AiSovereignObservability. Clone alongside this repo.

The telemetry shippers (OTel Collector, Promtail) run inside whichever runtime hosts memory-server:

Runtime OTel Collector Promtail (log shipper)
k3s (production) DaemonSet in the AuditTrace-AI Helm chart (charts/audittrace/templates/observability/otel-collector-daemonset.yaml) DaemonSet in the AuditTrace-AI Helm chart (charts/audittrace/templates/observability/promtail-daemonset.yaml), Kubernetes pod-log discovery
Docker Compose dev Container in the observability-stack repo Container in observability-stack, Docker-socket discovery

Backends are the same in both cases; the shippers differ because their discovery mechanism differs.

# Start the sibling backends stack
./scripts/setup-observability.sh

# Stop
./scripts/setup-observability.sh --down
Backend URL Purpose
Grafana http://localhost:3001 (admin / audittrace) Dashboards
Prometheus http://localhost:19090 Metrics + PromQL queries
Loki http://localhost:3100 Log aggregation + LogQL queries
Tempo http://localhost:3200 (HTTP query API; OTLP ingest on :14318) Trace storage + service-map
Langfuse http://localhost:3000 LLM-specific observations (generations, tool calls)

Data flow (both runtimes):

  • Memory-server emits OTLP traces + metrics to the in-runtime OTel Collector. The Collector fans traces out to Tempo + Langfuse and pushes metrics to Prometheus via a scrape endpoint on :8889.
  • The in-runtime Promtail ships pod/container stdout+stderr to Loki (label set: namespace, pod, container on k3s; compose_service, container_name on Docker Compose).
  • Langfuse also receives LLM-specific traces via the Langfuse SDK directly from memory-server (separate from the OTLP pipeline — ADR-014.4).
  • Grafana queries Prometheus + Loki + Tempo for the unified operations dashboard and the call-flow service map.

Pre-provisioned dashboards: "Sovereign AI Operations" (P50/P95/P99 latency, error rates, tokens/sec, peak-tokens KV high-water, MinIO ops, container error logs) and "AuditTrace-AI — Call Flow" (Tempo service graph + recent traces).

Memory Seeding (ADR-027)

After a fresh clone or when knowledge changes:

# Generate secrets (includes MinIO KMS key)
./scripts/setup-secrets.sh

# Upload to MinIO + index into ChromaDB
# Same script targets either runtime — env vars determine endpoint.
python scripts/seed-memory.py --user-id <keycloak-sub-claim>

# Index only (skip MinIO upload)
python scripts/index-chromadb.py --user-id <keycloak-sub-claim>

# Selective re-index
python scripts/index-chromadb.py --collections decisions skills

# Preview without writing
python scripts/index-chromadb.py --dry-run --user-id <sub>
Collection Source Content
decisions docs/ADR-*.md Architecture Decision Records
skills ~/work/claude-config/skills/ Domain knowledge skill files
ai_research ~/work/ai-knowledge/ Research papers (PDF + text)
scm_coursework ~/work/scm-knowledge/ MIT SCM coursework (transcripts, notes, slides)

Auditing conversations (/interactions)

The /interactions endpoint is the human-facing view over the audit trail — distinct from the model-facing recall_* tools. The LLM never calls it; it's what you (or a dashboard) curl when you want to browse or filter history.

# Set BASE per your runtime:
BASE=https://audittrace.local:30952      # k3s
# BASE=https://localhost                  # docker-compose

# Every row the caller is allowed to see, newest first
curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/interactions?limit=20"

# Scoped to one project
curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/interactions?project=AuditTrace-AI&limit=50"

# Everything in one session
curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/interactions?session_id=opencode-2026-04-14-d1c6f4a0f67f4f67"

# Rolling window since a point in time
curl -sk -H "Authorization: Bearer $TOKEN" "$BASE/interactions?since=2026-04-14T00:00:00&limit=100"

Filters: project, user_id, session_id, source, since (ISO-8601). Pagination: limit (1-1000, default 100) + offset. Response: {interactions: [...], total, limit, offset} — total reflects the filter match count for the caller.

Per-user isolation is automatic. Postgres RLS sets app.current_user_id from the caller's Keycloak sub on every transaction (ADR-026). You cannot accidentally see another user's rows, even with a forged user_id query parameter — the RLS policy intersects WHERE user_id = current_setting(...). Scope required on the token: audittrace:audit.

Project tagging contract (ADR-029). Every interaction carries a project tag resolved on entry: X-Project header → body.metadata.project → body.project → "default". Configure the header per-project once via scripts/configure-project.py <name>.

Failure rows are audited too (migration 007 / ADR-033 seed, 2026-04-16). Every upstream failure on /v1/chat/completions — proxy timeout, llama-server error, connect failure, unexpected exception — produces an interaction row with status='failed' and a controlled-vocabulary failure_class. Pre-migration, silent streaming/tools-mode timeouts left zero audit trail. Query SELECT * FROM interactions WHERE status='failed' ORDER BY timestamp DESC to enumerate the failure history. 5xx responses on non-streaming paths now carry a 3-audience envelope (code, status, message, operator_hint, trace_id, user_facing_message) — the operator pivots from the trace_id into Loki / Langfuse / Grafana.

Configuration

Defaults shown for each AUDITTRACE_* variable apply to both runtimes unless explicitly split into k3s / dev rows.

Database (ADR-020)

Variable Default Description
AUDITTRACE_POSTGRES_URL -- Full PostgreSQL URL. Production connects as the non-superuser audittrace_app role so RLS policies actually apply
AUDITTRACE_POSTGRES_PASSWORD -- Database password (required)
AUDITTRACE_APP_PASSWORD fallback to AUDITTRACE_POSTGRES_PASSWORD Dedicated password for the audittrace_app role
AUDITTRACE_CHROMA_URL k3s: http://audittrace-chromadb:8000 · dev: http://chromadb:8000 ChromaDB server URL
AUDITTRACE_CHROMA_TOKEN -- ChromaDB auth token
AUDITTRACE_REDIS_URL k3s: redis://audittrace-redis-master:6379/0 · dev: redis://redis:6379/0 Redis URL for TokenCache + ToolResultCache
AUDITTRACE_REDIS_PASSWORD -- Required for the TokenCache hot path
AUDITTRACE_TOKEN_CACHE_TTL_SECONDS 300 JWT TTL in TokenCache

Auth (ADR-022, ADR-023, ADR-026, ADR-032, ADR-042, ADR-044)

Variable Default Description
AUDITTRACE_AUTH_REQUIRED true When true, every request needs a valid JWT
AUDITTRACE_KEYCLOAK_ISSUER k3s: https://audittrace.local:30952/realms/audittrace · dev: http://keycloak:8080/realms/audittrace Must match the JWT iss claim
AUDITTRACE_KEYCLOAK_JWKS_URL (matches issuer + /protocol/openid-connect/certs) JWKS endpoint for RS256 validation (cached 5 min)
AUDITTRACE_JWT_AUDIENCE audittrace-server Must match the JWT aud claim
AUDITTRACE_KEYCLOAK_EXTRA_ISSUERS "" Comma-separated additional accepted iss values for IdP federation (ADR-044)

Memory-as-Tools (ADR-025)

Variable Default Description
AUDITTRACE_MEMORY_MODE tools tools = proxy-internal tool-call loop (current default); inject = legacy 4-layer dump into system message. See Memory Access Modes.
AUDITTRACE_MEMORY_TOOL_LOOP_MAX_ITERATIONS 5 Hard cap on tool-call round-trips per request
AUDITTRACE_MEMORY_TOOL_CACHE_TTL_SECONDS 900 Redis tool result cache TTL; 0 disables caching
AUDITTRACE_TOOLS_CONFIG_PATH tools.toml Optional TOML overlay to disable/rename/retune tools

Observability (ADR-014.4)

Variable Default Description
AUDITTRACE_LOG_LEVEL INFO Log level
AUDITTRACE_OTLP_ENDPOINT "" (no export) OTLP/HTTP collector URL
AUDITTRACE_OTEL_SERVICE_NAME audittrace-server OTel service name
AUDITTRACE_TRACING_ENABLED true Enable OTel tracing
AUDITTRACE_METRICS_ENABLED true Enable OTel metrics

Langfuse (ADR-021.2)

Variable Default Description
AUDITTRACE_LANGFUSE_HOST k3s: http://10.42.0.1:3000 (host gateway from cluster) · dev: http://host.docker.internal:3000 Langfuse server URL
AUDITTRACE_LANGFUSE_PUBLIC_KEY -- Langfuse public key
AUDITTRACE_LANGFUSE_SECRET_KEY -- Langfuse secret key
AUDITTRACE_LANGFUSE_ENABLED false Enable Langfuse integration

Vault (ADR-043, k3s only)

Variable Default Description
VAULT_ADDR http://audittrace-vault:8200 Vault address (set by Vault Agent injector — application code never touches this directly)
VAULT_ROLE audittrace-server Vault role granting read access to the secret/audittrace/* path

The Vault Agent injector renders secret/audittrace/postgres, secret/audittrace/chromadb, secret/audittrace/redis, secret/audittrace/minio, secret/audittrace/keycloak, secret/audittrace/langfuse into pod-local files, which the application reads via AUDITTRACE_* env vars (resolved from the rendered files at startup). Auto-unseal at boot is handled by the audittrace-vault-auto-unseal.service systemd unit (script: scripts/audittrace-vault-auto-unseal.sh).

Documentation

Architecture Decision Records

Architecture Diagrams

Operator Playbooks

Evaluation Reports

  • Memory Access Modes — 2026-04-14 smoke — inject vs tools baseline; tools wins on latency, reliability, and tool-selection accuracy. Follow-ups: LangGraph-style exit conditions + ADR-030 before re-measuring at full N=100.

Legal Disclaimer & Copyright

Copyright © 2026 Luis Filipe de Sousa (allaboutdata.eu). All rights reserved.

  • Independent Research: This project, including the "Sovereign Local AI Stack" framework and the AuditTrace-AI implementation, is the result of independent research conducted by the author in a private capacity.
  • Resource Separation: No employer data, confidential information, or corporate infrastructure was used in the development of this work.
  • Institutional Independence: The views, architectures, and code presented here do not represent the positions, strategies, or opinions of any current or former employer.
  • Licensing: While the core engine is licensed under AGPL v3 to support Open Science, the underlying theoretical framework and specific architectural designs (ADRs) remain the intellectual property of allaboutdata.eu.

"The software artefact that separates compliant deployments from merely well-architected ones."