From 2c7306cc481e3942739ddf2b4c4e50bfa3c8c0e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 10:13:11 +0000 Subject: [PATCH 1/3] docs: ADR + plan for OpenAI-compatible remote inference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an opt-in backend that runs inference against any OpenAI-compatible endpoint — localhost (Ollama, LM Studio, llama.cpp, vLLM, Infinity, MLX) or internet-facing (OpenRouter, Venice, Jina) — instead of in the browser. Documentation only, following the IMPROVEMENT_PLAN.md precedent. No source changes. The design embeds media directly rather than captioning it, so the remote path is a peer of the nomic/sapiens2 image-feature-extraction backends rather than a variant of chrome-ai. A second pipeline covers VLMs and video LLMs, which emit text rather than vectors. Only downscaled JPEG thumbnails are uploaded, and only after an explicit per-host consent gate. Two findings shape the design and are recorded in the ADR: - There is no single wire format for image embeddings. Five request shapes are needed to cover the named targets, so the format becomes a probeable configuration dimension rather than a hard-coded body. - Ollama's embedding endpoints are text-only (ollama#5304, open since June 2024), so it is served by the VLM pipeline, not the direct one. The UI must say so rather than surfacing an HTTP 400. The plan also catalogues nine latent defects the change would hit — among them isDownloadError() claiming API 401s and opening the HuggingFace upload modal, embedText() skipping L2 normalisation, and zero vectors outranking dissimilar items in cosine search — and lands them as a behaviour-preserving first milestone. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ --- AGENT.md | 9 + REMOTE_INFERENCE_PLAN.md | 471 ++++++++++++++++++ ...0002-openai-compatible-remote-inference.md | 229 +++++++++ docs/adr/README.md | 47 ++ 4 files changed, 756 insertions(+) create mode 100644 REMOTE_INFERENCE_PLAN.md create mode 100644 docs/adr/0002-openai-compatible-remote-inference.md create mode 100644 docs/adr/README.md diff --git a/AGENT.md b/AGENT.md index 3bd1315..09d239e 100644 --- a/AGENT.md +++ b/AGENT.md @@ -23,12 +23,21 @@ - **Logic**: Use the `IProjection` interface for dimensionality reduction algorithms. - **State**: Centralized in the `state` object in `src/app.ts`. - **Storage**: Use `src/db.ts` for IndexedDB operations. +- **Secrets**: API keys never enter `state`, `mc_settings`, a URL, or `console.error` — the last of these + is shipped to a remote error sink by `captureConsoleIntegration` (`src/sentry.ts:15`). See + [ADR-0002](docs/adr/0002-openai-compatible-remote-inference.md). +- **Embedding space**: index-time and query-time vectors must come from the same embedder with the same + prompt prefixes. Any change of embedder, model, or dimension must change the IndexedDB cache namespace + (`currentCachePrefix`, `src/app.ts:354`). ## Architecture ### Application Modes 1. **AI Mode** (default): Loads embeddings, runs projections, enables semantic search 2. **Viewer-Only Mode**: Skips AI, arranges by folder/date grid, no search +3. **Remote AI Mode** (proposed, opt-in): inference runs against a user-configured OpenAI-compatible + endpoint instead of in the browser — downscaled JPEG thumbnails are uploaded. See + [ADR-0002](docs/adr/0002-openai-compatible-remote-inference.md) and `REMOTE_INFERENCE_PLAN.md`. ### State Management - All state in `state` object (phase, files, vectors, points, clusters, thumbnails, settings) diff --git a/REMOTE_INFERENCE_PLAN.md b/REMOTE_INFERENCE_PLAN.md new file mode 100644 index 0000000..2bd801c --- /dev/null +++ b/REMOTE_INFERENCE_PLAN.md @@ -0,0 +1,471 @@ +# Plan: Remote OpenAI-Compatible Inference + +Implementation plan for [ADR-0002](docs/adr/0002-openai-compatible-remote-inference.md), which decides to +add a `openai` model variant that runs inference against any OpenAI-compatible endpoint — localhost or +internet — instead of in the browser. Written 2026-08-01. +This document is a plan only — no changes are implemented by the PR that introduces it. + +Milestones are `M0`…`M7`, in dependency order. `M0` is a pure refactor that stands alone, and `M1` is a +self-contained module with no app wiring. + +## Context + +**Current problem:** + +- Inference is browser-only. The default `sapiens2-fp16` costs a 229 MB download before the first photo + is processed (`src/app.ts:206`, `src/sapiens2.ts:17-34`), and throughput is bounded by the local GPU — + or by WASM when WebGPU is missing (`src/app.ts:1175-1181`, `src/sapiens2.ts:218-224`). +- `customModelHost` (`src/types.ts:61`, `src/app.ts:2896-2901`) looks like it addresses this but only + redirects **model file downloads** to a HuggingFace-compatible mirror. Inference still runs locally. +- Models that would most improve clustering and search — purpose-built multimodal retrieval models, video + LLMs — cannot run in a tab at all. +- Videos are represented by a single frame (`src/app.ts:504-555`), the same limitation + `VIDEO_LM_PLAN.md` documents. + +**Desired outcome:** + +- A user can point the app at `http://localhost:11434/v1`, `https://openrouter.ai/api/v1`, or anything + else, supply a key, and start embedding within seconds. +- Images are embedded directly where the provider supports it; VLMs and video LLMs are supported through + a second pipeline. +- Only downscaled JPEG thumbnails ever leave the device, and only after explicit consent. +- The API key cannot reach a log line, an error report, or a URL. + +## Implementation approach + +### Libraries to add + +**None.** `p-limit` is already a dependency and is already used for exactly this shape of concurrency +control (`src/app.ts:678`). Everything else is `fetch`, `createImageBitmap` and `OffscreenCanvas`. + +### The two pipelines + +| Pipeline | Setting | Request | Produces | Works with | +| --- | --- | --- | --- | --- | +| **A** `direct` | `openai.pipeline = 'direct'` | `POST {baseUrl}/embeddings` | vector | OpenRouter, vLLM, Jina, Infinity | +| **B** `vlm` | `openai.pipeline = 'vlm'` | `POST {baseUrl}/chat/completions` → then embed the caption | caption + vector | every chat provider, incl. Ollama; the only route for video LLMs | + +Pipeline B's second stage is itself configurable: remote `{baseUrl}/embeddings`, or the local +`nomic-embed-text` that the `chrome-ai` path already loads (`src/app.ts:1054-1082`). Local is the safer +default — it works against providers with no embeddings endpoint at all, and needs no second round-trip +per image. + +Both sit behind one interface so `embedAll()` gains one branch, not two: + +```typescript +export interface RemoteEmbedder { + embedMedia(items: MediaInput[], signal?: AbortSignal): Promise[]> + embedQuery(text: string, signal?: AbortSignal): Promise + readonly dim: number // probed at load, before any cache key is computed + readonly namespace: string // IndexedDB cache namespace +} + +export interface EmbedResult { vector: Float32Array; caption?: string } // caption: pipeline B only +export type MediaInput = + | { kind: 'image'; frames: [ImageBitmap] } + | { kind: 'video'; frames: ImageBitmap[] } +``` + +`embedQuery` **must** reach the same endpoint and model as `embedMedia`. See "Vector-space consistency". + +### Wire formats + +`/v1/chat/completions` with an `image_url` part is universal. Image input to `/v1/embeddings` is not — +see the table in ADR-0002. Model it as a discriminated union in `src/types.ts`: + +```typescript +export type EmbedWireFormat = + | 'openai-multimodal' // OpenRouter: input: [{ content: [{ type: 'image_url', image_url: { url } }] }] + | 'chat-messages' // vLLM: messages: [{ role: 'user', content: [...] }] + | 'jina' // Jina: input: [{ image: '' }] / [{ text: '...' }] + | 'plain-input' // Infinity: input: [''] + | 'llamacpp' // llama.cpp: { content: 'Image: [img-1]', image_data: [{ id, data }] } +``` + +`probeWireFormat()` sends one 64×64 solid-colour JPEG through each in order and keeps the first that +returns a numeric vector, persisting the winner. This is what the **Test connection** button runs, and it +reports both the winning format and the resulting dimension. Default `'auto'`; a manual override exists +for endpoints where probing costs money. + +Response parsing handles `{ data: [{ embedding, index }] }` and llama.cpp's `{ embedding: [...] }`. +**Sort by `index`** — order is not guaranteed by the spec, and getting it wrong silently assigns the +wrong vector to a file, which is invisible until someone notices the map is nonsense. + +### Files to modify + +1. **`src/types.ts`** — extend `ModelVariant` with `'openai'`; add `OpenAISettings`, `EmbedWireFormat`, + `MediaInput`, `EmbedResult`, `RemoteEmbedder`, `OpenAICompatConfig`; add the new `DOMElements` refs. + Per `AGENT.md`, interfaces live here. +2. **`src/app.ts`** — new branch in `loadModelOnce()` (`:1013`); new branch in `embedAll()` (`:1433`) + alongside `isSapiens2` / `isChromeAI` (`:1442-1443`); `currentCachePrefix()` (`:354-360`) gains the + remote namespace; `embedText()` (`:623-634`) is rewritten onto a shared funnel; four + `new Float32Array(768)` sites become `zeroVector()`; lazy modal captions (`:2500-2553`) become + variant-aware; settings listeners (`:2760-2900`). +3. **`index.html`** — one `