feat: remote OpenAI-compatible inference (M1 MVP, wired) - #25
feat: remote OpenAI-compatible inference (M1 MVP, wired)#25barakplasma wants to merge 6 commits into
Conversation
Standalone HTTP surface for remote OpenAI-compatible inference. No app wiring yet — nothing imports this module, so behaviour is unchanged. This is milestone M1 from REMOTE_INFERENCE_PLAN.md cut to a minimum viable core: - normalizeBaseUrl: handles bare hosts with ports, hosts with a version path, and full endpoint URLs pasted from provider docs. Infers http for localhost/private ranges, https otherwise. Returns '' for unparseable input rather than a URL that fails confusingly later. - redactSecrets + OpenAICompatError: two independent redaction passes so both our own key and a token echoed back in a provider error body are scrubbed. Errors redact at construction, so an un-redacted message cannot escape by any route. Errors carry host only, never a full URL. - requestJSON: the single HTTP chokepoint. Omits Authorization entirely when the key is empty (local servers reject a bare "Bearer"). Retries 408/429/5xx and network TypeErrors; never retries 400/401/403/404/413. Honours Retry-After in both forms, clamped so a hostile value cannot hang the UI. Backoff sleep is abortable, so Cancel stays responsive. - parseEmbeddings: sorts by index — response order is not guaranteed, and getting it wrong silently pairs the wrong vector with the wrong file. - computeTargetSize / imageToDataURL: downscale to a JPEG data URI. The size arithmetic is split out as a pure function because jsdom has neither OffscreenCanvas nor createImageBitmap. - embedTexts / embedQuery / embedImages / probeDimension / openaiCacheNamespace. Deferred from full M1: wire-format auto-probing (images use the openai-multimodal shape only), the VLM caption pipeline, listModels, and multi-frame video. 54 unit tests against a stubbed fetch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
Deploying media-clusterer with
|
| Latest commit: |
c5c6641
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8d9ca3c5.media-clusterer.pages.dev |
| Branch Preview URL: | https://claude-openai-compatible-inf.media-clusterer.pages.dev |
CI status
Why MegaLinter is redSix repository-scope scanners fail, and all six fail for reasons that predate this branch:
Two independent confirmations that this is not caused by the PR:
Not fixing it hereBoth root causes are out of scope for this PR and neither is a one-line change:
Happy to take either on as its own PR. Generated by Claude Code |
Makes the openaiCompat module actually reachable: "Remote endpoint ·
OpenAI-compatible" is now a selectable vision model that embeds photos at
a user-configured endpoint instead of in the browser.
End-to-end path:
- Settings block for endpoint URL, API key, model, plus a Test connection
button that probes and reports the vector dimension.
- loadModelOnce() gains an 'openai' branch: consent gate, then probe the
endpoint for its dimension before anything is embedded.
- embedAll() downscales each image to a JPEG data URI and batches them
through /embeddings.
- embedText() routes search queries to the same endpoint, so index-time
and query-time vectors share an embedding space. No 'search_query:'
prefix — that scheme is nomic's, and it would embed a stray literal
word against any other model.
Key handling (ADR-0002): the key lives under its own storage key, never
in mc_settings, because sentry.ts reads that blob and saveSettings()
rewrites it constantly. Unchecked "remember" puts it in sessionStorage.
The input is cleared after entry so the key is not left in the DOM.
Consent: a modal naming the destination host gates the first upload, and
is remembered per host — agreeing to upload to a LAN box is not agreement
to upload to a third-party API.
Supporting fixes:
- zeroVector() replaces four hard-coded new Float32Array(768) sites. A
remote endpoint can return any width; a placeholder of the wrong width
would corrupt every cosine score in the run.
- Cached vectors whose width does not match the active dimension are
treated as misses, which is what lets the remote cache namespace leave
the dimension out and still be safe.
- isDownloadError() is no longer consulted in remote mode. It matches on
message text ('unauthorized', '403'), so an endpoint rejecting a key
would otherwise open the "upload your .onnx files" modal in a mode that
downloads no model at all.
- Enabling text search in remote mode no longer downloads the 134 MB
local text model, which was both wasted bandwidth and the wrong space.
- Declining consent re-enables the Load button instead of stranding it
disabled with no way to retry.
- Settings merge fills in the nested openai object, which the shallow
spread would otherwise leave undefined for existing users.
Verified in Chromium against a stub endpoint returning 512-d vectors
(deliberately not 768, to prove the probed dimension is used): 16/16
checks, no uncaught exceptions. 143 unit tests pass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
barakplasma
changed the title
Seven of the eight gitleaks findings on this branch were mine, not the pre-existing Unsplash key: the openaiCompat tests contain key-shaped literals, which the default generic-api-key rule matches. They cannot be avoided by renaming. Several of those tests exist to prove that key-shaped input is redacted, so the fixture has to look like a key for the assertion to mean anything. Allowlisted by path, scoped to that one file, so a real key committed anywhere else still trips the scanner. Verified locally: 8 findings -> 1, and the one left is the Unsplash key at app.ts:56 (commit ddc9934) that .gitleaks.toml intentionally does not hide. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
Correction: most of the gitleaks findings were mineMy earlier comment said all 7 gitleaks errors came from the Unsplash key in git history. That was wrong. I inferred it from the summary table instead of reading the findings. Running gitleaks locally against Seven of eight were my own test fixtures. Only the last one is pre-existing. Fixed in Also new: djlint (HTML)The
My settings block adds 9 errors, all H021, and both H023 are pre-existing and untouched. Those 9 exist because the new markup follows the same inline-style convention as every other Remaining MegaLinter failures, accurately this time
Generated by Claude Code |
The HTML descriptor started failing on this branch not because the markup
got worse, but because djlint had never run before — no earlier commit
touched an HTML file. Measured on both sides:
H021 H023 total
origin/main 55 2 57
this branch 64 2 66
H021 is "inline styles should be avoided". index.html is built entirely
from inline styles, which is why the base already carried 55 of them; the
9 added here follow the same convention as every other setting-item in
the file. The descriptor could not go green by fixing only those 9.
Adds .djlintrc ignoring H021, recording the convention the file already
follows. Kept as .djlintrc rather than HTML_DJLINT_ARGUMENTS in
.mega-linter.yml so a local `djlint index.html` gives the same answer CI
does.
Also replaces the two ‹/› entity references with the literal
characters, clearing the remaining H023 pair (the file is UTF-8 and
already uses literal glyphs like ↑ on the adjacent lines).
djlint now reports 0 errors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
Everything this PR introduced is now clean
djlintSet Also replaced the two What remains red, and why it isn't this PR's
All four predate the branch and fail identically on Local verification on Generated by Claude Code |
CI still reported 2 djlint errors after the last commit, where local runs
reported 0. Cause: MegaLinter ships a djlint older than 1.42, which still
carries two rules the current release has dropped or that do not apply
here. Reproduced by installing 1.41.0 locally.
The two, both pre-existing and untouched by this branch — origin/main
reports the same pair under the same version:
H031 2:0 Consider adding meta keywords. <html lang="en">
H006 712:4 Img tag should have height and width attributes.
<img id="modal-img"
H031 asks for a <meta name="keywords"> tag. Search engines stopped
honouring that around 2009, and djlint itself removed the rule in 1.42 —
satisfying it would mean adding a tag that does nothing.
H006 wants fixed width/height on #modal-img. That is the lightbox image;
it displays arbitrary photos at arbitrary aspect ratios and is sized by
CSS, so there are no correct values to hard-code. The rule targets layout
shift on fixed-size images, which this is not.
Verified 0 errors under djlint 1.36.4, 1.41.0, 1.43.2 and 1.44.0, so the
result does not drift when MegaLinter bumps its pin.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1e6780c1c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| setStatus(`Connected to ${openaiHost(baseUrl)} (${activeEmbeddingDim}-d).`) | ||
| setProgress(100) | ||
| setTimeout(() => setProgress(0), 500) | ||
| return |
There was a problem hiding this comment.
When the selected variant is openai and probeDimension succeeds, this return exits before any of the model-ready UI updates used by the other branches run. The click handler has already disabled Load, and Open Media/Demo/Resume are still disabled from page load while state.phase remains loading_model, so the normal remote path gets stuck at “Connected…” with no enabled way to choose files. Please mirror the ready-state/button updates before returning.
Useful? React with 👍 / 👎.
| await yieldMain() | ||
| } | ||
| state.vectors = cachedVectors.map((v) => v || new Float32Array(768)) | ||
| state.vectors = cachedVectors.map((v) => v || zeroVector()) |
There was a problem hiding this comment.
When a session was created in remote mode, embeddings were cached under currentCachePrefix() (for example @openai:host:model/), but the resume code above still uses '' for every non-Sapiens/non-Chrome variant. On resume with modelVariant === 'openai', the batch read misses the remote namespace and this line zero-fills the restored vectors, breaking search/reprojection while the UI reports a successful restore. Use the same OpenAI cache prefix (and width guard) here.
Useful? React with 👍 / 👎.
| if (isRemote) { | ||
| // imageToDataURL only closes bitmaps it created itself, so the | ||
| // one cached in state.thumbnails stays valid. | ||
| missInputs.push(await imageToDataURL(thumb, state.settings.openai.maxImageWidth)) |
There was a problem hiding this comment.
When a video frame can't be extracted in remote mode, the VIDEO branch falls through to the existing else that pushes f.file, but embedImages only accepts data URL strings and will serialize that File as the image_url.url value. A single unsupported/corrupt video can make the provider reject the whole remote batch and zero-fill otherwise valid items; remote should zero-fill/skip that video instead of enqueuing the File.
Useful? React with 👍 / 👎.
All three reported by Codex on f1e6780, all confirmed real. 1. Connecting dead-ended (P1). The openai branch of loadModelOnce returned straight after probing, skipping the ready-state handover every other backend performs. state.phase stayed 'loading_model' and Open Media / Demo / Resume stayed disabled from page load, so a successful connection left the user staring at "Connected…" with no way to pick files. Now mirrors the chrome-ai handover. 2. A failed video frame poisoned its whole batch (P2). When frame extraction failed, the VIDEO branch fell through to pushing f.file. embedImages expects data URIs, so JSON.stringify turned the File into {} and the provider rejected the request — zero-filling every valid image batched alongside it. Remote now zero-fills just that video. 3. Resume restored nothing in remote mode (P3). resumePrefix was a copy of currentCachePrefix() written before this backend existed, so it had no 'openai' case and fell through to ''. Every cached read missed the remote namespace and was zero-filled while the UI reported a successful restore — search and re-projection silently ruined. Replaced with a currentCachePrefix() call so the logic exists once, and applied the same width guard the embedding loop uses. The first was a gap in my own verification: the browser test asserted the status text said "Connected" and stopped there, never checking the user could act. Added four assertions covering the post-connect ready state, and confirmed they fail against the pre-fix build before fixing it. 20/20 browser checks, 143 unit tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WGZ5syBsRr54E7Cce6HthJ
|
All three Codex findings were real and are fixed in P1 — remote load never reached ready stateCorrect, and the worst of the three. The This was a hole in my own verification, not just the code: my browser test asserted the status text read "Connected" and stopped there. I've added four assertions for the post-connect ready state and confirmed they fail against the pre-fix build before applying the fix: P2 — failed video frame poisoned its batchCorrect. When frame extraction failed, the VIDEO branch fell through to P3 — resume restored nothing in remote modeCorrect, and the most insidious, since it fails silently. Rather than add a fifth branch to the copy, I replaced it with a Verification: 20/20 browser checks against a stub endpoint returning 512-d vectors, 143 unit tests, type-check and build clean. CI note: MegaLinter's Generated by Claude Code |
2 participants
Adds Remote AI Mode: a selectable vision model that embeds photos at a user-configured OpenAI-compatible endpoint instead of in the browser. Milestone M1 from
REMOTE_INFERENCE_PLAN.mdplus the minimum wiring from M2–M4 needed to make it usable.The path
Pick Remote endpoint · OpenAI-compatible in Settings → enter a base URL and model → Test connection → load photos. Images are downscaled to small JPEGs and embedded at the endpoint; search queries go to the same endpoint so both sides share an embedding space.
normalizeBaseUrllocalhost:11434), host with version path (openrouter.ai/api/v1), and a full endpoint URL from provider docs. Infershttpfor localhost/private ranges. Returns''for unparseable input rather than a URL that fails confusingly later.requestJSONAuthorizationentirely when the key is empty — local servers reject a bareBearer. Retries 408/429/5xx and networkTypeError; never 400/401/403/404/413. HonoursRetry-Afterin both forms, clamped to 20s. The backoff sleep is abortable, so Cancel stays responsive.redactSecrets+OpenAICompatErrorhost, never a full URL.parseEmbeddingsindex— response order isn't guaranteed, and getting it wrong pairs the wrong vector with the wrong file, invisible until someone notices the map is nonsense.Key handling and consent
The API key is not in
Settings. It lives under its own storage key, becausesrc/sentry.tsreadsmc_settingsandsaveSettings()rewrites that blob on nearly every UI interaction. Unchecked "remember" puts it insessionStorageso it dies with the tab; the input is cleared after entry so it isn't left sitting in the DOM.A consent modal naming the destination host gates the first upload, remembered per host — agreeing to upload to a box on your LAN is not agreement to upload to a third-party API.
Supporting fixes
zeroVector()replaces four hard-codednew Float32Array(768)sites. A remote endpoint can return any width, and a placeholder of the wrong width would corrupt every cosine score in the run.isDownloadErroris no longer consulted in remote mode. It matches on message text (unauthorized,403), so an endpoint rejecting a key would have opened the "upload your .onnx files" modal — in a mode that downloads no model at all.openaiobject, which the shallow spread would leaveundefinedfor existing users.Bugs found in review and fixed (
c5c6641)Three real defects, all reported by Codex on
f1e6780:openaibranch returned straight after probing, skipping the ready-state handover every other backend performs —state.phasestayedloading_modeland Open Media / Demo / Resume stayed disabled, so a successful connection left no way to pick files.FilewhereembedImagesexpects a data URI;JSON.stringifyrenders that as{}and the provider rejects the request, zero-filling every valid image batched alongside it.resumePrefixwas a copy ofcurrentCachePrefix()predating this backend, so it had noopenaicase and fell through to''— every read missed the namespace and zero-filled while the UI reported success. Replaced the copy with a call to the real function, since the duplication is what let them drift.Deferred
openai-multimodal(OpenRouter) shape only; vLLM, Jina, Infinity and llama.cpp each want a different body and will 400.probeWireFormat()is the bulk of remaining M1.listModels.computeTargetSizebounds width only, matchingcreateImageBitmap'sresizeWidth, so a tall portrait uploads more than a landscape image of the same width.embedBatchesthrows on the first failing batch rather than returning per-item results; a toast reports the first failure of each kind and the run zero-fills the rest.Testing
143 unit tests (76 existing + 67 new) against a stubbed
fetch: URL normalization, redaction including a provider echoing the key back, the full retry/no-retry matrix, timeout, cancel-during-backoff, response ordering, batch splitting, key storage, and per-host consent.20/20 browser checks in Chromium against a stub endpoint returning 512-d vectors — deliberately not 768, to prove the probed dimension is actually used rather than the old constant:
The four ready-state assertions were added after the P1 bug above, and confirmed failing against the pre-fix build before the fix was applied — the original suite asserted the status text said "Connected" and stopped there, which is exactly how that bug got through.
npm test,npm run type-checkandnpm run buildare all clean (the chunk-size warning is pre-existing onmain).imageToDataURLhas no unit test — it needs canvas APIs jsdom lacks, which is why the size arithmetic is split into a purecomputeTargetSize. It is exercised by the browser run.CI
Everything this PR introduced is green:
djlint0,betterleaks0,htmlhint,secretlint,trufflehog,syft,trivy-sbom,git_diff, plusvalidate-and-deployand Cloudflare Pages.MegaLinter is still red on five pre-existing findings that fail identically on
main—gitleaks1 (Unsplash key,ddc9934),checkov1,osv-scanner8,grype2,trivy1. Evidence and per-linter attribution in this comment; they need the key rotated and anoverridesentry for the transitiveadm-zip/sharpadvisories, both outside this PR.