docs: ADR + plan for a small video language model; ci: force MegaLinter auto-fix

[barakplasma]

Why

Two problems converge on one decision.

The captioner does not fit the target device. Captioning via Chrome's Prompt API (Gemini Nano) crashes an 8 GB Chromebook. Chrome's published floor is >4 GB VRAM, or 16 GB RAM + 4 cores on CPU, plus 22 GB free disk. The size is not negotiable — the browser owns the model, so the app cannot ask for a smaller quantisation. Its only lever today is to disable the feature (src/app.ts:2696-2713).

The app claims video support it does not have. extractVideoFrame() (src/app.ts:504-555) seeks once to min(1.0, duration/2) and returns one 224 px bitmap. That still is the entire representation of the video. The caption prompt is image-specific in its wording too (src/chromeAI.ts:52-56) — it asks about colour and lighting, never motion.

What this adds

Two separate strands. No src/ changes — the source tree is untouched.

1. The ADR and plan (the main deliverable)

• docs/adr/README.md — establishes an ADR convention the repo lacked (numbering, immutability, status vocabulary, the src/file.ts:line citation style already used in IMPROVEMENT_PLAN.md).
• docs/adr/0001-small-video-language-model.md — the decision: adopt SmolVLM2 (256M / 500M) via @huggingface/transformers, in a Web Worker, in three tiers.
• VIDEO_LM_PLAN.md — six-milestone implementation plan, model facts, risks, verification checklist.
• AGENT.md gains a docs/adr/ pointer, and its Tech Stack section is corrected — it claimed Nomic while the default is sapiens2-fp16. IMPROVEMENT_PLAN.md P1-1 cross-references milestone M5.

Implementation lands in later PRs, per the IMPROVEMENT_PLAN.md precedent.

2. MegaLinter auto-fix (requested mid-PR)

.mega-linter.yml already set APPLY_FIXES: all, but the workflow never acted on it — fixed files were only copied into the report artifact, so every run re-reported the same drift and nothing was ever fixed. .github/workflows/mega-linter.yml now sets APPLY_FIXES_EVENT: all / APPLY_FIXES_MODE: commit with a commit-back step.

Three details that took a round each to get right:

• !cancelled() on the commit step. MegaLinter exits non-zero whenever any linter errors — here the security scanners always do — and a failed step skips every later step by default. Without this the fixes were never pushed.
• file_pattern: ':!megalinter-reports' plus a .gitignore entry. The first auto-fix commit correctly fixed AGENT.md, but also swept in MegaLinter's own run output. Belt and braces so it cannot recur.
• commit_author pinned. The action defaults it to github.actor, attributing a machine-generated commit to a human.

Fork PRs resolve FIX_BRANCH to empty, so they are linted but never pushed to. MegaLinter is already on the latest release (v9.6.0) — no bump was needed.

The decision, briefly

Tier	ModelVariant	Download (q4f16)	Produces
A	smolvlm2-vision	55 MB	768-d vectors only
B	smolvlm2-256m	189 MB	Captions and vectors
C	smolvlm2-500m	358 MB	Same, higher quality

@huggingface/transformers is already a dependency at ^4.2.0 and SmolVLM landed upstream in 3.4.0 — no dependency bump.

The load-bearing insight: SmolVLM2's vision_config.hidden_size is 768, exactly the app's existing vector width. When the captioner runs, the vision tower runs anyway — so pooling its output yields the caption and the clustering vector from a single forward pass. That is what drops nomic-embed-text (134 MB) from the default path. Today's chrome-ai chain needs two models to produce one vector.

Multi-frame video is affordable: pixel_shuffle_factor: 4 reduces each frame's 1024 patches to 64 visual tokens, so 4 frames cost ~256 tokens of prefill against a 576-hidden decoder — not the 4096 a naive reading suggests.

For scale, tier A at 55 MB is the cheapest embedder the app has ever had (vs 116 MB sapiens2-int8, 380 MB nomic).

Trade-offs stated in the ADR

• Lost: the shared image/text embedding space. Semantic search is re-designed — enableTextSearch (which already gates the text-model download) flips to off by default with lexical caption search; turning it on restores today's behaviour exactly.
• Unproven: mean-pooled SigLIP features are a general-purpose signal, not a purpose-trained retrieval embedder. Milestone M2 ships tier A standalone precisely so clustering quality is measured against sapiens2-fp16 before anything depends on it.
• New ground: this introduces the repo's first Web Worker.
• Gained for free: search-by-example over the pooled vision vectors.
• chrome-ai is supplemented, not replaced — it stays selectable and unchanged.

One assumption flagged for day-one confirmation

vision_encoder.onnx at fp32 is 374 MB ≈ 93.6 M params ≈ the bare SigLIP-base tower, implying the export stops before the pixel-shuffle connector and emits 768-d embeddings. That is inferred from file size, not read off the graph. The plan makes confirming session.outputNames + dims an explicit M2 day-one step, with a visionDim tier field as the fallback if it turns out to be 576-d.

Verification

• npm run type-check clean; npm test 76/76 passing
• actionlint ✅ 0 and zizmor ✅ 0 on the changed workflow — including the deliberate persist-credentials: true, which job-scoped permissions and an ignore[artipacked] make explicit rather than accidental
• markdownlint / markdown-table-formatter / prettier / yamllint / v8r all ✅ 0 errors, 0 fixed — i.e. the auto-fixer has converged and now pushes nothing
• Every src/file.ts:line citation in both documents was checked to resolve to the code it claims

The red checks are pre-existing

Both failed identically on main at f63dd37, the commit this branch is based on.

MegaLinter — gitleaks (1), grype (2), osv-scanner (2), trivy (1). The leaked UNSPLASH_ACCESS_KEY at src/app.ts:56 dates from April and needs rotating, not deleting; adm-zip and sharp need version bumps. Both warrant their own PR.

Cloudflare Pages — resolved. Automatic deployments have now been disabled on the Pages project, so the git-integration build no longer runs. The failed check remains frozen on 33131ff because check runs are immutable once posted; later commits simply won't have the check.

The cause is worth recording, because the dashboard was actively misleading. From the build log:

Found wrangler.toml file. Reading build configuration...
pages_build_output_dir: dist
No build command specified. Skipping build step.
Error: Output directory "dist" not found.

The Pages dashboard did have Build command: npm run build configured. But wrangler.toml contains pages_build_output_dir, and per Cloudflare's docs that file then becomes the source of truth — "you can not edit the same fields in the dashboard once you are using this file." Since wrangler.toml has no build-command key for Pages, the dashboard value was inert: displayed as correct, ignored by the builder. So the build was skipped, and validation then failed on dist/, which is .gitignored and therefore never in the clone.

Not a broken build — a build that never ran. Throughout, deploy.yml built, tested and deployed every commit on this branch successfully via wrangler pages deploy direct upload, which is unaffected and now the only publishing path — with type-check and npm test gating it, which the git build never did.

[cloudflare-workers-and-pages]

Deploying media-clusterer with    Cloudflare Pages

Latest commit:	96a6ce3
Status:	✅  Deploy successful!
Preview URL:	https://050d81b0.media-clusterer.pages.dev
Branch Preview URL:	https://claude-video-language-model.media-clusterer.pages.dev

View logs

[barakplasma]

CI status: MegaLinter red on the base branch too; Cloudflare build is not reproducible locally

MegaLinter — all four failures pre-date this PR

I checked before touching anything. The MegaLinter run on main at f63dd37 — the exact commit this branch is based on — fails with the identical set:

Linter	main (run 30296874499)	This PR (run 30694708397)
gitleaks	❌ 1	❌ 1
grype	❌ 2	❌ 2
osv-scanner	❌ 2	❌ 2
trivy	❌ 1	❌ 1
markdownlint	⚠️ 2 errors	✅ 0 errors
markdown-table-formatter	✅	✅

Same linters, same counts. This PR changes no source files, and the markdown linters — the only ones that look at what it actually adds — pass.

gitleaks (1) — UNSPLASH_ACCESS_KEY at src/app.ts:56, from commit ddc9934 dated 2026-04-06. gitleaks scans all 186 commits, so it fires on every branch regardless of content. Already documented in IMPROVEMENT_PLAN.md's security note. Not fixable from here for two reasons: the key needs rotating by the account owner (deleting the line doesn't un-leak it), and purging it from history means rewriting 186 commits.

grype / osv-scanner / trivy (the same 2 vulns, triple-reported) — both transitive deps in package-lock.json, neither reachable from shipped app code:

• adm-zip 0.5.17 → 0.6.0 (GHSA-xcpc-8h2w-3j85, DoS via crafted ZIP)
• sharp 0.34.5 → 0.35.0 (GHSA-f88m-g3jw-g9cj, inherited libvips CVEs)

Both are fixable with overrides entries, but that edits package-lock.json and needs its own build-and-test cycle — a separate PR, not a rider on a docs change. Say the word and I'll open one.

Cloudflare Pages — builds clean locally

npm run build (tsc && vite build) succeeds here on 5a6636f, exit 0:

dist/index.html                    39.48 kB │ gzip:   8.91 kB
dist/assets/index-BU1tDCTe.css     88.63 kB │ gzip:  11.75 kB
dist/assets/index-2owZurvl.js   1,187.69 kB │ gzip: 344.28 kB
✓ built in 3.25s

Since this PR adds only markdown, there is no plausible mechanism by which it breaks a Vite build. The failure detail is behind the Cloudflare dashboard, which I can't reach — if you open the log and paste the error I'll take it from there. My guess is a Pages-side config issue (missing VITE_UNSPLASH_ACCESS_KEY, or the project's build settings), not the commit.

Pushed on this branch

• 5a6636f — applied markdown-table-formatter's output to the three new docs so the branch matches what CI generates rather than drifting. Table separator padding only; git diff --ignore-all-space shows nothing but | --- | → |-----|.
• 07fda8b — forces MegaLinter to auto-fix and commit back, per request. APPLY_FIXES: all was already set in .mega-linter.yml, but the workflow never acted on the result: fixed files were only copied into the report artifact, so every run re-reported the same drift. Now APPLY_FIXES_EVENT: all + APPLY_FIXES_MODE: commit with a commit-back step. Checkout targets the branch rather than the PR merge commit and persists credentials so the push can happen; contents: write is scoped to the job. Fork PRs get an empty FIX_BRANCH, so they're linted but not fixed.

MegaLinter is already on the latest release, v9.6.0 — the pinned SHA ef3e84b is exactly v9.6.0, so there was nothing to bump. The "MegaLinter 9.5.0 is out!" banner in the log is stale text baked into the image.

One expectation to set: forcing auto-fix will not turn this check green. All four remaining failures are security findings that no formatter can rewrite — they need the key rotated and the two deps bumped.

---

Generated by Claude Code

[barakplasma]

Cloudflare Pages: two build paths, and only the dashboard-configured one is broken

Diagnosis for the Cloudflare Pages check — it is not caused by anything in this PR, and it isn't fixable from the repository.

This project has two independent build paths pointed at the same Pages project:

1. .github/workflows/deploy.yml — npm ci → type-check → test → npm run build → npx wrangler pages deploy dist --project-name media-clusterer. This is a direct upload.
2. Cloudflare's git integration — Cloudflare clones the repo itself and runs whatever build command is set in the Pages dashboard.

Path 1 succeeded on every commit of this branch:

Commit	CI/CD - Deploy to Cloudflare Pages	Cloudflare Pages (git integration)
72de8bd	✅ success	❌ failure
5a6636f	✅ success	❌ failure
07fda8b	✅ success	❌ failure

So on the same commits, the same npm run build compiles and deploys to the same project through wrangler — while Cloudflare's own clone-and-build fails. That rules out the code: a build cannot both succeed and fail on identical input unless the two builds are configured differently.

It also builds clean locally at 07fda8b, exit 0:

dist/index.html                    39.48 kB │ gzip:   8.91 kB
dist/assets/index-BU1tDCTe.css     88.63 kB │ gzip:  11.75 kB
dist/assets/index-2owZurvl.js   1,187.69 kB │ gzip: 344.28 kB
✓ built in 3.25s

Why I'm not fixing it here

The failure is in the Pages project settings, not in any file in this repo. The usual cause for a project created via wrangler pages deploy that later got a git connection is that the git build has no build command and no output directory configured, so it publishes an empty/So the wrong tree — or fails outright. Two ways to resolve it, both in the dashboard:

• Disconnect the git integration (Settings → Builds & deployments), leaving deploy.yml as the single source of deploys. Given the Action already deploys per-branch with --branch, the git integration is redundant. This is what I'd recommend.
• Or configure it: build command npm run build, output directory dist, and add any VITE_* env vars the build expects (.env.example lists VITE_UNSPLASH_ACCESS_KEY).

Cloudflare's check posts only a dashboard link and no error text — output.text on the check run is empty — and the dashboard needs credentials I don't have. If you open the log and paste the error, I'll confirm or correct the above.

Status of this PR

Everything it is responsible for passes: type-check clean, 76/76 tests, markdownlint 0 issues, and the deploy workflow green on all three commits. The two red checks — MegaLinter and Cloudflare Pages — are both pre-existing infrastructure issues, detailed in the previous comment and above.

---

Generated by Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 33131ff6a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Re-run checks on the auto-fix commit

When a same-repository PR or main run applies fixes, this push uses the persisted repository GITHUB_TOKEN, but events generated by that token do not trigger another workflow run. The generated commit therefore receives no MegaLinter validation, contrary to this convergence assumption; on a PR it may also become the new head without required checks. Use a GitHub App/PAT that can trigger workflows, or explicitly validate the rewritten tree before pushing. See GitHub's GITHUB_TOKEN documentation.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Compare semantic queries in a compatible vector space

For any smolvlm2-* tier, state.vectors will contain pooled SigLIP vision features, while today's embedText() produces Nomic text vectors and searchImages() directly compares those vectors in src/app.ts:623-654. As the ADR itself notes at lines 131-133, these spaces are not aligned, so enabling this option cannot restore today's behavior and will instead return meaningless rankings. The plan needs either Nomic search_document vectors for the captions or a text encoder aligned with the chosen vision vectors.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Preserve a low-resolution thumbnail extraction path

For folders containing videos, the existing thumbnail path calls extractVideoFrame() at src/app.ts:606; making it a wrapper around extractVideoFrames(file, 1) means it returns the 512-pixel VLM frame specified at lines 271-274 rather than today's 224-pixel bitmap. That increases each cached video thumbnail's pixel memory by more than 5× and contradicts the separate display/VLM resolutions described above, undermining the low-memory target. Keep a distinct thumbnail-sized extraction or parameterize the requested resolution.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Supply captions before enabling lexical search

On devices without WebGPU, this milestone withholds tiers B/C at line 304, leaving tier A, which explicitly produces no captions. A lexical index over captions is therefore empty on precisely the low-end fallback tier, so the default search and verification step 7 cannot return any results. The plan should provide a caption source for tier A, retain a compatible fallback search route, or explicitly disable text search for that tier.

Useful? React with 👍 / 👎.

[barakplasma]

All four Codex comments were valid. Fixed in 96a6ce3 — three were real design flaws, one a false comment in the workflow.

P1 · semantic search vector space — correct, and the most important catch. searchImages() calls searchByCosine(queryVector, state.vectors) directly (src/app.ts:653). That is sound today only because in chrome-ai mode state.vectors already are nomic-text vectors (src/app.ts:1590-1594). Under a smolvlm2-* tier they hold pooled SigLIP features, so a nomic-text query gives a well-defined cosine over an arbitrary ranking. My claim that enabling enableTextSearch "restores today's behaviour exactly" was wrong. Plan and ADR now specify a parallel state.searchVectors built by embedding captions with search_document:, kept strictly separate from state.vectors, plus the corollary: tier A has no captions, so it cannot offer semantic text search at all.

P2 · thumbnail resolution — correct, and an internal contradiction in my own plan: I wrote a "two resolutions, two purposes" design decision and then an implementation step that collapsed them. extractVideoFrame() as a wrapper around a 512 px extractVideoFrames() would have promoted every cached video thumbnail from 224 px, >5× pixel memory across a 2000-entry LRU, on the low-memory device the plan exists to serve. Resolution is now a parameter (THUMB_FRAME_PX / VLM_FRAME_PX). I also added the adjacent trap: VLM frames must not be written into state.thumbnails, which today's code does at src/app.ts:1484-1486 and would reintroduce the regression by the back door.

P2 · empty lexical index on tier A — correct. Tier A produces no captions and pickVlmTier() selects it whenever WebGPU is absent, so the caption index would be empty precisely on the mainstream fallback, and verification step 7 was unsatisfiable there. The index now falls back to filename, folder and EXIF/date (already parsed and memoised on PhotoFile, src/app.ts:2454-2479), and the text-search toggle is disabled with a reason rather than silently returning nothing. M6 now carries a per-tier capability table.

P1 · re-run checks on the auto-fix commit — correct on the mechanism. I'd observed this independently (the run created for 6d52b56 sat in action_required and never executed) but left a workflow comment asserting the opposite: "The follow-up push re-runs this workflow once." That comment is now fixed. It rules out an auto-fix loop by construction, but the tradeoff — the fix commit is never linted, and on a PR becomes the head without checks — is real and now documented inline, with the App-token/PAT remedy noted. Leaving the token choice to the repo owner rather than widening CI credentials unprompted.

Docs-only change; src/ still untouched. type-check clean, 76/76 tests, markdownlint 0 issues.

---

Generated by Claude Code