Skip to content

feat: pluggable storage adapter — R2, GCS, S3-region config - #47

Open
amal66 wants to merge 1 commit into
upstream-mainfrom
upstream-pr/storage-adapter-gcs
Open

feat: pluggable storage adapter — R2, GCS, S3-region config#47
amal66 wants to merge 1 commit into
upstream-mainfrom
upstream-pr/storage-adapter-gcs

Conversation

@amal66

@amal66 amal66 commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Summary

Route every storage operation through a small StorageAdapter interface so the object-storage backend can be swapped without touching call sites. Cloudflare R2 stays the default and behaves exactly as before; a new Google Cloud Storage adapter (ADC auth) can be installed with one setStorageAdapter() call at startup. The S3 client region becomes configurable via R2_REGION (default "auto", unchanged).

Changes

  • backend/src/lib/storage/adapter.ts — the StorageAdapter contract (upload / download / delete / list / getSignedUrl / checkReady).
  • backend/src/lib/storage/r2.tsR2StorageAdapter: the existing R2 code moved behind the interface. Same env vars, same "mike" bucket default, same error messages and null/empty fallbacks; region now reads R2_REGION (default "auto", the previous hardcoded value).
  • backend/src/lib/storage/gcs.tsGCSStorageAdapter: Google Cloud Storage via Application Default Credentials (GCS_BUCKET_NAME, GCS_PROJECT_ID, GCS_SIGNED_URL_TTL).
  • backend/src/lib/storage.ts — now a thin public API: delegates uploadFile / downloadFile / deleteFile / listFiles / getSignedUrl to the active adapter, adds setStorageAdapter() + checkStorageReady(), and re-exports everything callers already import.
  • backend/src/core/storagePaths.ts — the pure path/filename helpers (storageKey, buildContentDisposition, …) moved out of storage.ts; re-exported from lib/storage so no import changes anywhere.
  • backend/tsconfig.json — exclude src/**/__tests__/** and *.test.ts from the build (tests run under vitest, not tsc).
  • Tests: backend/src/lib/__tests__/storage.test.ts (25 tests, path/filename helpers), backend/src/lib/storage/__tests__/gcs.test.ts (16 tests, GCS adapter with a mocked SDK).
  • backend/package.json / package-lock.json — add @google-cloud/storage@^7.19.0.

Why

Deploying anywhere other than Cloudflare requires either S3-compatible credentials or a fork of storage.ts. The adapter makes the backend portable: GCS out of the box, and any other backend (Supabase, local FS, S3) by implementing one interface. New runtime dependency: @google-cloud/storage@^7.19.0 (already shipped by the fork). Zero behavior change unless configured: the default adapter is R2 with identical env vars, defaults, and fallbacks — verified by reading each method against the previous storage.ts; R2_REGION and the GCS vars only take effect when explicitly set.

Testing

  • npm install && npm run build (tsc) green on the branch as committed, from a clean node_modules.
  • With the vitest harness (upstream-pr/test-harness) merged locally: npm test → 3 files, 53 tests passed (25 storage + 16 gcs from this branch, 12 pre-existing downloadTokens).

Provenance

All changes are mechanical ports of amal66/mike@origin/main (commit b3166dd): apps/api/src/lib/storage/{adapter,gcs,r2}.tsbackend/src/lib/storage/, apps/api/src/lib/storage.tsbackend/src/lib/storage.ts, apps/api/src/core/storagePaths.tsbackend/src/core/storagePaths.ts, plus their tests. Exceptions: (1) the fork reads config through its zod lib/env module, which this base lacks — env reads were translated to direct process.env access with the fork's defaults (R2_BUCKET_NAME/GCS_BUCKET_NAME"mike", R2_REGION"auto", GCS_SIGNED_URL_TTL3600), and the two tests' vi.mock("../env") blocks became plain process.env set/delete; (2) the tsconfig test-exclusion line, matching the other upstream-pr branches.

Credits & prior art

  • @nforum (nforum/mike) — independently parallels this work: their fork added Google Cloud Storage support to Mike. Different implementation (this PR routes all backends through a StorageAdapter interface with R2 behavior preserved verbatim), same portability need.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC

Route all storage operations through a StorageAdapter interface. The
default R2StorageAdapter preserves existing R2 behavior exactly (same
env vars, same bucket default, same error handling); GCSStorageAdapter
adds Google Cloud Storage via Application Default Credentials.
setStorageAdapter() swaps backends without touching call sites. The S3
region is now configurable via R2_REGION (default "auto", unchanged).
Path/filename helpers move to core/storagePaths.ts and are re-exported
from lib/storage, so every existing import keeps working.

Mechanical port of the storage adapter work from amal66/mike@main
(commit b3166dd), translated to the upstream backend/ layout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
amal66 added a commit that referenced this pull request Aug 21, 2026
…in the app)

WHY THIS MATTERS
The hosted default makes a downloaded Mike.app usable immediately, but it
makes the app a client of the cloud. Some users (law firms with data-
residency rules, air-gapped environments, privacy-first individuals) want
the opposite: everything on their own machine, with the same one-click
download. Today that means running a docker-compose stack — a non-starter
for non-developers. This document is the researched plan for folding that
whole stack INSIDE the app.

WHAT IS A "SELF-CONTAINED" DESKTOP APP
Instead of the Electron shell loading a remote URL, the app bundles and
supervises its own local services — database, auth, API, storage — all on
loopback, and points the window at http://localhost. The user never knows
a server exists. The key design constraint carried over from the shell:
the web app and backend run byte-identical to the compose stack (shell-
over-rewrite); only the process manager changes from docker-compose to a
supervisor inside the app.

WHAT THE RESEARCH FOUND (the plan is cheaper than it looks)
- Postgres needs only pgcrypto + pg_trgm — no pgvector on main.
- The frontend touches Supabase ONLY for auth; all data flows through the
  backend, whose sole DB path is PostgREST (436 call sites). So bundling
  the real GoTrue + PostgREST + Postgres binaries (all permissively
  licensed, all arm64-mac-buildable) preserves every API contract with
  zero rewrites.
- No Redis/queues on main; document conversion degrades gracefully
  without LibreOffice; storage already has an adapter seam in flight
  (upstream PR #47 lineage) that a local-filesystem driver can plug into.
- LLM access already supports per-user keys and local Ollama; the keyless
  demo mode (PR open-legal-products#260) covers the first-run answer.

HOW THE PLAN IS STRUCTURED
Five phases, each independently shippable: (0) upstream seams as normal
web-app PRs — storage adapter + fs driver, migration runner, Next
standalone flag; (1) a local-stack supervisor in the shell with a "Run
locally on this Mac" mode on the connect screen; (2) packaging + signing
of the bundled binaries; (3) upgrade/backup lifecycle; (4) first-run
polish. Roughly 6-9 engineering weeks. Open product decisions (auth UX,
password recovery without SMTP, LibreOffice, Intel support) are listed
explicitly rather than silently decided.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
amal66 added a commit that referenced this pull request Aug 25, 2026
…in the app)

WHY THIS MATTERS
The hosted default makes a downloaded Mike.app usable immediately, but it
makes the app a client of the cloud. Some users (law firms with data-
residency rules, air-gapped environments, privacy-first individuals) want
the opposite: everything on their own machine, with the same one-click
download. Today that means running a docker-compose stack — a non-starter
for non-developers. This document is the researched plan for folding that
whole stack INSIDE the app.

WHAT IS A "SELF-CONTAINED" DESKTOP APP
Instead of the Electron shell loading a remote URL, the app bundles and
supervises its own local services — database, auth, API, storage — all on
loopback, and points the window at http://localhost. The user never knows
a server exists. The key design constraint carried over from the shell:
the web app and backend run byte-identical to the compose stack (shell-
over-rewrite); only the process manager changes from docker-compose to a
supervisor inside the app.

WHAT THE RESEARCH FOUND (the plan is cheaper than it looks)
- Postgres needs only pgcrypto + pg_trgm — no pgvector on main.
- The frontend touches Supabase ONLY for auth; all data flows through the
  backend, whose sole DB path is PostgREST (436 call sites). So bundling
  the real GoTrue + PostgREST + Postgres binaries (all permissively
  licensed, all arm64-mac-buildable) preserves every API contract with
  zero rewrites.
- No Redis/queues on main; document conversion degrades gracefully
  without LibreOffice; storage already has an adapter seam in flight
  (upstream PR #47 lineage) that a local-filesystem driver can plug into.
- LLM access already supports per-user keys and local Ollama; the keyless
  demo mode (PR open-legal-products#260) covers the first-run answer.

HOW THE PLAN IS STRUCTURED
Five phases, each independently shippable: (0) upstream seams as normal
web-app PRs — storage adapter + fs driver, migration runner, Next
standalone flag; (1) a local-stack supervisor in the shell with a "Run
locally on this Mac" mode on the connect screen; (2) packaging + signing
of the bundled binaries; (3) upgrade/backup lifecycle; (4) first-run
polish. Roughly 6-9 engineering weeks. Open product decisions (auth UX,
password recovery without SMTP, LibreOffice, Intel support) are listed
explicitly rather than silently decided.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant