feat: durable job queues for conversion, extraction and embedding (ADR) by amal66 · Pull Request #40 · amal66/mike

amal66 · 2026-07-17T07:59:51Z

Design (ADR)

Summarized from the fork's docs/adr/0003-default-synchronous-env-gated-async.md, docs/adr/0004-database-enforced-idempotency.md and docs/async-jobs.md.

Context. Two workloads are expensive and can outlive a single request: DOCX→PDF conversion and tabular cell extraction. Running them inline is simple and needs no infrastructure, but the work dies with the request (a client disconnect or server restart loses it) and can't retry. Making them async needs Redis and workers — infrastructure a self-hoster kicking the tires should not be forced to run.

Decision. Every async-capable path is synchronous by default and opt-in to async via an ASYNC_* env flag, each defaulting to "false":

Off → the work runs inline on the request thread (the historical behavior); no Redis required. The server only starts workers when anyWorkerEnabled() is true, so the default process has no Redis dependency at all.
On → the work is enqueued to a BullMQ queue and an in-process worker drains it, gaining durability across disconnect/restart and retry-with-backoff (attempts: 3, exponential). Bulkhead concurrency caps per worker (conversion 2, extraction 3) bound resource use.
The same core function serves both modes — extractDocumentColumns is called by the sync route and the async worker, differing only in missing-cell policy — so async is a deployment choice, not a rewrite.
Correctness is pushed into the queue's identity model: each queue derives its jobId deterministically from the work identity (versionId for conversion, (reviewId, documentId) for extraction), so a double submit (client retry, reconnect) collapses into the in-flight job instead of running twice. Durable state lives in DB rows (documents.status, tabular_cells), not queue history; extraction jobs re-read cell state and only process columns not already done, so retries never redo finished work.
The async /generate request becomes a reconnectable view: it subscribes to the review's Redis progress channel before enqueuing, forwards the same cell_update SSE frames the sync path emits, and a 3-second DB-poll backstop reconciles missed pub/sub frames so a dropped message can never leave the stream hung. GET /:reviewId/generate/stream lets a dropped client resume without re-triggering work.

Consequences. Trivial default onboarding (fresh clone runs everything inline); durability and retries where they matter; two code paths kept honest by sharing one extraction core; workers run in-process by default (split into a dedicated process by calling startWorkers() from a separate entrypoint when scaling apart).

Alternatives considered. Always-async (rejected: forces Redis on every self-hoster); app-level dedupe/locking for double submits (rejected: racy across replicas — the queue's jobId uniqueness and the DB are the only single points of serialization); a cron/poller instead of a queue (no retry semantics, no backpressure, higher latency).

Summary

Fewer stuck documents and fewer lost review runs. Today, a large DOCX upload blocks its HTTP request on LibreOffice, and a tabular review generation dies if the browser tab closes or the server restarts mid-run — the grid is left with spinners that never resolve. With the flags on, uploads return immediately and convert in the background with retries, and review extraction survives disconnects and restarts, retries transient LLM/storage failures with backoff, and lets a client reconnect to a running generation and catch up. Failures become explicit terminal states (document error, cell error) instead of hangs.

Changes

backend/src/lib/queue/ — shared lazy Redis connection (BullMQ-safe maxRetriesPerRequest: null), conversionQueue, extractionQueue (deterministic jobIds, retry/backoff, bounded history), and runProgress (Redis pub/sub bridge for live cell updates).
backend/src/workers/ — conversionWorker, extractionWorker, a declarative WORKER_REGISTRY, and startWorkers()/stopWorkers() lifecycle.
backend/src/lib/tabular/ — the extraction core factored out of routes/tabular.ts so the sync route and the async worker share one loop: tabular.extract.ts (LLM cell extraction + PDF/DOCX/Office text extraction), tabular.extractDoc.ts (extractDocumentColumns), tabular.generate.ts (prepareTabularGenerate guard), tabular.generateStream.ts (async enqueue + reconnectable SSE tail), tabular.shared.ts, tabular.prompt.ts. routes/tabular.ts now imports these instead of its inline copies (moved, not changed — bodies are byte-identical apart from import/type-annotation mechanics).
backend/src/routes/tabular.ts — POST /:reviewId/generate gains the async path behind ASYNC_TABULAR_EXTRACTION; new GET /:reviewId/generate/stream resume endpoint. Inline path remains the default.
backend/src/routes/documents.ts — upload defers Office→PDF conversion to the queue behind ASYNC_DOCUMENT_CONVERSION (doc stays processing until the worker flips it to ready).
backend/src/index.ts — start workers only when a flag is on; graceful SIGTERM/SIGINT shutdown (drain server, stop workers, close queues + Redis, 15s force-exit guard).
backend/src/lib/sseHeartbeat.ts, backend/src/lib/pdfjs.ts — small helpers the above use (SSE keepalive comments; typed facade over pdfjs-dist).
backend/.env.example, backend/tsconfig.json (exclude tests from tsc, as the fork does), backend/package.json + lockfile.
Tests: lib/queue/__tests__/{conversionQueue,extractionQueue}.test.ts, workers/__tests__/{conversionWorker,extractionWorker}.test.ts, lib/tabular/__tests__/{tabular.extractDoc,tabular.generateStream}.test.ts.

Why

Leads with the cost posture: both flags default "false", and with them off the server never dials Redis — behavior and infrastructure requirements are exactly what they are today. A cost-sensitive firm on a £20 VPS changes nothing and pays nothing extra. Turning a queue on needs only a Redis container on the same box (REDIS_URL, defaults to redis://localhost:6379).

New runtime deps (both shipped by the fork for this feature): bullmq ^5.34.0 (the queue) and ioredis 5.10.1 (its Redis client; the lockfile pins bullmq 5.79.2, matching the fork's lockfile, so a single deduped ioredis serves both). No other dependency, schema, or API change.

Note on the title: the fork's third queue (document embedding) is not in this PR — its queue/worker import the RAG ingest module, so that wiring rides in the follow-up RAG PR where the dependency is real.

Testing

npm install && npm run build (tsc) green on the branch as committed.
With the vitest harness merged locally (not part of this branch): npx vitest run → 7 test files, 39 tests, all passing (the 6 ported suites above plus the harness's existing downloadTokens suite).
Flags-off path: the queue modules are only imported by the gated call sites; nothing constructs a Redis connection unless a flag is "true" (connection is lazy and only reached via enqueue*/startWorkers).

Provenance

All added lines are mechanical ports of amal66/mike@origin/main (b3166dd) — path moves (apps/api/src/* → backend/src/*, modules/tabular/* → lib/tabular/*), import rewrites, and inlined types; exceptions:

The fork's zod-validated lib/env.ts module is not ported; env reads are inlined as process.env.X with the fork's defaults preserved (ASYNC_* default "false", REDIS_URL default redis://localhost:6379).
The fork's observability infrastructure is not ported: pino logger calls are translated to console.*, and OpenTelemetry trace propagation (withTraceContext, the otel?: OtelCarrier payload field, runWithRequestContext/withExtractedContext wrappers) is removed from queue payloads and worker callbacks. The Log type becomes Pick<Console, "error">.
tabular.prompt.ts ports only formatPromptSuffix (identical to the inline copy this PR deletes from routes/tabular.ts); tabular.generate.ts ports only prepareTabularGenerate; tabular.extract.ts omits the fork's extractTabularAnnotations and the redline-summary enrichment of extractDocxMarkdown — those depend on fork-only chat/tracked-changes code, so the ported body matches upstream's existing inline function exactly.
routes/tabular.ts deletions are the inline copies of the moved helpers (verified byte-identical modulo import/type annotations); the new /generate body is the fork's route body with req.log → console.
Test files: the fork's vi.mock("../../env", …) shims are dropped (no env module here) and mock import paths follow the path moves.
bullmq/ioredis version pins match the fork's package.json + lockfile resolution.

Credits & prior art

@nwhitehouse (nwhitehouse/mike) — independently parallels this work: their fork built a durable job worker pool (plus an agent event audit log and loop controller) for long-running work. Different implementation (this PR is BullMQ/Redis with env-gated sync fallback), same conclusion that request-lifetime execution loses work.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC

…xtraction Port of the fork's (amal66/mike) durable job-queue architecture: - lib/queue/: shared lazy Redis connection (BullMQ-safe options), the document-conversion and tabular-extraction queues with deterministic jobIds (dedupe on double submit), attempts:3 + exponential backoff, and the Redis pub/sub bridge (runProgress) that lets an SSE request tail a running extraction. - workers/: in-process BullMQ workers behind a declarative registry. Conversion mirrors the synchronous upload path's semantics (conversion failure is non-fatal; a permanently failed job flips the document to "error"). Extraction re-derives everything from the DB at run time (no secrets in Redis), is idempotent/retry-safe, and marks surviving cells "error" once retries are exhausted. - lib/tabular/: the extraction core factored out of routes/tabular.ts (extractDocumentColumns + LLM/text-extraction helpers + prepare guard) so the synchronous route and the async worker share one loop. - routes/tabular.ts: POST /:reviewId/generate gains the async path (enqueue + reconnectable tail) behind ASYNC_TABULAR_EXTRACTION, and a new GET /:reviewId/generate/stream resume endpoint. The inline path remains the default. - routes/documents.ts: upload defers DOCX->PDF conversion to the queue behind ASYNC_DOCUMENT_CONVERSION; default stays inline. - index.ts: workers start only when a flag is on; graceful shutdown closes the server, workers, queues and Redis. Both flags default "false": with no configuration nothing dials Redis and behavior is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CEguyEgXa9JjCciXCcVemC

…xtraction WHY THIS MATTERS Two workloads in Mike are expensive and can outlive the HTTP request that started them: DOCX -> PDF conversion (LibreOffice) and tabular-review cell extraction (one LLM call per row). Today both run inline on the request thread, so a closed laptop lid, a dropped connection, or a server restart mid-run silently loses the work — the review grid is left with spinners that never resolve, and a large upload blocks its request on LibreOffice. WHAT IS A DURABLE JOB QUEUE A job queue moves work out of the request/response cycle: the request records WHAT should happen (a small JSON payload in Redis) and returns; a worker process picks the job up, runs it, and retries it with exponential backoff if it fails. "Durable" means the job survives the death of the thing that created it — the queue (BullMQ on Redis) holds the job until a worker finishes it, no matter what happens to the original HTTP request or even the server process (BullMQ re-queues jobs whose worker crashed via its stalled-job detection). The classic hazard of queues is the DOUBLE SUBMIT: a client that reconnects and re-POSTs would enqueue the same work twice. This design pushes correctness into the queue's identity model — every job's id is derived deterministically from the work itself (`convert:<versionId>`, `extract:<reviewId>:<rowId>`), so BullMQ collapses a duplicate submit into the already-in-flight job. Durable STATE lives only in Postgres (documents.status, tabular_cells); jobs re-read that state when they run and skip columns already done, which is what makes retries idempotent. HOW IT WORKS - Both queues are OFF by default and opt-in per deployment: ASYNC_DOCUMENT_CONVERSION / ASYNC_TABULAR_EXTRACTION (default "false"). With the flags off the server never dials Redis — the queue connection is created lazily and only reached via enqueue/startWorkers, so a fresh clone still runs fully synchronously with zero new infrastructure. - lib/queue/: a shared lazy Redis connection (maxRetriesPerRequest: null, which BullMQ's blocking commands require), the two queues with deterministic jobIds + retry/backoff, and runProgress — a Redis pub/sub bridge that carries per-cell progress frames from workers to any HTTP request that is watching. - workers/: conversionWorker (DOCX->PDF off the request thread; conversion failure finalizes the document without a PDF rendition, matching the sync path) and extractionWorker (throws on incomplete extraction so BullMQ retries; after the last retry a permanent-failure handler flips surviving cells to "error" so the grid never shows an eternal spinner). A declarative registry + startWorkers()/stopWorkers() lifecycle, started from index.ts only when a flag is on, with graceful SIGTERM/SIGINT drain. - lib/tabular/: the extraction core factored out of routes/tabular.ts so the synchronous route and the async worker share ONE loop (extractRowColumns). The unit of work is the review ROW — one document, or a folder of source documents extracted together — matching the row model main adopted for folder-grouped reviews. tabular.rows.ts carries the row loaders (loadReviewRows / loadRowDocumentText) that both the routes and the worker need. - POST /:reviewId/generate keeps its exact synchronous behavior by default; with the flag on it enqueues one job per row, subscribes to the review's progress channel BEFORE enqueuing (so a fast worker cannot publish into the void), and forwards the same cell_update SSE frames the sync path emits. A 3-second DB-poll backstop reconciles any missed pub/sub frame, so a dropped message can never hang the stream. A new GET /:reviewId/generate/stream lets a disconnected client reattach to a running generation without re-triggering work. Ported from amal66/mike (upstream-pr/durable-queues, #40) and re-derived against current main: the extraction core is row-based (not document-based) to match the folder-grouped row model (open-legal-products#274) and db pagination (open-legal-products#263) that landed after the original branch, and the moved helper bodies match main's current copies byte-for-byte (multi-document citation prompts, Ollama key exemption). Tests: 510 passing (was 499), including queue jobId determinism, worker idempotency/retry/permanent-failure policy, row extraction core, and the pending-cell targeting used by the reconnectable stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ewrites clear their stale renditions WHY THIS MATTERS The first two commits put ASYNC_DOCUMENT_CONVERSION in front of the five LibreOffice call sites on the upload/version routes — but a whole-tree survey found two more sites hiding inside the chat tools, reached through the SSE stream itself: 1. replicate_document converts the source inline when it has no PDF rendition to copy (toolDispatcher.ts), and 2. generate_ppt converts the generated PPTX inline (documentOps.ts / persistGeneratedFile). With the flag on, a deployment reasonably believes "LibreOffice no longer runs on my request threads" — yet a chat turn that replicates or generates a deck still paid the 3-15s LibreOffice cold start in-band, and a conversion failure was swallowed (devLog) leaving the document permanently rendition-less with no retry path. Closing these keeps the flag's promise honest: it now covers every LibreOffice call site in the app. WHAT IS A RENDITION AND WHO CONSUMES IT A "rendition" is the per-version converted PDF (pdf_storage_path) that /single-documents/:id/display serves in place of raw Office bytes. The frontend routes DOCX to DocxView by FILENAME (docx-preview over raw bytes), so DOCX renditions are almost never displayed — but PPTX has no dedicated viewer and renders exclusively through its rendition via PdfView, which is why generate_ppt's silent conversion failure was a real product gap. HOW IT WORKS Both sites follow the exact pattern of the five route sites: - flag off → inline docxToPdf, byte-for-byte the historical behavior. - flag on → the document/copy is inserted "ready" with pdf_storage_path: null and one conversion job per new version is enqueued (deduped on convert:<versionId>, attempts: 3, exponential backoff). finalizeDocumentStatus: false because these documents are usable from their raw bytes — a rendition failure must never flip a healthy document to "error". - an enqueue failure degrades to the sync path's conversion-failure behavior (usable document, no rendition) instead of failing the tool. THE INVARIANT HALF (accept/reject + in-place re-edit) Accept/reject of a tracked change and an in-place assistant re-edit both rewrite a version's DOCX bytes at its existing storage path. Any rendition recorded for that version now describes bytes that no longer exist — and a stale rendition is not hypothetical decoration: /display would serve it, and replicate_document COPIES it onto every replica. Both rewrite sites now null pdf_storage_path in the same update that re-hashes the content, the same ordering discipline as the content_sha256 clear-then-set that surrounds the byte write. In today's flows assistant_edit versions never carry a rendition, so this is an invariant made explicit, not a behavior change. DELIBERATELY NOT DONE edit_document / generate_docx versions do NOT get renditions enqueued: DOCX renders through DocxView from raw bytes everywhere (the tracked- changes UI depends on it), so a rendition would cost a LibreOffice run per chat edit and display nothing. Recorded here so the omission reads as a decision, not an oversight. TESTS - replicateRenditionQueue.test.ts: flag off → one inline conversion, no queue; flag on → zero in-band LibreOffice, one job per copy with per-copy pdfKey and finalize:false; enqueue failure → copies still usable. - documentOps.generatedRendition.test.ts: same contract for generate_ppt, plus xlsx never converts or enqueues (spreadsheets are served raw). Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…rminal writes + cross-process job cancellation WHY THIS MATTERS POST /tabular-review/:id/clear-cells resets a row's cells to "pending" so a user can blank bad results and start over. But extraction runs concurrently with it — inline on another request (sync mode) or on a worker that survives disconnects (async mode, introduced by this PR). A run that finished AFTER the clear would write "done" (or "error") straight over the user's reset, and in async mode a queued-but-unstarted job would happily re-fill the freshly cleared row seconds later. Durability made the old race WIDER, so this PR owns the fix. WHAT IS A LOST UPDATE Two writers race the same row: the user's reset (status = pending) and the extraction's terminal write (status = done). Without a guard the last writer wins, and the extraction — which started before the reset and knows nothing of it — silently undoes the user's action. The classic cure is an optimistic condition: make the terminal write assert the state it believes it owns. HOW IT WORKS — three layers, weakest to strongest 1. GUARDED TERMINAL WRITES (both modes). Every terminal cell write now carries AND status = 'generating': the shared core's done-write, the sync route's missing→error writes, the sync regenerate-cell writes, and the worker's permanent-failure cleanup. An extraction only ever finalizes a claim it still holds; if clear-cells revoked the claim, the write matches zero rows and the SSE/Redis announce is skipped too — a tailing stream never shows a "done" the DB doesn't hold. The model's result still counts as "received", so the caller neither marks the cleared cell "error" nor retries over it. 2. QUEUE CANCELLATION (async mode, flag-gated so sync deployments never dial Redis). clear-cells addresses every deterministic jobId for the cleared rows — extract:<review>:<row> and each :<col> variant — and REMOVES jobs still waiting/delayed, so they never start. 3. PERSISTED CANCEL MARKER for jobs already active. BullMQ's Job#discard() is only an in-memory flag on the worker's own Job instance — calling it from the API process is a silent no-op (a live-Redis smoke caught exactly this: the "discarded" job's retry was scheduled anyway). Instead the job's data is marked canceled: true via updateData(), which IS persisted to Redis; each retry attempt re-fetches job data, and runExtractionJob now returns immediately on the marker instead of re-claiming the cleared cells from scratch. TRADEOFF, FLAGGED The permanent-failure handler now flips only cells still claimed ("generating") to "error". A job that dies before ever claiming its cells (e.g. the settings lookup fails on all 3 attempts) leaves them "pending" — a blank, re-runnable state rather than a red one; the resume stream then runs to its 15-minute cap instead of resolving early. Chosen deliberately: silently overwriting a user's reset is worse than a quieter failure under already-broken infrastructure. TESTS - tabular.extractRow: a done-write that matches zero rows (cell cleared mid-flight) suppresses the announce and is not reported "missing". - extractionWorker: canceled jobs touch nothing; permanent failure leaves cleared (pending) cells alone while still erroring claimed ones. - extractionQueue: waiting jobs removed; active jobs get the PERSISTED updateData marker (never Job#discard); remove() losing the race falls back to the marker; per-job failures are swallowed (best-effort by design — the write guards are the correctness layer). - Live-Redis smoke (real BullMQ): dedupe, remove-waiting, and cancel-active verified end-to-end — attempt 1 active during the cancel, attempt 2 saw canceled: true and completed without touching the DB. Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…xtraction WHY THIS MATTERS Two workloads in Mike are expensive and can outlive the HTTP request that started them: DOCX -> PDF conversion (LibreOffice) and tabular-review cell extraction (one LLM call per row). Today both run inline on the request thread, so a closed laptop lid, a dropped connection, or a server restart mid-run silently loses the work — the review grid is left with spinners that never resolve, and a large upload blocks its request on LibreOffice. WHAT IS A DURABLE JOB QUEUE A job queue moves work out of the request/response cycle: the request records WHAT should happen (a small JSON payload in Redis) and returns; a worker process picks the job up, runs it, and retries it with exponential backoff if it fails. "Durable" means the job survives the death of the thing that created it — the queue (BullMQ on Redis) holds the job until a worker finishes it, no matter what happens to the original HTTP request or even the server process (BullMQ re-queues jobs whose worker crashed via its stalled-job detection). The classic hazard of queues is the DOUBLE SUBMIT: a client that reconnects and re-POSTs would enqueue the same work twice. This design pushes correctness into the queue's identity model — every job's id is derived deterministically from the work itself (`convert:<versionId>`, `extract:<reviewId>:<rowId>`), so BullMQ collapses a duplicate submit into the already-in-flight job. Durable STATE lives only in Postgres (documents.status, tabular_cells); jobs re-read that state when they run and skip columns already done, which is what makes retries idempotent. HOW IT WORKS - Both queues are OFF by default and opt-in per deployment: ASYNC_DOCUMENT_CONVERSION / ASYNC_TABULAR_EXTRACTION (default "false"). With the flags off the server never dials Redis — the queue connection is created lazily and only reached via enqueue/startWorkers, so a fresh clone still runs fully synchronously with zero new infrastructure. - lib/queue/: a shared lazy Redis connection (maxRetriesPerRequest: null, which BullMQ's blocking commands require), the two queues with deterministic jobIds + retry/backoff, and runProgress — a Redis pub/sub bridge that carries per-cell progress frames from workers to any HTTP request that is watching. - workers/: conversionWorker (DOCX->PDF off the request thread; conversion failure finalizes the document without a PDF rendition, matching the sync path) and extractionWorker (throws on incomplete extraction so BullMQ retries; after the last retry a permanent-failure handler flips surviving cells to "error" so the grid never shows an eternal spinner). A declarative registry + startWorkers()/stopWorkers() lifecycle, started from index.ts only when a flag is on, with graceful SIGTERM/SIGINT drain. - lib/tabular/: the extraction core factored out of routes/tabular.ts so the synchronous route and the async worker share ONE loop (extractRowColumns). The unit of work is the review ROW — one document, or a folder of source documents extracted together — matching the row model main adopted for folder-grouped reviews. tabular.rows.ts carries the row loaders (loadReviewRows / loadRowDocumentText) that both the routes and the worker need. - POST /:reviewId/generate keeps its exact synchronous behavior by default; with the flag on it enqueues one job per row, subscribes to the review's progress channel BEFORE enqueuing (so a fast worker cannot publish into the void), and forwards the same cell_update SSE frames the sync path emits. A 3-second DB-poll backstop reconciles any missed pub/sub frame, so a dropped message can never hang the stream. A new GET /:reviewId/generate/stream lets a disconnected client reattach to a running generation without re-triggering work. Ported from amal66/mike (upstream-pr/durable-queues, #40) and re-derived against current main: the extraction core is row-based (not document-based) to match the folder-grouped row model (open-legal-products#274) and db pagination (open-legal-products#263) that landed after the original branch, and the moved helper bodies match main's current copies byte-for-byte (multi-document citation prompts, Ollama key exemption). Tests: 510 passing (was 499), including queue jobId determinism, worker idempotency/retry/permanent-failure policy, row extraction core, and the pending-cell targeting used by the reconnectable stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ewrites clear their stale renditions WHY THIS MATTERS The first two commits put ASYNC_DOCUMENT_CONVERSION in front of the five LibreOffice call sites on the upload/version routes — but a whole-tree survey found two more sites hiding inside the chat tools, reached through the SSE stream itself: 1. replicate_document converts the source inline when it has no PDF rendition to copy (toolDispatcher.ts), and 2. generate_ppt converts the generated PPTX inline (documentOps.ts / persistGeneratedFile). With the flag on, a deployment reasonably believes "LibreOffice no longer runs on my request threads" — yet a chat turn that replicates or generates a deck still paid the 3-15s LibreOffice cold start in-band, and a conversion failure was swallowed (devLog) leaving the document permanently rendition-less with no retry path. Closing these keeps the flag's promise honest: it now covers every LibreOffice call site in the app. WHAT IS A RENDITION AND WHO CONSUMES IT A "rendition" is the per-version converted PDF (pdf_storage_path) that /single-documents/:id/display serves in place of raw Office bytes. The frontend routes DOCX to DocxView by FILENAME (docx-preview over raw bytes), so DOCX renditions are almost never displayed — but PPTX has no dedicated viewer and renders exclusively through its rendition via PdfView, which is why generate_ppt's silent conversion failure was a real product gap. HOW IT WORKS Both sites follow the exact pattern of the five route sites: - flag off → inline docxToPdf, byte-for-byte the historical behavior. - flag on → the document/copy is inserted "ready" with pdf_storage_path: null and one conversion job per new version is enqueued (deduped on convert:<versionId>, attempts: 3, exponential backoff). finalizeDocumentStatus: false because these documents are usable from their raw bytes — a rendition failure must never flip a healthy document to "error". - an enqueue failure degrades to the sync path's conversion-failure behavior (usable document, no rendition) instead of failing the tool. THE INVARIANT HALF (accept/reject + in-place re-edit) Accept/reject of a tracked change and an in-place assistant re-edit both rewrite a version's DOCX bytes at its existing storage path. Any rendition recorded for that version now describes bytes that no longer exist — and a stale rendition is not hypothetical decoration: /display would serve it, and replicate_document COPIES it onto every replica. Both rewrite sites now null pdf_storage_path in the same update that re-hashes the content, the same ordering discipline as the content_sha256 clear-then-set that surrounds the byte write. In today's flows assistant_edit versions never carry a rendition, so this is an invariant made explicit, not a behavior change. DELIBERATELY NOT DONE edit_document / generate_docx versions do NOT get renditions enqueued: DOCX renders through DocxView from raw bytes everywhere (the tracked- changes UI depends on it), so a rendition would cost a LibreOffice run per chat edit and display nothing. Recorded here so the omission reads as a decision, not an oversight. TESTS - replicateRenditionQueue.test.ts: flag off → one inline conversion, no queue; flag on → zero in-band LibreOffice, one job per copy with per-copy pdfKey and finalize:false; enqueue failure → copies still usable. - documentOps.generatedRendition.test.ts: same contract for generate_ppt, plus xlsx never converts or enqueues (spreadsheets are served raw). Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…rminal writes + cross-process job cancellation WHY THIS MATTERS POST /tabular-review/:id/clear-cells resets a row's cells to "pending" so a user can blank bad results and start over. But extraction runs concurrently with it — inline on another request (sync mode) or on a worker that survives disconnects (async mode, introduced by this PR). A run that finished AFTER the clear would write "done" (or "error") straight over the user's reset, and in async mode a queued-but-unstarted job would happily re-fill the freshly cleared row seconds later. Durability made the old race WIDER, so this PR owns the fix. WHAT IS A LOST UPDATE Two writers race the same row: the user's reset (status = pending) and the extraction's terminal write (status = done). Without a guard the last writer wins, and the extraction — which started before the reset and knows nothing of it — silently undoes the user's action. The classic cure is an optimistic condition: make the terminal write assert the state it believes it owns. HOW IT WORKS — three layers, weakest to strongest 1. GUARDED TERMINAL WRITES (both modes). Every terminal cell write now carries AND status = 'generating': the shared core's done-write, the sync route's missing→error writes, the sync regenerate-cell writes, and the worker's permanent-failure cleanup. An extraction only ever finalizes a claim it still holds; if clear-cells revoked the claim, the write matches zero rows and the SSE/Redis announce is skipped too — a tailing stream never shows a "done" the DB doesn't hold. The model's result still counts as "received", so the caller neither marks the cleared cell "error" nor retries over it. 2. QUEUE CANCELLATION (async mode, flag-gated so sync deployments never dial Redis). clear-cells addresses every deterministic jobId for the cleared rows — extract:<review>:<row> and each :<col> variant — and REMOVES jobs still waiting/delayed, so they never start. 3. PERSISTED CANCEL MARKER for jobs already active. BullMQ's Job#discard() is only an in-memory flag on the worker's own Job instance — calling it from the API process is a silent no-op (a live-Redis smoke caught exactly this: the "discarded" job's retry was scheduled anyway). Instead the job's data is marked canceled: true via updateData(), which IS persisted to Redis; each retry attempt re-fetches job data, and runExtractionJob now returns immediately on the marker instead of re-claiming the cleared cells from scratch. TRADEOFF, FLAGGED The permanent-failure handler now flips only cells still claimed ("generating") to "error". A job that dies before ever claiming its cells (e.g. the settings lookup fails on all 3 attempts) leaves them "pending" — a blank, re-runnable state rather than a red one; the resume stream then runs to its 15-minute cap instead of resolving early. Chosen deliberately: silently overwriting a user's reset is worse than a quieter failure under already-broken infrastructure. TESTS - tabular.extractRow: a done-write that matches zero rows (cell cleared mid-flight) suppresses the announce and is not reported "missing". - extractionWorker: canceled jobs touch nothing; permanent failure leaves cleared (pending) cells alone while still erroring claimed ones. - extractionQueue: waiting jobs removed; active jobs get the PERSISTED updateData marker (never Job#discard); remove() losing the race falls back to the marker; per-job failures are swallowed (best-effort by design — the write guards are the correctness layer). - Live-Redis smoke (real BullMQ): dedupe, remove-waiting, and cancel-active verified end-to-end — attempt 1 active during the cancel, attempt 2 saw canceled: true and completed without touching the DB. Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…xtraction WHY THIS MATTERS Two workloads in Mike are expensive and can outlive the HTTP request that started them: DOCX -> PDF conversion (LibreOffice) and tabular-review cell extraction (one LLM call per row). Today both run inline on the request thread, so a closed laptop lid, a dropped connection, or a server restart mid-run silently loses the work — the review grid is left with spinners that never resolve, and a large upload blocks its request on LibreOffice. WHAT IS A DURABLE JOB QUEUE A job queue moves work out of the request/response cycle: the request records WHAT should happen (a small JSON payload in Redis) and returns; a worker process picks the job up, runs it, and retries it with exponential backoff if it fails. "Durable" means the job survives the death of the thing that created it — the queue (BullMQ on Redis) holds the job until a worker finishes it, no matter what happens to the original HTTP request or even the server process (BullMQ re-queues jobs whose worker crashed via its stalled-job detection). The classic hazard of queues is the DOUBLE SUBMIT: a client that reconnects and re-POSTs would enqueue the same work twice. This design pushes correctness into the queue's identity model — every job's id is derived deterministically from the work itself (`convert:<versionId>`, `extract:<reviewId>:<rowId>`), so BullMQ collapses a duplicate submit into the already-in-flight job. Durable STATE lives only in Postgres (documents.status, tabular_cells); jobs re-read that state when they run and skip columns already done, which is what makes retries idempotent. HOW IT WORKS - Both queues are OFF by default and opt-in per deployment: ASYNC_DOCUMENT_CONVERSION / ASYNC_TABULAR_EXTRACTION (default "false"). With the flags off the server never dials Redis — the queue connection is created lazily and only reached via enqueue/startWorkers, so a fresh clone still runs fully synchronously with zero new infrastructure. - lib/queue/: a shared lazy Redis connection (maxRetriesPerRequest: null, which BullMQ's blocking commands require), the two queues with deterministic jobIds + retry/backoff, and runProgress — a Redis pub/sub bridge that carries per-cell progress frames from workers to any HTTP request that is watching. - workers/: conversionWorker (DOCX->PDF off the request thread; conversion failure finalizes the document without a PDF rendition, matching the sync path) and extractionWorker (throws on incomplete extraction so BullMQ retries; after the last retry a permanent-failure handler flips surviving cells to "error" so the grid never shows an eternal spinner). A declarative registry + startWorkers()/stopWorkers() lifecycle, started from index.ts only when a flag is on, with graceful SIGTERM/SIGINT drain. - lib/tabular/: the extraction core factored out of routes/tabular.ts so the synchronous route and the async worker share ONE loop (extractRowColumns). The unit of work is the review ROW — one document, or a folder of source documents extracted together — matching the row model main adopted for folder-grouped reviews. tabular.rows.ts carries the row loaders (loadReviewRows / loadRowDocumentText) that both the routes and the worker need. - POST /:reviewId/generate keeps its exact synchronous behavior by default; with the flag on it enqueues one job per row, subscribes to the review's progress channel BEFORE enqueuing (so a fast worker cannot publish into the void), and forwards the same cell_update SSE frames the sync path emits. A 3-second DB-poll backstop reconciles any missed pub/sub frame, so a dropped message can never hang the stream. A new GET /:reviewId/generate/stream lets a disconnected client reattach to a running generation without re-triggering work. Ported from amal66/mike (upstream-pr/durable-queues, #40) and re-derived against current main: the extraction core is row-based (not document-based) to match the folder-grouped row model (open-legal-products#274) and db pagination (open-legal-products#263) that landed after the original branch, and the moved helper bodies match main's current copies byte-for-byte (multi-document citation prompts, Ollama key exemption). Tests: 510 passing (was 499), including queue jobId determinism, worker idempotency/retry/permanent-failure policy, row extraction core, and the pending-cell targeting used by the reconnectable stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ewrites clear their stale renditions WHY THIS MATTERS The first two commits put ASYNC_DOCUMENT_CONVERSION in front of the five LibreOffice call sites on the upload/version routes — but a whole-tree survey found two more sites hiding inside the chat tools, reached through the SSE stream itself: 1. replicate_document converts the source inline when it has no PDF rendition to copy (toolDispatcher.ts), and 2. generate_ppt converts the generated PPTX inline (documentOps.ts / persistGeneratedFile). With the flag on, a deployment reasonably believes "LibreOffice no longer runs on my request threads" — yet a chat turn that replicates or generates a deck still paid the 3-15s LibreOffice cold start in-band, and a conversion failure was swallowed (devLog) leaving the document permanently rendition-less with no retry path. Closing these keeps the flag's promise honest: it now covers every LibreOffice call site in the app. WHAT IS A RENDITION AND WHO CONSUMES IT A "rendition" is the per-version converted PDF (pdf_storage_path) that /single-documents/:id/display serves in place of raw Office bytes. The frontend routes DOCX to DocxView by FILENAME (docx-preview over raw bytes), so DOCX renditions are almost never displayed — but PPTX has no dedicated viewer and renders exclusively through its rendition via PdfView, which is why generate_ppt's silent conversion failure was a real product gap. HOW IT WORKS Both sites follow the exact pattern of the five route sites: - flag off → inline docxToPdf, byte-for-byte the historical behavior. - flag on → the document/copy is inserted "ready" with pdf_storage_path: null and one conversion job per new version is enqueued (deduped on convert:<versionId>, attempts: 3, exponential backoff). finalizeDocumentStatus: false because these documents are usable from their raw bytes — a rendition failure must never flip a healthy document to "error". - an enqueue failure degrades to the sync path's conversion-failure behavior (usable document, no rendition) instead of failing the tool. THE INVARIANT HALF (accept/reject + in-place re-edit) Accept/reject of a tracked change and an in-place assistant re-edit both rewrite a version's DOCX bytes at its existing storage path. Any rendition recorded for that version now describes bytes that no longer exist — and a stale rendition is not hypothetical decoration: /display would serve it, and replicate_document COPIES it onto every replica. Both rewrite sites now null pdf_storage_path in the same update that re-hashes the content, the same ordering discipline as the content_sha256 clear-then-set that surrounds the byte write. In today's flows assistant_edit versions never carry a rendition, so this is an invariant made explicit, not a behavior change. DELIBERATELY NOT DONE edit_document / generate_docx versions do NOT get renditions enqueued: DOCX renders through DocxView from raw bytes everywhere (the tracked- changes UI depends on it), so a rendition would cost a LibreOffice run per chat edit and display nothing. Recorded here so the omission reads as a decision, not an oversight. TESTS - replicateRenditionQueue.test.ts: flag off → one inline conversion, no queue; flag on → zero in-band LibreOffice, one job per copy with per-copy pdfKey and finalize:false; enqueue failure → copies still usable. - documentOps.generatedRendition.test.ts: same contract for generate_ppt, plus xlsx never converts or enqueues (spreadsheets are served raw). Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…rminal writes + cross-process job cancellation WHY THIS MATTERS POST /tabular-review/:id/clear-cells resets a row's cells to "pending" so a user can blank bad results and start over. But extraction runs concurrently with it — inline on another request (sync mode) or on a worker that survives disconnects (async mode, introduced by this PR). A run that finished AFTER the clear would write "done" (or "error") straight over the user's reset, and in async mode a queued-but-unstarted job would happily re-fill the freshly cleared row seconds later. Durability made the old race WIDER, so this PR owns the fix. WHAT IS A LOST UPDATE Two writers race the same row: the user's reset (status = pending) and the extraction's terminal write (status = done). Without a guard the last writer wins, and the extraction — which started before the reset and knows nothing of it — silently undoes the user's action. The classic cure is an optimistic condition: make the terminal write assert the state it believes it owns. HOW IT WORKS — three layers, weakest to strongest 1. GUARDED TERMINAL WRITES (both modes). Every terminal cell write now carries AND status = 'generating': the shared core's done-write, the sync route's missing→error writes, the sync regenerate-cell writes, and the worker's permanent-failure cleanup. An extraction only ever finalizes a claim it still holds; if clear-cells revoked the claim, the write matches zero rows and the SSE/Redis announce is skipped too — a tailing stream never shows a "done" the DB doesn't hold. The model's result still counts as "received", so the caller neither marks the cleared cell "error" nor retries over it. 2. QUEUE CANCELLATION (async mode, flag-gated so sync deployments never dial Redis). clear-cells addresses every deterministic jobId for the cleared rows — extract:<review>:<row> and each :<col> variant — and REMOVES jobs still waiting/delayed, so they never start. 3. PERSISTED CANCEL MARKER for jobs already active. BullMQ's Job#discard() is only an in-memory flag on the worker's own Job instance — calling it from the API process is a silent no-op (a live-Redis smoke caught exactly this: the "discarded" job's retry was scheduled anyway). Instead the job's data is marked canceled: true via updateData(), which IS persisted to Redis; each retry attempt re-fetches job data, and runExtractionJob now returns immediately on the marker instead of re-claiming the cleared cells from scratch. TRADEOFF, FLAGGED The permanent-failure handler now flips only cells still claimed ("generating") to "error". A job that dies before ever claiming its cells (e.g. the settings lookup fails on all 3 attempts) leaves them "pending" — a blank, re-runnable state rather than a red one; the resume stream then runs to its 15-minute cap instead of resolving early. Chosen deliberately: silently overwriting a user's reset is worse than a quieter failure under already-broken infrastructure. TESTS - tabular.extractRow: a done-write that matches zero rows (cell cleared mid-flight) suppresses the announce and is not reported "missing". - extractionWorker: canceled jobs touch nothing; permanent failure leaves cleared (pending) cells alone while still erroring claimed ones. - extractionQueue: waiting jobs removed; active jobs get the PERSISTED updateData marker (never Job#discard); remove() losing the race falls back to the marker; per-job failures are swallowed (best-effort by design — the write guards are the correctness layer). - Live-Redis smoke (real BullMQ): dedupe, remove-waiting, and cancel-active verified end-to-end — attempt 1 active during the cancel, attempt 2 saw canceled: true and completed without touching the DB. Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…xtraction WHY THIS MATTERS Two workloads in Mike are expensive and can outlive the HTTP request that started them: DOCX -> PDF conversion (LibreOffice) and tabular-review cell extraction (one LLM call per row). Today both run inline on the request thread, so a closed laptop lid, a dropped connection, or a server restart mid-run silently loses the work — the review grid is left with spinners that never resolve, and a large upload blocks its request on LibreOffice. WHAT IS A DURABLE JOB QUEUE A job queue moves work out of the request/response cycle: the request records WHAT should happen (a small JSON payload in Redis) and returns; a worker process picks the job up, runs it, and retries it with exponential backoff if it fails. "Durable" means the job survives the death of the thing that created it — the queue (BullMQ on Redis) holds the job until a worker finishes it, no matter what happens to the original HTTP request or even the server process (BullMQ re-queues jobs whose worker crashed via its stalled-job detection). The classic hazard of queues is the DOUBLE SUBMIT: a client that reconnects and re-POSTs would enqueue the same work twice. This design pushes correctness into the queue's identity model — every job's id is derived deterministically from the work itself (`convert:<versionId>`, `extract:<reviewId>:<rowId>`), so BullMQ collapses a duplicate submit into the already-in-flight job. Durable STATE lives only in Postgres (documents.status, tabular_cells); jobs re-read that state when they run and skip columns already done, which is what makes retries idempotent. HOW IT WORKS - Both queues are OFF by default and opt-in per deployment: ASYNC_DOCUMENT_CONVERSION / ASYNC_TABULAR_EXTRACTION (default "false"). With the flags off the server never dials Redis — the queue connection is created lazily and only reached via enqueue/startWorkers, so a fresh clone still runs fully synchronously with zero new infrastructure. - lib/queue/: a shared lazy Redis connection (maxRetriesPerRequest: null, which BullMQ's blocking commands require), the two queues with deterministic jobIds + retry/backoff, and runProgress — a Redis pub/sub bridge that carries per-cell progress frames from workers to any HTTP request that is watching. - workers/: conversionWorker (DOCX->PDF off the request thread; conversion failure finalizes the document without a PDF rendition, matching the sync path) and extractionWorker (throws on incomplete extraction so BullMQ retries; after the last retry a permanent-failure handler flips surviving cells to "error" so the grid never shows an eternal spinner). A declarative registry + startWorkers()/stopWorkers() lifecycle, started from index.ts only when a flag is on, with graceful SIGTERM/SIGINT drain. - lib/tabular/: the extraction core factored out of routes/tabular.ts so the synchronous route and the async worker share ONE loop (extractRowColumns). The unit of work is the review ROW — one document, or a folder of source documents extracted together — matching the row model main adopted for folder-grouped reviews. tabular.rows.ts carries the row loaders (loadReviewRows / loadRowDocumentText) that both the routes and the worker need. - POST /:reviewId/generate keeps its exact synchronous behavior by default; with the flag on it enqueues one job per row, subscribes to the review's progress channel BEFORE enqueuing (so a fast worker cannot publish into the void), and forwards the same cell_update SSE frames the sync path emits. A 3-second DB-poll backstop reconciles any missed pub/sub frame, so a dropped message can never hang the stream. A new GET /:reviewId/generate/stream lets a disconnected client reattach to a running generation without re-triggering work. Ported from amal66/mike (upstream-pr/durable-queues, amal66#40) and re-derived against current main: the extraction core is row-based (not document-based) to match the folder-grouped row model (#274) and db pagination (#263) that landed after the original branch, and the moved helper bodies match main's current copies byte-for-byte (multi-document citation prompts, Ollama key exemption). Tests: 510 passing (was 499), including queue jobId determinism, worker idempotency/retry/permanent-failure policy, row extraction core, and the pending-cell targeting used by the reconnectable stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ewrites clear their stale renditions WHY THIS MATTERS The first two commits put ASYNC_DOCUMENT_CONVERSION in front of the five LibreOffice call sites on the upload/version routes — but a whole-tree survey found two more sites hiding inside the chat tools, reached through the SSE stream itself: 1. replicate_document converts the source inline when it has no PDF rendition to copy (toolDispatcher.ts), and 2. generate_ppt converts the generated PPTX inline (documentOps.ts / persistGeneratedFile). With the flag on, a deployment reasonably believes "LibreOffice no longer runs on my request threads" — yet a chat turn that replicates or generates a deck still paid the 3-15s LibreOffice cold start in-band, and a conversion failure was swallowed (devLog) leaving the document permanently rendition-less with no retry path. Closing these keeps the flag's promise honest: it now covers every LibreOffice call site in the app. WHAT IS A RENDITION AND WHO CONSUMES IT A "rendition" is the per-version converted PDF (pdf_storage_path) that /single-documents/:id/display serves in place of raw Office bytes. The frontend routes DOCX to DocxView by FILENAME (docx-preview over raw bytes), so DOCX renditions are almost never displayed — but PPTX has no dedicated viewer and renders exclusively through its rendition via PdfView, which is why generate_ppt's silent conversion failure was a real product gap. HOW IT WORKS Both sites follow the exact pattern of the five route sites: - flag off → inline docxToPdf, byte-for-byte the historical behavior. - flag on → the document/copy is inserted "ready" with pdf_storage_path: null and one conversion job per new version is enqueued (deduped on convert:<versionId>, attempts: 3, exponential backoff). finalizeDocumentStatus: false because these documents are usable from their raw bytes — a rendition failure must never flip a healthy document to "error". - an enqueue failure degrades to the sync path's conversion-failure behavior (usable document, no rendition) instead of failing the tool. THE INVARIANT HALF (accept/reject + in-place re-edit) Accept/reject of a tracked change and an in-place assistant re-edit both rewrite a version's DOCX bytes at its existing storage path. Any rendition recorded for that version now describes bytes that no longer exist — and a stale rendition is not hypothetical decoration: /display would serve it, and replicate_document COPIES it onto every replica. Both rewrite sites now null pdf_storage_path in the same update that re-hashes the content, the same ordering discipline as the content_sha256 clear-then-set that surrounds the byte write. In today's flows assistant_edit versions never carry a rendition, so this is an invariant made explicit, not a behavior change. DELIBERATELY NOT DONE edit_document / generate_docx versions do NOT get renditions enqueued: DOCX renders through DocxView from raw bytes everywhere (the tracked- changes UI depends on it), so a rendition would cost a LibreOffice run per chat edit and display nothing. Recorded here so the omission reads as a decision, not an oversight. TESTS - replicateRenditionQueue.test.ts: flag off → one inline conversion, no queue; flag on → zero in-band LibreOffice, one job per copy with per-copy pdfKey and finalize:false; enqueue failure → copies still usable. - documentOps.generatedRendition.test.ts: same contract for generate_ppt, plus xlsx never converts or enqueues (spreadsheets are served raw). Part of the durable-queues row (amal66#40 → olp PR #294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…rminal writes + cross-process job cancellation WHY THIS MATTERS POST /tabular-review/:id/clear-cells resets a row's cells to "pending" so a user can blank bad results and start over. But extraction runs concurrently with it — inline on another request (sync mode) or on a worker that survives disconnects (async mode, introduced by this PR). A run that finished AFTER the clear would write "done" (or "error") straight over the user's reset, and in async mode a queued-but-unstarted job would happily re-fill the freshly cleared row seconds later. Durability made the old race WIDER, so this PR owns the fix. WHAT IS A LOST UPDATE Two writers race the same row: the user's reset (status = pending) and the extraction's terminal write (status = done). Without a guard the last writer wins, and the extraction — which started before the reset and knows nothing of it — silently undoes the user's action. The classic cure is an optimistic condition: make the terminal write assert the state it believes it owns. HOW IT WORKS — three layers, weakest to strongest 1. GUARDED TERMINAL WRITES (both modes). Every terminal cell write now carries AND status = 'generating': the shared core's done-write, the sync route's missing→error writes, the sync regenerate-cell writes, and the worker's permanent-failure cleanup. An extraction only ever finalizes a claim it still holds; if clear-cells revoked the claim, the write matches zero rows and the SSE/Redis announce is skipped too — a tailing stream never shows a "done" the DB doesn't hold. The model's result still counts as "received", so the caller neither marks the cleared cell "error" nor retries over it. 2. QUEUE CANCELLATION (async mode, flag-gated so sync deployments never dial Redis). clear-cells addresses every deterministic jobId for the cleared rows — extract:<review>:<row> and each :<col> variant — and REMOVES jobs still waiting/delayed, so they never start. 3. PERSISTED CANCEL MARKER for jobs already active. BullMQ's Job#discard() is only an in-memory flag on the worker's own Job instance — calling it from the API process is a silent no-op (a live-Redis smoke caught exactly this: the "discarded" job's retry was scheduled anyway). Instead the job's data is marked canceled: true via updateData(), which IS persisted to Redis; each retry attempt re-fetches job data, and runExtractionJob now returns immediately on the marker instead of re-claiming the cleared cells from scratch. TRADEOFF, FLAGGED The permanent-failure handler now flips only cells still claimed ("generating") to "error". A job that dies before ever claiming its cells (e.g. the settings lookup fails on all 3 attempts) leaves them "pending" — a blank, re-runnable state rather than a red one; the resume stream then runs to its 15-minute cap instead of resolving early. Chosen deliberately: silently overwriting a user's reset is worse than a quieter failure under already-broken infrastructure. TESTS - tabular.extractRow: a done-write that matches zero rows (cell cleared mid-flight) suppresses the announce and is not reported "missing". - extractionWorker: canceled jobs touch nothing; permanent failure leaves cleared (pending) cells alone while still erroring claimed ones. - extractionQueue: waiting jobs removed; active jobs get the PERSISTED updateData marker (never Job#discard); remove() losing the race falls back to the marker; per-job failures are swallowed (best-effort by design — the write guards are the correctness layer). - Live-Redis smoke (real BullMQ): dedupe, remove-waiting, and cancel-active verified end-to-end — attempt 1 active during the cancel, attempt 2 saw canceled: true and completed without touching the DB. Part of the durable-queues row (amal66#40 → olp PR #294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…xtraction WHY THIS MATTERS Two workloads in Mike are expensive and can outlive the HTTP request that started them: DOCX -> PDF conversion (LibreOffice) and tabular-review cell extraction (one LLM call per row). Today both run inline on the request thread, so a closed laptop lid, a dropped connection, or a server restart mid-run silently loses the work — the review grid is left with spinners that never resolve, and a large upload blocks its request on LibreOffice. WHAT IS A DURABLE JOB QUEUE A job queue moves work out of the request/response cycle: the request records WHAT should happen (a small JSON payload in Redis) and returns; a worker process picks the job up, runs it, and retries it with exponential backoff if it fails. "Durable" means the job survives the death of the thing that created it — the queue (BullMQ on Redis) holds the job until a worker finishes it, no matter what happens to the original HTTP request or even the server process (BullMQ re-queues jobs whose worker crashed via its stalled-job detection). The classic hazard of queues is the DOUBLE SUBMIT: a client that reconnects and re-POSTs would enqueue the same work twice. This design pushes correctness into the queue's identity model — every job's id is derived deterministically from the work itself (`convert:<versionId>`, `extract:<reviewId>:<rowId>`), so BullMQ collapses a duplicate submit into the already-in-flight job. Durable STATE lives only in Postgres (documents.status, tabular_cells); jobs re-read that state when they run and skip columns already done, which is what makes retries idempotent. HOW IT WORKS - Both queues are OFF by default and opt-in per deployment: ASYNC_DOCUMENT_CONVERSION / ASYNC_TABULAR_EXTRACTION (default "false"). With the flags off the server never dials Redis — the queue connection is created lazily and only reached via enqueue/startWorkers, so a fresh clone still runs fully synchronously with zero new infrastructure. - lib/queue/: a shared lazy Redis connection (maxRetriesPerRequest: null, which BullMQ's blocking commands require), the two queues with deterministic jobIds + retry/backoff, and runProgress — a Redis pub/sub bridge that carries per-cell progress frames from workers to any HTTP request that is watching. - workers/: conversionWorker (DOCX->PDF off the request thread; conversion failure finalizes the document without a PDF rendition, matching the sync path) and extractionWorker (throws on incomplete extraction so BullMQ retries; after the last retry a permanent-failure handler flips surviving cells to "error" so the grid never shows an eternal spinner). A declarative registry + startWorkers()/stopWorkers() lifecycle, started from index.ts only when a flag is on, with graceful SIGTERM/SIGINT drain. - lib/tabular/: the extraction core factored out of routes/tabular.ts so the synchronous route and the async worker share ONE loop (extractRowColumns). The unit of work is the review ROW — one document, or a folder of source documents extracted together — matching the row model main adopted for folder-grouped reviews. tabular.rows.ts carries the row loaders (loadReviewRows / loadRowDocumentText) that both the routes and the worker need. - POST /:reviewId/generate keeps its exact synchronous behavior by default; with the flag on it enqueues one job per row, subscribes to the review's progress channel BEFORE enqueuing (so a fast worker cannot publish into the void), and forwards the same cell_update SSE frames the sync path emits. A 3-second DB-poll backstop reconciles any missed pub/sub frame, so a dropped message can never hang the stream. A new GET /:reviewId/generate/stream lets a disconnected client reattach to a running generation without re-triggering work. Ported from amal66/mike (upstream-pr/durable-queues, #40) and re-derived against current main: the extraction core is row-based (not document-based) to match the folder-grouped row model (open-legal-products#274) and db pagination (open-legal-products#263) that landed after the original branch, and the moved helper bodies match main's current copies byte-for-byte (multi-document citation prompts, Ollama key exemption). Tests: 510 passing (was 499), including queue jobId determinism, worker idempotency/retry/permanent-failure policy, row extraction core, and the pending-cell targeting used by the reconnectable stream. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ewrites clear their stale renditions WHY THIS MATTERS The first two commits put ASYNC_DOCUMENT_CONVERSION in front of the five LibreOffice call sites on the upload/version routes — but a whole-tree survey found two more sites hiding inside the chat tools, reached through the SSE stream itself: 1. replicate_document converts the source inline when it has no PDF rendition to copy (toolDispatcher.ts), and 2. generate_ppt converts the generated PPTX inline (documentOps.ts / persistGeneratedFile). With the flag on, a deployment reasonably believes "LibreOffice no longer runs on my request threads" — yet a chat turn that replicates or generates a deck still paid the 3-15s LibreOffice cold start in-band, and a conversion failure was swallowed (devLog) leaving the document permanently rendition-less with no retry path. Closing these keeps the flag's promise honest: it now covers every LibreOffice call site in the app. WHAT IS A RENDITION AND WHO CONSUMES IT A "rendition" is the per-version converted PDF (pdf_storage_path) that /single-documents/:id/display serves in place of raw Office bytes. The frontend routes DOCX to DocxView by FILENAME (docx-preview over raw bytes), so DOCX renditions are almost never displayed — but PPTX has no dedicated viewer and renders exclusively through its rendition via PdfView, which is why generate_ppt's silent conversion failure was a real product gap. HOW IT WORKS Both sites follow the exact pattern of the five route sites: - flag off → inline docxToPdf, byte-for-byte the historical behavior. - flag on → the document/copy is inserted "ready" with pdf_storage_path: null and one conversion job per new version is enqueued (deduped on convert:<versionId>, attempts: 3, exponential backoff). finalizeDocumentStatus: false because these documents are usable from their raw bytes — a rendition failure must never flip a healthy document to "error". - an enqueue failure degrades to the sync path's conversion-failure behavior (usable document, no rendition) instead of failing the tool. THE INVARIANT HALF (accept/reject + in-place re-edit) Accept/reject of a tracked change and an in-place assistant re-edit both rewrite a version's DOCX bytes at its existing storage path. Any rendition recorded for that version now describes bytes that no longer exist — and a stale rendition is not hypothetical decoration: /display would serve it, and replicate_document COPIES it onto every replica. Both rewrite sites now null pdf_storage_path in the same update that re-hashes the content, the same ordering discipline as the content_sha256 clear-then-set that surrounds the byte write. In today's flows assistant_edit versions never carry a rendition, so this is an invariant made explicit, not a behavior change. DELIBERATELY NOT DONE edit_document / generate_docx versions do NOT get renditions enqueued: DOCX renders through DocxView from raw bytes everywhere (the tracked- changes UI depends on it), so a rendition would cost a LibreOffice run per chat edit and display nothing. Recorded here so the omission reads as a decision, not an oversight. TESTS - replicateRenditionQueue.test.ts: flag off → one inline conversion, no queue; flag on → zero in-band LibreOffice, one job per copy with per-copy pdfKey and finalize:false; enqueue failure → copies still usable. - documentOps.generatedRendition.test.ts: same contract for generate_ppt, plus xlsx never converts or enqueues (spreadsheets are served raw). Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

…rminal writes + cross-process job cancellation WHY THIS MATTERS POST /tabular-review/:id/clear-cells resets a row's cells to "pending" so a user can blank bad results and start over. But extraction runs concurrently with it — inline on another request (sync mode) or on a worker that survives disconnects (async mode, introduced by this PR). A run that finished AFTER the clear would write "done" (or "error") straight over the user's reset, and in async mode a queued-but-unstarted job would happily re-fill the freshly cleared row seconds later. Durability made the old race WIDER, so this PR owns the fix. WHAT IS A LOST UPDATE Two writers race the same row: the user's reset (status = pending) and the extraction's terminal write (status = done). Without a guard the last writer wins, and the extraction — which started before the reset and knows nothing of it — silently undoes the user's action. The classic cure is an optimistic condition: make the terminal write assert the state it believes it owns. HOW IT WORKS — three layers, weakest to strongest 1. GUARDED TERMINAL WRITES (both modes). Every terminal cell write now carries AND status = 'generating': the shared core's done-write, the sync route's missing→error writes, the sync regenerate-cell writes, and the worker's permanent-failure cleanup. An extraction only ever finalizes a claim it still holds; if clear-cells revoked the claim, the write matches zero rows and the SSE/Redis announce is skipped too — a tailing stream never shows a "done" the DB doesn't hold. The model's result still counts as "received", so the caller neither marks the cleared cell "error" nor retries over it. 2. QUEUE CANCELLATION (async mode, flag-gated so sync deployments never dial Redis). clear-cells addresses every deterministic jobId for the cleared rows — extract:<review>:<row> and each :<col> variant — and REMOVES jobs still waiting/delayed, so they never start. 3. PERSISTED CANCEL MARKER for jobs already active. BullMQ's Job#discard() is only an in-memory flag on the worker's own Job instance — calling it from the API process is a silent no-op (a live-Redis smoke caught exactly this: the "discarded" job's retry was scheduled anyway). Instead the job's data is marked canceled: true via updateData(), which IS persisted to Redis; each retry attempt re-fetches job data, and runExtractionJob now returns immediately on the marker instead of re-claiming the cleared cells from scratch. TRADEOFF, FLAGGED The permanent-failure handler now flips only cells still claimed ("generating") to "error". A job that dies before ever claiming its cells (e.g. the settings lookup fails on all 3 attempts) leaves them "pending" — a blank, re-runnable state rather than a red one; the resume stream then runs to its 15-minute cap instead of resolving early. Chosen deliberately: silently overwriting a user's reset is worse than a quieter failure under already-broken infrastructure. TESTS - tabular.extractRow: a done-write that matches zero rows (cell cleared mid-flight) suppresses the announce and is not reported "missing". - extractionWorker: canceled jobs touch nothing; permanent failure leaves cleared (pending) cells alone while still erroring claimed ones. - extractionQueue: waiting jobs removed; active jobs get the PERSISTED updateData marker (never Job#discard); remove() losing the race falls back to the marker; per-job failures are swallowed (best-effort by design — the write guards are the correctness layer). - Live-Redis smoke (real BullMQ): dedupe, remove-waiting, and cancel-active verified end-to-end — attempt 1 active during the cancel, attempt 2 saw canceled: true and completed without touching the DB. Part of the durable-queues row (#40 → olp PR open-legal-products#294). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JDev46QdqBsdyRCRBxcvk2

amal66 mentioned this pull request Jul 17, 2026

Reference index: the amal66 fork, decomposed into 25 reviewable PRs open-legal-products/mike#205

Closed

amal66 mentioned this pull request Aug 6, 2026

[Architecture] feat: durable job queues — BullMQ-primary with Postgres fallback, worker threads, async-on for new installs open-legal-products/mike#294

Open

Navigation Menu

feat: durable job queues for conversion, extraction and embedding (ADR) - #40

feat: durable job queues for conversion, extraction and embedding (ADR)#40
amal66 wants to merge 1 commit into
upstream-mainfrom
upstream-pr/durable-queues

amal66 commented Jul 17, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

amal66 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design (ADR)

Summary

Changes

Why

Testing

Provenance

Credits & prior art

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

amal66 commented Jul 17, 2026 •

edited

Loading