ADFA-4894 feat(dashboard): download control surface — paused phase + pause/resume/retry

The engine half of the download-control contract (ADFA-4893), on top of the resilient runners
already shipped in 1.2.3. The app-facing buttons land separately under ADFA-4896.

- jobs.ts: a 'paused' job phase + pause(id) / resume(id) / retry(id) on the durable engine, and a
  shared classifyStop(ctx) so every runner treats pause vs cancel the same way. Pause stops the
  runner but KEEPS the partial (the PAUSED-vs-CANCELLED distinction from the APK ADFA-5119); resume
  re-launches (kiwix continues via --continue, books per item); retry re-runs an errored job.
  resume/retry move the row to an ACTIVE phase before relaunch so completion/UI stay consistent.
- kiwix.exec / books.exec: on stop, keep the partial on pause and clean only on cancel.
- routes.ts: POST /:type/jobs/:id/{pause,resume,retry}, returning 409 when the verb doesn't apply.
- jobs.test.ts (npm run test:db — needs the native better-sqlite3 build): pause->paused,
  resume->done, retry-from-error, and the out-of-phase no-ops.

Version 1.2.4. tsc --noEmit clean; default suite 79/79. Device-verified: pause keeps the partial
and resume continues via --continue from a non-zero %, the verbs return 409 out of phase, a paused
job survives a dash-node restart, and test:db passes 3/3 in the box.

Author: luisguzman-adfa

static/dashboard/CHANGELOG.md

@@ -4,6 +4,7 @@ One line per version, newest first. Every REST-facing change bumps the version i
 (the app surfaces it via `/system/dashboard/update-check` and the "Update available" pill), so this
 file is the human record of what each bump enables. Keep entries short: `version - change (TICKET)`.
 
+- **1.2.4** - Download control surface (ADFA-4894, the engine half; app buttons land under ADFA-4896). Adds a **`paused`** job phase and `pause(id)` / `resume(id)` / `retry(id)` to the durable engine, exposed as `POST /:type/jobs/:id/{pause,resume,retry}`. Pause stops the runner but KEEPS the partial (resume continues via aria2 `--continue` for kiwix / per-item for books — the PAUSED-vs-CANCELLED distinction mirrors the APK ADFA-5119); retry re-runs an errored job. maps resume still restarts layers (no per-layer checkpoint yet → ADFA-4898). (ADFA-4894)
 - **1.2.3-dev** - Resilient content downloads (ADFA-4894, Area 1 of the ADFA-4893 contract). A network drop mid-job recovers or fails cleanly instead of failing the whole batch at the first blip: a shared `withRetry` backoff helper; job cancel now aborts an in-flight `fetch` (new `AbortSignal` on the runner), not just child processes; **books** get a per-attempt timeout + per-item retry/backoff and no longer fail a batch on one dropped EPUB; **kiwix** prunes the partial `.zim` + `.aria2` on cancel (clean-on-cancel) and gains aria2 reconnect flags (`--max-tries`/`--retry-wait`/`--timeout`, documented divergence from the app's `--max-tries=1` outer-loop model). kiwix also gets an **outer** reconnect loop (`withRetry` around aria2) so a full Wi-Fi drop — where aria2 exits on DNS (code 19) and can't retry itself — recovers across the handoff, resuming via `--continue`; books' retry budget widened to ~30s to outlast a real reassociation. Maps per-layer checkpoint is pending the on-box `tile-extract.py`. Version stays `1.2.3-dev.N` through the PR (device-verified on -dev.1); final bump to `1.2.3` at merge. (ADFA-4894)
 - **1.2.2** - New `GET /kolibri/subtree/:nodeId`: serves a channel's topic tree in Studio's `contentnode_tree` shape, with per-node byte sizes summed from the local content DB over each node's MPTT range, so the app browses an imported channel's whole tree offline (parsed by the same mapper it uses for Studio). Read-only; **404** when the channel's metadata is not imported, which the app reads as "fall back to Studio". Ships the endpoint PR added without a version bump. (ADFA-5094)
 - **1.2.1** - Four Kolibri defects found by testing against a real device. `/kolibri/estimate` now reports free space (the call was missing the mandatory `?path=Content` and had been failing silently since it was written) and answers **409** with a readable message for a channel that is not installed, instead of letting Kolibri return a bare 500; `/kolibri/catalog` reports real sizes and resource counts (it was reading Studio's field names on an endpoint that uses Kolibri's); a failed import now carries the cause instead of just the exception class name. Also finishes the English pass: `routes.ts` was the last file the 1.1.6 sweep missed — REST-facing too, since one response string changes (`POST /credentials/kolibri` on 401). (ADFA-4954)

static/dashboard/package.json

@@ -1,10 +1,11 @@
 {
   "name": "dashboard-console",
-  "version": "1.2.3-dev.2",
+  "version": "1.2.4",
   "description": "",
   "main": "index.js",
   "scripts": {
     "test": "node --require ts-node/register --test sockets/maps.socket.test.ts sockets/rolling-log.test.ts sockets/kolibri.session.test.ts sockets/credentials.test.ts sockets/net-retry.test.ts",
+    "test:db": "node --require ts-node/register --test sockets/jobs.test.ts",
     "typecheck": "tsc --noEmit",
     "build": "tsc",
     "start": "node dist/server.js"

static/dashboard/routes.ts

@@ -722,3 +722,27 @@ apiRouter.post('/:type/jobs/:id/cancel', (req: Request, res: Response): void =>
     jobs.cancel(job.id);
     res.json({ ok: true });
 });
+
+// ADFA-4894 (control surface): pause / resume / retry over the durable job engine. Pause keeps the
+// partial (resume continues from it via --continue / per-item); retry re-runs an errored job. Each
+// returns 409 when the job is not in a phase the verb applies to, so the caller reflects real state.
+apiRouter.post('/:type/jobs/:id/pause', (req: Request, res: Response): void => {
+    const job = jobs.get(String(req.params.id));
+    if (!job || job.type !== String(req.params.type)) { res.status(404).json({ error: 'not found' }); return; }
+    if (!jobs.pause(job.id)) { res.status(409).json({ error: 'job is not pausable in its current phase' }); return; }
+    res.json({ ok: true });
+});
+
+apiRouter.post('/:type/jobs/:id/resume', (req: Request, res: Response): void => {
+    const job = jobs.get(String(req.params.id));
+    if (!job || job.type !== String(req.params.type)) { res.status(404).json({ error: 'not found' }); return; }
+    if (!jobs.resume(job.id)) { res.status(409).json({ error: 'job is not paused' }); return; }
+    res.json({ ok: true });
+});
+
+apiRouter.post('/:type/jobs/:id/retry', (req: Request, res: Response): void => {
+    const job = jobs.get(String(req.params.id));
+    if (!job || job.type !== String(req.params.type)) { res.status(404).json({ error: 'not found' }); return; }
+    if (!jobs.retry(job.id)) { res.status(409).json({ error: 'job is not in error' }); return; }
+    res.json({ ok: true });
+});

static/dashboard/sockets/books.exec.ts

@@ -4,9 +4,9 @@
 // the EPUB and upload it to the local Calibre-Web. Ported from the Phase 1 books.socket
 // download_books_batch handler (auth/CSRF + fetch + upload), made durable and reporting
 // structured per-book progress. A job item is { id, title, url }.
-import { jobs, RunnerContext, CanceledError } from './jobs';
+import { jobs, RunnerContext, CanceledError, PausedError, classifyStop } from './jobs';
 import { getCredential } from './credentials';
-import { withRetry, Aborted } from './net-retry';
+import { withRetry } from './net-retry';
 import fs from 'fs';
 import path from 'path';
 
@@ -151,9 +151,12 @@ const booksRunner: (ctx: RunnerContext) => Promise<void> = async (ctx) => {
                 onRetry: ({ attempt, err }) => ctx.log(`[books] retry ${attempt} for "${title}": ${err instanceof Error ? err.message : String(err)}`),
             });
         } catch (err) {
-            // withRetry throws Aborted on cancel; surface it as the engine's CanceledError so the
-            // job lands on 'canceled', not 'error'.
-            if (err instanceof Aborted || ctx.isCanceled()) throw new CanceledError();
+            // ADFA-4894: pause stops the fetch but keeps nothing mid-file (books resumes per item on
+            // the next launch); cancel is terminal. Either way, distinguish from a real error via the
+            // job flags so the phase lands on 'paused' / 'canceled', not 'error'.
+            const stop = classifyStop(ctx);
+            if (stop === 'paused') throw new PausedError();
+            if (stop === 'canceled') throw new CanceledError();
             throw err;
         } finally {
             if (fs.existsSync(tmp)) fs.unlinkSync(tmp);

static/dashboard/sockets/jobs.test.ts

@@ -0,0 +1,95 @@
+/// <reference types="node" />
+// Run with `npm run test:db` — this exercises the JobManager, which loads better-sqlite3's native
+// binding, so it needs the package built (present wherever the dashboard is built/deployed). It is
+// kept out of the default `npm test` so that suite stays green on hosts without the native build.
+//
+// Use an isolated throwaway DB so the test never touches the real /library jobs.db. Must be set
+// BEFORE ./jobs is loaded (the JobManager opens the DB in its constructor) — hence the require()
+// below runs after this line, rather than a hoisted top-of-file import.
+process.env.K2GO_JOBS_DB = `/tmp/jobs-test-${process.pid}-${Date.now()}.db`;
+
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import type { RunnerContext } from './jobs';
+
+// require (not import): a static import is hoisted above the env line; a dynamic import() goes
+// through the ESM loader, which needs a file extension ts-node's CJS require does not.
+const { jobs, CanceledError, PausedError } = require('./jobs') as typeof import('./jobs');
+
+const tick = (ms = 30) => new Promise((r) => setTimeout(r, ms));
+
+/**
+ * A fake runner that reports 'downloading' then blocks on a gate until either released (→ done) or
+ * the job's AbortSignal fires (pause/cancel → the matching engine error). Models a real runner's
+ * pause-vs-cancel handling without a child process or network.
+ */
+function gatedRunner() {
+    let release: (() => void) | null = null;
+    const runner = async (ctx: RunnerContext) => {
+        ctx.update({ phase: 'downloading', percent: 10 });
+        try {
+            await new Promise<void>((resolve, reject) => {
+                release = resolve;
+                ctx.signal.addEventListener('abort', () => reject(new Error('abort')), { once: true });
+            });
+        } catch {
+            if (ctx.isPaused()) throw new PausedError();
+            if (ctx.isCanceled()) throw new CanceledError();
+            throw new Error('unexpected stop');
+        }
+        ctx.update({ phase: 'done', percent: 100 });
+    };
+    return { runner, release: () => release?.() };
+}
+
+test('pause marks paused; resume re-runs to done', async () => {
+    const g = gatedRunner();
+    jobs.registerRunner('kiwix', g.runner);
+
+    const job = jobs.create('kiwix', ['wikipedia/x.zim']);
+    await tick();
+    assert.equal(jobs.get(job.id)?.phase, 'downloading');
+
+    assert.equal(jobs.pause(job.id), true);
+    await tick();
+    assert.equal(jobs.get(job.id)?.phase, 'paused');   // paused, not canceled
+
+    assert.equal(jobs.resume(job.id), true);
+    await tick();                                        // let the relaunched run reach its gate
+    g.release();
+    await tick();
+    assert.equal(jobs.get(job.id)?.phase, 'done');
+});
+
+test('retry re-runs a job that ended in error', async () => {
+    let attempt = 0;
+    jobs.registerRunner('kiwix', async (ctx: RunnerContext) => {
+        attempt++;
+        ctx.update({ phase: 'downloading' });
+        if (attempt === 1) throw new Error('boom');   // first run fails
+        ctx.update({ phase: 'done', percent: 100 });  // retry succeeds
+    });
+
+    const job = jobs.create('kiwix', ['wikipedia/y.zim']);
+    await tick();
+    assert.equal(jobs.get(job.id)?.phase, 'error');
+
+    assert.equal(jobs.retry(job.id), true);
+    await tick();
+    assert.equal(jobs.get(job.id)?.phase, 'done');
+    assert.equal(attempt, 2);
+});
+
+test('the verbs no-op outside their phase', async () => {
+    jobs.registerRunner('kiwix', async (ctx: RunnerContext) => {
+        ctx.update({ phase: 'done', percent: 100 });   // completes immediately
+    });
+
+    const job = jobs.create('kiwix', ['wikipedia/z.zim']);
+    await tick();
+    assert.equal(jobs.get(job.id)?.phase, 'done');
+
+    assert.equal(jobs.pause(job.id), false);   // not active
+    assert.equal(jobs.resume(job.id), false);  // not paused
+    assert.equal(jobs.retry(job.id), false);   // not in error
+});

static/dashboard/sockets/jobs.ts

@@ -16,6 +16,9 @@ import { RollingLog, LogSlice } from './rolling-log';
 export type JobType = 'kiwix' | 'maps' | 'books' | 'kolibri';
 export type JobPhase =
     | 'queued' | 'downloading' | 'indexing' | 'processing'
+    // ADFA-4894 (control surface): 'paused' is a stopped-but-resumable state — like 'canceled' it
+    // halts the runner, but UNLIKE cancel it keeps the partial on disk so resume() continues from it.
+    | 'paused'
     | 'done' | 'error' | 'canceled';
 
 /** A persisted job row. `percent` is -1 when indeterminate; `speed` is bytes/sec. */
@@ -50,7 +53,10 @@ export interface RunnerContext {
     update(patch: JobUpdate): void;
     isCanceled(): boolean;
     throwIfCanceled(): void;
-    /** ADFA-4894: aborted when the job is canceled, so an in-flight fetch (books) can be
+    /** ADFA-4894 (control surface): true when the job was PAUSED (not canceled). A runner uses this
+     *  to skip its clean-on-cancel — a paused download keeps its partial so resume() continues it. */
+    isPaused(): boolean;
+    /** ADFA-4894: aborted when the job is canceled OR paused, so an in-flight fetch (books) can be
      *  interrupted — a child process is killed via SIGKILL, but a fetch has no process. */
     readonly signal: AbortSignal;
     /** Spawn a child that is killed automatically on cancel and untracked on exit. */
@@ -65,13 +71,30 @@ export class CanceledError extends Error {
     constructor() { super('canceled'); this.name = 'CanceledError'; }
 }
 
+/** ADFA-4894: a runner throws this when stopped by pause(); mapped to 'paused' (partial kept). */
+export class PausedError extends Error {
+    constructor() { super('paused'); this.name = 'PausedError'; }
+}
+
+/**
+ * ADFA-4894: why a runner is stopping — so every runner handles pause vs cancel the same way (pause
+ * keeps the partial; cancel cleans it). Returns null for a real error. Keyed on the job flags, not
+ * the error type: a fetch/withRetry only aborts because the job was paused or canceled, so the flags
+ * are the source of truth and each runner does just its own cleanup on 'canceled'.
+ */
+export function classifyStop(ctx: RunnerContext): 'paused' | 'canceled' | null {
+    if (ctx.isPaused()) return 'paused';
+    if (ctx.isCanceled()) return 'canceled';
+    return null;
+}
+
 const DB_PATH = process.env.K2GO_JOBS_DB || '/library/dashboard/jobs.db';
 const ACTIVE: JobPhase[] = ['queued', 'downloading', 'indexing', 'processing'];
 
 class JobManager {
     private db: Database.Database;
     private runners = new Map<JobType, Runner>();
-    private runtime = new Map<string, { canceled: boolean; procs: Set<ChildProcess>; ac: AbortController }>();
+    private runtime = new Map<string, { canceled: boolean; paused: boolean; procs: Set<ChildProcess>; ac: AbortController }>();
     // ADFA-4879: bounded rolling log tail per job for the live-log REST endpoint.
     private readonly logTail = new RollingLog();
 
@@ -143,6 +166,58 @@ class JobManager {
         return true;
     }
 
+    /**
+     * ADFA-4894 (control surface): pause a running job. Stops the runner like cancel — kills its
+     * children, aborts an in-flight fetch — but marks it 'paused' and, crucially, the runner does NOT
+     * clean its partial (it checks {@link RunnerContext.isPaused}), so {@link #resume} continues from
+     * the checkpoint. Only an ACTIVE job can be paused. Idempotent on a non-active job.
+     */
+    pause(id: string): boolean {
+        const job = this.get(id);
+        if (!job) return false;
+        if (!ACTIVE.includes(job.phase)) return false;
+        const rt = this.runtime.get(id);
+        if (rt) {
+            rt.paused = true;
+            for (const p of rt.procs) { try { p.kill('SIGKILL'); } catch { /* already gone */ } }
+            try { rt.ac.abort(); } catch { /* interrupt an in-flight fetch */ }
+        }
+        this.patch(id, { phase: 'paused' });
+        return true;
+    }
+
+    /**
+     * ADFA-4894: resume a paused job — re-launch the runner, which continues from its on-disk
+     * checkpoint (aria2 --continue for kiwix; per-item for books). No-op unless the job is 'paused'
+     * and nothing is already running for it.
+     */
+    resume(id: string): boolean {
+        const job = this.get(id);
+        if (!job || job.phase !== 'paused') return false;
+        if (this.runtime.has(id)) return false;   // already running
+        // Leave 'paused' for an ACTIVE phase before relaunching, so the completion guard in launch()
+        // (which only finalizes an ACTIVE job to 'done') and the UI both see it running at once.
+        this.patch(id, { phase: 'queued' });
+        this.launch(job);
+        return true;
+    }
+
+    /**
+     * ADFA-4894: retry a job that ended in 'error' — re-launch it. Where the runner is resumable the
+     * partial is still on disk (kiwix keeps it on a real error), so this continues rather than
+     * restarts. No-op unless the job is in 'error' and nothing is already running for it.
+     */
+    retry(id: string): boolean {
+        const job = this.get(id);
+        if (!job || job.phase !== 'error') return false;
+        if (this.runtime.has(id)) return false;
+        // Move off 'error' to an ACTIVE phase before relaunch (clears the error, satisfies the
+        // completion guard, and the UI shows it running immediately).
+        this.patch(id, { phase: 'queued', error: null });
+        this.launch(job);
+        return true;
+    }
+
     /** Resume jobs that were mid-flight when the dashboard last stopped. */
     reconcileOnBoot(): void {
         const stuck = this.db.prepare(
@@ -172,7 +247,7 @@ class JobManager {
         const runner = this.runners.get(job.type);
         if (!runner) { this.patch(job.id, { phase: 'error', error: `no runner for ${job.type}` }); return; }
 
-        const rt = { canceled: false, procs: new Set<ChildProcess>(), ac: new AbortController() };
+        const rt = { canceled: false, paused: false, procs: new Set<ChildProcess>(), ac: new AbortController() };
         this.runtime.set(job.id, rt);
 
         let items: unknown[] = [];
@@ -184,6 +259,7 @@ class JobManager {
             update: (p) => this.patch(job.id, p),
             isCanceled: () => rt.canceled,
             throwIfCanceled: () => { if (rt.canceled) throw new CanceledError(); },
+            isPaused: () => rt.paused,
             signal: rt.ac.signal,
             spawn: (cmd, args, opts) => {
                 const child = opts ? spawn(cmd, args, opts) : spawn(cmd, args);
@@ -201,7 +277,9 @@ class JobManager {
                 if (cur && ACTIVE.includes(cur.phase)) this.patch(job.id, { phase: 'done', percent: 100 });
             })
             .catch((err: unknown) => {
-                if (err instanceof CanceledError || rt.canceled) {
+                if (err instanceof PausedError || rt.paused) {
+                    this.patch(job.id, { phase: 'paused' });
+                } else if (err instanceof CanceledError || rt.canceled) {
                     this.patch(job.id, { phase: 'canceled' });
                 } else {
                     const msg = err instanceof Error ? err.message : String(err);

static/dashboard/sockets/kiwix.exec.ts

@@ -5,8 +5,8 @@
 // bytes/sec) instead of streamed as terminal text. Ported from the Phase 1
 // kiwix.socket handler, minus the socket/closure lifetime — the job outlives any
 // client (see jobs.ts).
-import { jobs, RunnerContext, CanceledError } from './jobs';
-import { withRetry, Aborted } from './net-retry';
+import { jobs, RunnerContext, CanceledError, PausedError, classifyStop } from './jobs';
+import { withRetry } from './net-retry';
 import { execSync } from 'child_process';
 import fs from 'fs';
 import path from 'path';
@@ -182,12 +182,12 @@ const kiwixRunner: (ctx: RunnerContext) => Promise<void> = async (ctx) => {
             onRetry: ({ attempt, err }) => ctx.log(`[kiwix] reconnect attempt ${attempt} after: ${err instanceof Error ? err.message : String(err)}`),
         });
     } catch (e) {
-        // ADFA-4894: a canceled download leaves nothing half-written; a real error KEEPS the partial
-        // (+ .aria2) so a later run resumes it via --continue rather than starting from zero.
-        if (e instanceof CanceledError || e instanceof Aborted || ctx.isCanceled()) {
-            cleanupPartials(files);
-            throw new CanceledError();
-        }
+        // ADFA-4894: pause KEEPS the partial (+ .aria2) so resume continues via --continue; cancel
+        // discards it; a real error also keeps it, so a later retry/reconcile resumes rather than
+        // starting from zero.
+        const stop = classifyStop(ctx);
+        if (stop === 'paused') throw new PausedError();
+        if (stop === 'canceled') { cleanupPartials(files); throw new CanceledError(); }
         throw e;
     }