Skip to content

feat: auto-resume workflows after a provider usage limit resets (closes #27) - #78

Merged
QuintinShaw merged 1 commit into
mainfrom
feat/auto-resume
Jul 16, 2026
Merged

feat: auto-resume workflows after a provider usage limit resets (closes #27)#78
QuintinShaw merged 1 commit into
mainfrom
feat/auto-resume

Conversation

@QuintinShaw

Copy link
Copy Markdown
Owner

Closes #27.

What

A run that pauses on a provider usage/quota limit (the #26 behavior) now auto-resumes once the quota is likely to have refilled — instead of sitting paused until someone manually runs /workflows resume. Exponential backoff + a hard attempt cap keep it from bouncing off a still-exhausted budget forever.

Design: standalone scheduler

src/usage-limit-scheduler.ts — a UsageLimitScheduler that consumes only the manager's public surface (on/off, listAllRuns, resume, getPersistence). No logic added inside WorkflowManager/executeRun/recoverStaleRuns. This keeps the footprint against the pending #74 manager/persistence rewrite minimal — the only edits to existing files are two additive PersistedRunState fields, ExecOptions.autoResume, and wiring.

Event-driven "fire and watch": arm a timer on a usage_limit pause; when it fires, call resume(); then step back and let the existing paused/complete/error/stopped subscriptions drive backoff and cleanup. An attempt is consumed only when a run enters a usage_limit pause — never when a resume merely fires — so a resume() returning false (lease busy, already gone) doesn't burn an attempt.

Default-on with per-run opt-out

Eligible unless ExecOptions.autoResume === false (persisted, so a cold start respects it). Tunables: attempt cap 5, delay floor 1m, ceiling 6h, resetHint-unparseable fallback 5m.

resetHint & cold start

resetHint is a verbatim human string ("Resets in ~3h") and not guaranteed present — parsed best-effort (~3h, 1h30m, in 5m, missing → fallback). Cold-start re-arm reschedules a still-paused run using time remaining (now − updatedAt), not a fresh full delay.

Tests

tests/usage-limit-scheduler.test.ts — 26 tests using an injectable fake clock/timer that fires callbacks synchronously, so the timer-fire path is genuinely exercised (normal fire, backoff-on-repause, attempt-cap give-up, cold-start remaining-time, opt-out skip, resetHint parsing, delay floor, resume()→false not consuming an attempt). 887 unit tests pass; tsc + biome clean.

Honest notes

  • Test boundary: true end-to-end validation needs a real provider usage limit to fire, which isn't reproducible on demand. Reliability here rests on the fake-clock unit tests covering every branch (which the prior attempt lacked) plus the existing real-session pause/resume integration test.
  • autoResumeAttempts persistence is best-effort: the scheduler writes it, and the manager's persistRun() would overwrite it while a run is executing — but that only matters for a process crash in the brief window between a resume and the next persist. While paused (when a cold start actually reads it) the value is stable, and the live in-memory counter is always authoritative. Fails safe toward more retries, not fewer.
  • feat: make workflow runs observable and resumable #74 overlap: if feat: make workflow runs observable and resumable #74 lands, its migrateRunState() will need to pass through the two new PersistedRunState fields.

…age limit resets

Closes #27. A run that pauses on a provider usage/quota limit now auto-resumes
once the quota is likely refilled, instead of sitting paused until a human runs
/workflows resume.

New standalone module src/usage-limit-scheduler.ts (UsageLimitScheduler):
consumes only the manager's public event/query surface, staying decoupled from
manager/persistence internals (minimal footprint against the pending large #74
rewrite). Event-driven 'fire and watch': arm a timer on a usage_limit pause,
call resume() when it fires, and let the existing 'paused'/'complete'/'error'
subscriptions drive backoff and cleanup.

Default-on with per-run opt-out via ExecOptions.autoResume (persisted).
Exponential backoff, hard attempt cap (5), 1m delay floor, 6h ceiling.
resetHint parsed best-effort ('~3h', '1h30m', 'in 5m'), falls back to 5m.
Cold-start re-arm: a run still paused-on-usage_limit at process start is
rescheduled using time REMAINING (now - updatedAt), not a fresh full delay.

An injectable clock/timer makes the timer-fire path fully unit-tested (26
tests): normal fire, backoff-on-repause, attempt-cap give-up, cold-start
remaining-time math, opt-out skip, resetHint parsing, delay floor, and
resume()->false not consuming an attempt.

Additive-only to existing files: ExecOptions.autoResume + two PersistedRunState
fields + scheduler wiring in extensions/workflow.ts (disposed on
session_shutdown).
@QuintinShaw
QuintinShaw merged commit 6b45b20 into main Jul 16, 2026
1 check passed
@QuintinShaw
QuintinShaw deleted the feat/auto-resume branch July 16, 2026 07:09
QuintinShaw added a commit that referenced this pull request Jul 16, 2026
…84)

The orchestrating model can re-call the workflow tool with resumeFromRunId plus
an edited script to iterate on a run: unchanged agent() calls replay from the
journal (cache), only edited/new ones re-run — instead of re-running the whole
workflow to fix one bad prompt in one agent() call.

- WorkflowManager.resume(runId, opts?: {script?, args?}): optional edited-script
  override. With no opts it is byte-for-byte the old behavior (persisted script),
  so #78 auto-resume and the TUI /workflows resume path are unchanged.
- workflow tool: optional resumeFromRunId; when set, resume that run with THIS
  call's (edited) script instead of starting a new run; clear error text when the
  run isn't resumable (not found / running / completed / stopped / busy).
- Discoverability WITHOUT growing the always-on prompt: the resumeFromRunId
  tool-definition description + a per-result revise hint carry it; no always-on
  guideline line, so the rendered-prompt budget is unchanged (respects the
  prompt-size work in #66).
- runId alignment: executeRun now passes managed.runId into runWorkflow, so the
  sync path's result.runId is the real resumable id instead of an ephemeral
  run-<ts> (a pre-existing latent mismatch).
- Positional-callIndex limit (reorder/insert/remove before an unchanged call
  invalidates the cache from there on) degrades correctly and is documented in
  the tool description + README.

895 unit tests pass; tsc + biome clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Auto-resume a workflow once a provider usage limit resets

1 participant