feat: auto-resume workflows after a provider usage limit resets (closes #27) - #78
Merged
Conversation
…age limit resets Closes #27. A run that pauses on a provider usage/quota limit now auto-resumes once the quota is likely refilled, instead of sitting paused until a human runs /workflows resume. New standalone module src/usage-limit-scheduler.ts (UsageLimitScheduler): consumes only the manager's public event/query surface, staying decoupled from manager/persistence internals (minimal footprint against the pending large #74 rewrite). Event-driven 'fire and watch': arm a timer on a usage_limit pause, call resume() when it fires, and let the existing 'paused'/'complete'/'error' subscriptions drive backoff and cleanup. Default-on with per-run opt-out via ExecOptions.autoResume (persisted). Exponential backoff, hard attempt cap (5), 1m delay floor, 6h ceiling. resetHint parsed best-effort ('~3h', '1h30m', 'in 5m'), falls back to 5m. Cold-start re-arm: a run still paused-on-usage_limit at process start is rescheduled using time REMAINING (now - updatedAt), not a fresh full delay. An injectable clock/timer makes the timer-fire path fully unit-tested (26 tests): normal fire, backoff-on-repause, attempt-cap give-up, cold-start remaining-time math, opt-out skip, resetHint parsing, delay floor, and resume()->false not consuming an attempt. Additive-only to existing files: ExecOptions.autoResume + two PersistedRunState fields + scheduler wiring in extensions/workflow.ts (disposed on session_shutdown).
QuintinShaw
added a commit
that referenced
this pull request
Jul 16, 2026
…84) The orchestrating model can re-call the workflow tool with resumeFromRunId plus an edited script to iterate on a run: unchanged agent() calls replay from the journal (cache), only edited/new ones re-run — instead of re-running the whole workflow to fix one bad prompt in one agent() call. - WorkflowManager.resume(runId, opts?: {script?, args?}): optional edited-script override. With no opts it is byte-for-byte the old behavior (persisted script), so #78 auto-resume and the TUI /workflows resume path are unchanged. - workflow tool: optional resumeFromRunId; when set, resume that run with THIS call's (edited) script instead of starting a new run; clear error text when the run isn't resumable (not found / running / completed / stopped / busy). - Discoverability WITHOUT growing the always-on prompt: the resumeFromRunId tool-definition description + a per-result revise hint carry it; no always-on guideline line, so the rendered-prompt budget is unchanged (respects the prompt-size work in #66). - runId alignment: executeRun now passes managed.runId into runWorkflow, so the sync path's result.runId is the real resumable id instead of an ephemeral run-<ts> (a pre-existing latent mismatch). - Positional-callIndex limit (reorder/insert/remove before an unchanged call invalidates the cache from there on) degrades correctly and is documented in the tool description + README. 895 unit tests pass; tsc + biome clean.
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #27.
What
A run that pauses on a provider usage/quota limit (the #26 behavior) now auto-resumes once the quota is likely to have refilled — instead of sitting paused until someone manually runs
/workflows resume. Exponential backoff + a hard attempt cap keep it from bouncing off a still-exhausted budget forever.Design: standalone scheduler
src/usage-limit-scheduler.ts— aUsageLimitSchedulerthat consumes only the manager's public surface (on/off,listAllRuns,resume,getPersistence). No logic added insideWorkflowManager/executeRun/recoverStaleRuns. This keeps the footprint against the pending #74 manager/persistence rewrite minimal — the only edits to existing files are two additivePersistedRunStatefields,ExecOptions.autoResume, and wiring.Event-driven "fire and watch": arm a timer on a
usage_limitpause; when it fires, callresume(); then step back and let the existingpaused/complete/error/stoppedsubscriptions drive backoff and cleanup. An attempt is consumed only when a run enters a usage_limit pause — never when a resume merely fires — so aresume()returningfalse(lease busy, already gone) doesn't burn an attempt.Default-on with per-run opt-out
Eligible unless
ExecOptions.autoResume === false(persisted, so a cold start respects it). Tunables: attempt cap 5, delay floor 1m, ceiling 6h, resetHint-unparseable fallback 5m.resetHint & cold start
resetHintis a verbatim human string ("Resets in ~3h") and not guaranteed present — parsed best-effort (~3h,1h30m,in 5m, missing → fallback). Cold-start re-arm reschedules a still-paused run using time remaining (now − updatedAt), not a fresh full delay.Tests
tests/usage-limit-scheduler.test.ts— 26 tests using an injectable fake clock/timer that fires callbacks synchronously, so the timer-fire path is genuinely exercised (normal fire, backoff-on-repause, attempt-cap give-up, cold-start remaining-time, opt-out skip, resetHint parsing, delay floor,resume()→false not consuming an attempt). 887 unit tests pass; tsc + biome clean.Honest notes
autoResumeAttemptspersistence is best-effort: the scheduler writes it, and the manager'spersistRun()would overwrite it while a run is executing — but that only matters for a process crash in the brief window between a resume and the next persist. While paused (when a cold start actually reads it) the value is stable, and the live in-memory counter is always authoritative. Fails safe toward more retries, not fewer.migrateRunState()will need to pass through the two newPersistedRunStatefields.