Repository navigation
v2 (envoy) actors ignore crash_policy: a lost runner parks the actor (Sleep) instead of rescheduling it #5908
Description
Environment
rivetkit2.3.27,rivetdev/engine:2.3.27(self-hosted, Docker), Postgres backend- Node 22, one runner pool (
default), actors created throughrivetkitand through the engine REST API
Expected
versions.mdx says that on SIGKILL "actors are rescheduled with a crash backoff penalty", and
actor-rescheduling.md maps a lost runner to reschedule_actor. Creating an actor with
crash_policy: "restart" should make it restart on another (or the restarted) runner after the
runner is lost.
Actual
After kill -9 of the runner process, the actor's last stop reports crash_policy: "sleep",
error: {envoy_connection_lost}, and the actor stays parked until something sends it a message or
an alarm fires. A workflow in progress does not resume on its own.
rivetkit's client always sends"sleep"and does not expose the option.- Creating the actor via the engine REST API with
crash_policy: "restart"still reports"sleep". - Engine source (2.3.27 and
main):pegboard/src/workflows/actor2/runtime.rshandle_stopped
mapsStoppedVariant::Lost => Decision::Sleepand only reallocates if an alarm is due;
pegboard/src/ops/actor/util.rsreports v2 actors asCrashPolicy::Sleepunconditionally.
Related: #3725 (closed, not planned), PR #4822 "restore crash_policy plumbing" (closed unmerged),
#5894 (open; wake ignored while Reallocating — different trigger).
Repro
// runner.ts — start with `tsx runner.ts`, then `kill -9` it while `work` is mid-loop, restart it.
import { actor, setup } from "rivetkit";
import { workflow } from "rivetkit/workflow";
const job = actor({
run: workflow(async (ctx) => {
await ctx.queue.next("go");
for (let i = 0; i < 20; i++) {
await ctx.step(`page-${i}`, async () => {
await new Promise((r) => setTimeout(r, 500));
console.log("page", i, "done");
});
}
}),
actions: { go: async (c) => c.queue.send("go", {}) },
});
const registry = setup({ use: { job }, endpoint: "http://127.0.0.1:6420", token: "<admin-token>", namespace: "default" });
await registry.startAndWait();- Start the engine (2.3.27) and
runner.ts; callclient.job.getOrCreate(["a"]).go(). - After ~4 pages,
kill -9the runner; restart it 3 s later. - Observe: no further
page N donelines. The actor stays asleep (GET /actors/{id}shows
crash_policy: "sleep",envoy_connection_lost) until it is messaged. - Same result when the actor is created with
POST /actorsand"crash_policy": "restart".
Measured
- Stock 2.3.27, 3 s runner downtime: the workflow did not finish within 300 s (2 of 2 runs, with
and withoutcrash_policy: "restart"on the REST create). - Workaround we ship: a durable alarm (
c.run.setWakeAt, 3 s, refreshed every 1 s, cleared in
onSleep) pluspegboard.envoy_lost_threshold = 6 s(default 15 s). Resume after restart:
3.4–8.5 s over 4 spike runs; 2.6–2.7 s in our integration suite (3 runs); 0 completed steps re-run. - SIGTERM drain (rolling deploy) is fine without any workaround: resume 0.4–0.7 s.
Question / suggested fix
Is Lost => Sleep for v2 actors intended? If yes, please fix versions.mdx /
actor-rescheduling.md. If not, honour crash_policy: "restart" for v2 actors (and expose it in
rivetkit actor options) so users don't need an alarm-based watchdog to survive a runner crash.
Activity
linear-code commentedon Oct 20, 2026
ACT-6514