Skip to content

v2 (envoy) actors ignore crash_policy: a lost runner parks the actor (Sleep) instead of rescheduling it #5908

Open

Description

@hoghweed

Environment

  • rivetkit 2.3.27, rivetdev/engine:2.3.27 (self-hosted, Docker), Postgres backend
  • Node 22, one runner pool (default), actors created through rivetkit and through the engine REST API

Expected

versions.mdx says that on SIGKILL "actors are rescheduled with a crash backoff penalty", and
actor-rescheduling.md maps a lost runner to reschedule_actor. Creating an actor with
crash_policy: "restart" should make it restart on another (or the restarted) runner after the
runner is lost.

Actual

After kill -9 of the runner process, the actor's last stop reports crash_policy: "sleep",
error: {envoy_connection_lost}, and the actor stays parked until something sends it a message or
an alarm fires. A workflow in progress does not resume on its own.

  • rivetkit's client always sends "sleep" and does not expose the option.
  • Creating the actor via the engine REST API with crash_policy: "restart" still reports "sleep".
  • Engine source (2.3.27 and main): pegboard/src/workflows/actor2/runtime.rs handle_stopped
    maps StoppedVariant::Lost => Decision::Sleep and only reallocates if an alarm is due;
    pegboard/src/ops/actor/util.rs reports v2 actors as CrashPolicy::Sleep unconditionally.

Related: #3725 (closed, not planned), PR #4822 "restore crash_policy plumbing" (closed unmerged),
#5894 (open; wake ignored while Reallocating — different trigger).

Repro

// runner.ts — start with `tsx runner.ts`, then `kill -9` it while `work` is mid-loop, restart it.
import { actor, setup } from "rivetkit";
import { workflow } from "rivetkit/workflow";

const job = actor({
  run: workflow(async (ctx) => {
    await ctx.queue.next("go");
    for (let i = 0; i < 20; i++) {
      await ctx.step(`page-${i}`, async () => {
        await new Promise((r) => setTimeout(r, 500));
        console.log("page", i, "done");
      });
    }
  }),
  actions: { go: async (c) => c.queue.send("go", {}) },
});
const registry = setup({ use: { job }, endpoint: "http://127.0.0.1:6420", token: "<admin-token>", namespace: "default" });
await registry.startAndWait();
  1. Start the engine (2.3.27) and runner.ts; call client.job.getOrCreate(["a"]).go().
  2. After ~4 pages, kill -9 the runner; restart it 3 s later.
  3. Observe: no further page N done lines. The actor stays asleep (GET /actors/{id} shows
    crash_policy: "sleep", envoy_connection_lost) until it is messaged.
  4. Same result when the actor is created with POST /actors and "crash_policy": "restart".

Measured

  • Stock 2.3.27, 3 s runner downtime: the workflow did not finish within 300 s (2 of 2 runs, with
    and without crash_policy: "restart" on the REST create).
  • Workaround we ship: a durable alarm (c.run.setWakeAt, 3 s, refreshed every 1 s, cleared in
    onSleep) plus pegboard.envoy_lost_threshold = 6 s (default 15 s). Resume after restart:
    3.4–8.5 s over 4 spike runs; 2.6–2.7 s in our integration suite (3 runs); 0 completed steps re-run.
  • SIGTERM drain (rolling deploy) is fine without any workaround: resume 0.4–0.7 s.

Question / suggested fix

Is Lost => Sleep for v2 actors intended? If yes, please fix versions.mdx /
actor-rescheduling.md. If not, honour crash_policy: "restart" for v2 actors (and expose it in
rivetkit actor options) so users don't need an alarm-based watchdog to survive a runner crash.

Activity

  1. linear-code commented on Oct 20, 2026

    @linear-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions