Actors stay unreachable ("Actor not found") after an engine node is lost, with the Postgres driver in multi-node mode #5915

Summary

Human: This is a real issue i have reproduced multiple time. Below the report made by claude opus on this problem.

With two engine nodes on Postgres and NATS, killing the engine node that holds the runners' (envoys') WebSocket connections while actors are busy leaves some actors unreachable for at least 5 minutes (up to about 15 minutes checked), until the runners restart. Requests return Actor not found; the runner logs received request for unknown actor; the engine still reports the actor as connectable. Neither POST /actors/{id}/sleep nor POST /actors/{id}/reschedule recovers it. Restarting the runner processes does.

It happened in 7 of 46 kills. The affected actors use @rivet-dev/pi (Pi Durable on the actor's SQLite). It did not happen in 30 kills with plain actors that only write to their SQLite.

Report written with the help of an AI coding assistant; the runs and logs below are real.

Versions

• Engine: rivetdev/engine:slim-2.3.30
• rivetkit 2.3.30 (envoy mode, RIVETKIT_RUNTIME_MODE default)
• @rivet-dev/pi 0.5.3, @earendil-works/pi-durable 1.3.0
• Node 24.22.1
• PostgreSQL 18.6 (single instance)
• NATS 2.15.1 (3-node cluster)

Environment

Local Kubernetes 1.36 (Docker Desktop, kind, 3 nodes, arm64). Engine config, all defaults otherwise:

{
  "postgres": { "url": "postgresql://<user>:<password>@postgres:5432/rivet?sslmode=disable" },
  "nats": { "addresses": ["nats-0:4222", "nats-1:4222", "nats-2:4222"], "username": "<user>", "password": "<password>" },
  "auth": { "admin_token": "<token>" },
  "telemetry": { "enabled": false }
}

• Two engine replicas (Deployment, required pod anti-affinity), behind one ClusterIP Service.
• Two runner pods (RIVET_ENDPOINT pointing at that Service, RIVET_NAMESPACE, RIVET_TOKEN a namespace JWT with runner: create), so each runner's WebSocket lands on either engine node.
• One namespace, runner config {"datacenters":{"default":{"normal":{}}}}.

Steps to reproduce

1. Create 100 actors (one Pi Durable agent each, crash_policy default sleep). From one client, send each actor 6 sequential prompt actions; each prompt runs one tool call, so each action commits several times to the actor's SQLite.
2. While this load runs, find the engine pod holding the most runner WebSockets (from its envoy websocket connected / closed log lines) and delete it: kubectl delete pod <engine-pod> --force --grace-period=0.
3. When the load ends, call a trivial action on each of the 100 actors. Repeat 5 minutes after the kill.

Expected

Actors interrupted by the lost node are rescheduled and answer within roughly the envoy lost threshold (15 s) plus start time, as most do.

Actual

In 7 of 46 runs, between 1 and 89 of the 100 actors answer Actor not found at the end of the load and still 5 minutes after the kill (the longest wait checked was about 15 minutes). For one of them, GET /actors?namespace=<ns>&name=agent&key=<key> returned:

"connectable_ts": <set, time of the kill>, "sleep_ts": null, "reschedule_ts": null, "destroy_ts": null, "crash_policy": "sleep"

POST /actors/{id}/sleep returned 200; the next call then timed out. POST /actors/{id}/reschedule returned 200; calls still returned Actor not found. Restarting the runner pods made every actor answer again, with its state intact.

Logs

One stuck actor (<actor-1>, key agent-52) from a run on 2026-10-24, UTC. Runner (the pod that hosted it):

01:11:14.895 error sqlite atomic commit failed actor_id=<actor-1> err="remote sqlite commit result is indeterminate after envoy disconnect"
01:11:29.899 warn  stopping all actors due to envoy lost threshold
01:11:29.899 error sqlite flush commit failed actor_id=<actor-1> err="envoy shut down"
01:11:29.900 warn  slow sqlite transaction database=<actor-1> duration_ms=15069
             warn  pi durable stopped after a failed commit, reopening it from storage actorKey=["agent-52"]
01:11:34.148 warn  actor framework action error response actor_id=<actor-1> generation=1 code=internal_error
             warn  pi durable busy check failed error="Database is closed. ..."
01:11:34.198 error graceful cleanup callback failed actor_id=<actor-1> error="Session failed after a storage error; close and reopen it"
01:11:34 .. 01:16:46  warn received request for unknown actor actor_id=<actor-1> actor_generation=None   (17 times)

Surviving engine node, same seconds:

01:11:33.644 info envoy websocket connected envoy_key=<envoy-A>
01:11:34.018 warn actor not allocated, ignoring events envoy_key=<envoy-A> events=[... EventActorStateUpdate { state: ActorStateRunning } ...]

The engine logged many actor not allocated, ignoring events lines after runners reconnected, including batches with ActorStateStopped. None of them names <actor-1>, so the logs do not show what happened to this actor's own events.

Frequency

No acknowledged action was lost and no tool effect was applied twice in any run.

Workaround

Restart the runner processes. We now run a guard in the runner that restarts it when it keeps receiving requests for an actor it does not know; in 72 further kills, the 5 stuck cases recovered on their own in 43 to 58 s.

Possibly related (hypothesis)

• fix(pegboard-envoy): retry failed actor event dispatch instead of dropping events #5859 (open): a failed actor event dispatch drops later events, including "stopped". The engine warnings above (events ignored, ActorStateStopped among them) point the same way.
• Existing actor stays unreachable (no_envoys, sliding reschedule) after runner restart if woken while no envoy is connected — Wake ignored in Reallocating (2.3.27) #5894 (open): an existing actor stays unreachable after a runner restart.
• pegboard_actor2 workflow dies with LeaseFenceMismatch; stale envoy routing survives restart (2.3.27, observed once) #5896 (open): stale envoy routing survives after a pegboard_actor2 workflow dies with LeaseFenceMismatch; Sleep does not revive the actor there either. That report is single-node RocksDB and was seen once.
• v2 (envoy) actors ignore crash_policy: a lost runner parks the actor (Sleep) instead of rescheduling it #5908 (open): a lost runner parks v2 actors (Sleep) regardless of crash_policy. Different symptom (the actor sleeps, it is not stuck as running), same area.
• The failing cleanup is the actor's Pi Durable session, run by @rivet-dev/pi (published from rivet-dev/agents, packages/pi); the error text "Session failed after a storage error" comes from @earendil-works/pi-durable. Part of the problem may be there rather than in the engine.

The common factor in the stuck cases is an actor whose SQLite commit becomes indeterminate when the runner's connection to the dead engine node drops, and whose cleanup then fails. Plain actors did not reach that state in our runs. We have not confirmed this in the engine source.

Not tested

• A reproduction without @rivet-dev/pi: the plain-actor variant above did not trigger it.
• The file-system (single-node) driver.
• Engine versions before 2.3.30, and the patch from fix(pegboard-envoy): retry failed actor event dispatch instead of dropping events #5859.
• @earendil-works/pi-durable 1.4.0, released since; only 1.3.0 was used.