Skip to content

DELETE /actors/{id} leaves the destroyed actor's data (workflow step output, queue bodies, state) in engine KV under Depot DELTA keys #5909

Closed

Description

@hoghweed

Environment

  • rivetkit 2.3.27, rivetdev/engine:2.3.27 (self-hosted, Docker), Postgres backend (rivet_engine database)

Expected

destroy.mdx: "The actor's state is permanently deleted." Engine actor2/mod.rs destroy runs
ClearKv + clear_v2_storage_for_destroy. After DELETE /actors/{id} we expect no row in the
engine kv table to contain values written by that actor.

Actual

Unique marker strings written by the actor as (a) a workflow step return value, (b) a queue message
body and (c) actor state are still present in the engine's kv table after destroy — 5 s, 35 s,
30 min and 13.2 h later (5 rows under Depot DELTA keys in the 13.2 h check).

Repro

// probe.ts — `tsx probe.ts` against a local engine whose Postgres is on 127.0.0.1:54329
import { randomUUID } from "node:crypto";
import pg from "pg";
import { actor, setup } from "rivetkit";
import { createClient } from "rivetkit/client";
import { workflow } from "rivetkit/workflow";

const ENDPOINT = "http://127.0.0.1:6420";
const TOKEN = "<admin-token>";
const step = `STEP-${randomUUID()}`, queued = `QUEUE-${randomUUID()}`, stateMark = `STATE-${randomUUID()}`;

const leaky = actor({
  state: { note: "" },
  run: workflow(async (ctx) => {
    const msg = await ctx.queue.next("go");
    await ctx.step("leak", async (s) => {
      s.state.note = stateMark;
      return { note: step, from: JSON.stringify(msg.body) };
    });
    await ctx.queue.next("never");
  }),
  actions: { go: async (c, body: string) => c.queue.send("go", { body }) },
});
const registry = setup({ use: { leaky }, endpoint: ENDPOINT, token: TOKEN, namespace: "default" });
await registry.startAndWait();
const client = createClient<typeof registry>({ endpoint: ENDPOINT, token: TOKEN, namespace: "default" });
const h = client.leaky.getOrCreate([`probe-${Date.now()}`]);
await h.go(queued);

const db = new pg.Pool({ connectionString: "postgres://rivet:rivet@127.0.0.1:54329/rivet_engine" });
const scan = async (label: string) => {
  const { rows } = await db.query<{ key: Buffer; value: Buffer }>("SELECT key, value FROM kv");
  const hay = Buffer.concat(rows.flatMap((r) => [r.key, r.value]));
  console.log(label, { step: hay.includes(step), queue: hay.includes(queued), state: hay.includes(stateMark) });
};
await new Promise((r) => setTimeout(r, 3000));
await scan("before destroy");
const id = await h.resolve();
await fetch(`${ENDPOINT}/actors/${id}?namespace=default`, { method: "DELETE", headers: { authorization: `Bearer ${TOKEN}` } });
await new Promise((r) => setTimeout(r, 35000));
await scan("35 s after DELETE /actors");

Measured

  • All three markers present after destroy at +5 s and +35 s; rows not reclaimed within 30 min.
  • Long check: actor destroyed, markers still present 13.2 h later, 5 rows under Depot DELTA keys.
  • In our purge integration test (destroy every actor of a deleted tenant, then scan engine KV for
    that tenant's identifiers): 79 / 83 / 92 matching rows across three runs, growing with the shared
    engine database. The actors themselves are gone (no live entry), only the data stays.
  • rivet_engine grew to 27 MB / 60k KV rows over a small spike (2 tenants, ~10 workflows).

Question / suggested fix

Should destroy (or a later GC) delete the actor's Depot DELTA keys? If retention is intended,
please document how long and how to purge — we need it for data-deletion requests (GDPR erasure).
Today we can't rely on DELETE /actors for deletion and avoid touching engine tables directly.

Activity

  1. linear-code commented on Oct 20, 2026

    @linear-code
  2. self-assigned this
    on Oct 22, 2026
  3. NathanFlurry commented on Oct 23, 2026

    @NathanFlurry
    Member

    We fixed a related issue with deletion of branch-backed actor SQLite data in Rivet Cloud, but that cleanup path still needs to be applied to OSS.

    Closing since this is resolved in 3.0 upcoming release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions