Skip to content

Recover job APIs when TaskStateManager actor dies #841

Description

@hedhoud

Context

Ray workers can exit because of memory pressure, native crashes, or a lost Ray connection. A transient worker failure should not leave OpenRAG's job APIs unusable until the API container is restarted.

Problem

When the TaskStateManager actor dies, OpenRAG retains its dead actor handle. Subsequent job-status, queue, and cancellation requests continue to fail with a raw ActorDiedError. Recreating only the actor may not restore these paths because services can still hold the stale handle.

This turns one worker crash into a persistent job-management outage and exposes an infrastructure exception instead of a useful service response.

Expected behavior

OpenRAG should recover or reacquire TaskStateManager after an unexpected actor exit. During recovery, callers should receive a controlled temporary-unavailability response and operators should get a clear diagnostic. Recovery should be covered by a test that kills the actor and verifies that job APIs become usable again without restarting the whole deployment.

Job-state durability and retention remain tracked by #660. Ingestion backpressure and resource pressure remain tracked by #663.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions