Skip to content

feat(runtime): evaluate Memory write-gate promotion evidence #1930

Open

Description

@AlexStocks

Scope

Track the concrete promotion evidence and offline evaluation artifacts required before the RFC #1770 Memory write evidence-sufficiency policy moves from shadow or advisory to enforcing.

This is follow-up work to RFC #1770 and PR #1921. It does not require replay as a runtime feature for a consumer that remains in shadow or advisory, and it is not a commitment to promote the policy, enable a hosted backend, or add any other decision-policy consumer.

Problem

PR #1921 defines the Memory write policy, defaults new policies to shadow, and persists scope-private observation sidecars with structured references and bounded policy/action/fallback/privacy facts. That establishes an observation baseline without retaining raw candidate or evidence bodies.

The policy's declared promotion_criteria are not measured promotion evidence. The current implementation does not provide a reproducible replay/rescore or re-evaluation workflow, a labeled calibration set, separate held-out evaluation, or a documented promotion and rollback decision. Those missing artifacts mean the observations may inform a future decision but cannot justify enforcing; enabling it merely because configuration permits the mode would treat configuration as evidence.

Acceptance criteria

  • Define a reproducible, authorization-aware replay or re-evaluation workflow for the Memory write gate, reporting incomplete input coverage separately from negative results.
  • Provide a labeled calibration set and separate held-out evaluation, including English and Chinese cases where the gate accepts both.
  • Evaluate candidate outcome mappings and confidence thresholds against false holds, false passes, fallback reasons, latency, request cost, and hold/review friction.
  • Document the measured promotion decision and rollback procedure, or explicitly retain shadow/advisory when the evidence is insufficient.
  • Preserve immutable Memory revisions, exact citations, Scope isolation, and the invariant used_fallback=true => coverage=unadjudicated.

Non-goals

  • No automatic promotion to enforcing.
  • No dashboard, public HTTP/OpenAPI surface, export API, generic decision workflow engine, hosted-redacted transport, query rewrite, or Handoff/Experience/Skill/duplicate-conflict consumer.
  • No retention of raw candidate text, evidence text, model rationale, or exception details for convenience.

References

Activity

  1. lizining1231 commented on Oct 11, 2026

    @lizining1231

    I'd like to take this issue. Is it okay if I start working on it?

  2. AlexStocks commented on Oct 11, 2026

    @AlexStocks
    ContributorAuthor

    Two citation issues in this tracking issue, verified against the live repo (2026-10-11):

    1. Tracking issue Tracking Issue: evaluate and integrate Jev with PowerContext #1649 does not exist. The REST API returns 404 for issues/1649 and it is not a discussion either. If this refers to a deleted object the number should be replaced; if it is a typo, please correct the reference, since the acceptance criteria depend on which tracking items are actually outstanding.

    2. "Draft PR feat(runtime): add decision policy governance types #1921" is outdated. PR feat(runtime): add decision policy governance types #1921 is now ready for review (isDraft: false). The factual premises this issue relies on are verifiable in that PR: decision_observations.py implements the shadow / advisory / enforcing modes, and it enforces fallback => unadjudicated with a ValueError. Suggest updating the wording to "PR feat(runtime): add decision policy governance types #1921".

    Everything else in the issue's premises checked out: RFC #1770 is merged, the three promotion modes and the fallback invariant match the implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions