Agent Evidence Graph

Verify agent releases from runtime evidence, not declarations.

English evidence report · 中文证据报告 · 2:10 live-device demo · Submission release · Judge replay instructions

The submission video shows the project functioning locally against a real DataHub Quickstart. It contains disclosed AI-generated neural narration and synchronized, burned-in English captions; no music is used. Build and QA details are documented in demo/README.md.

Agent Evidence Graph is a hackathon prototype that asks one narrow, useful question:

Does this exact agent version still behave the way its governed claim says it does?

It is built for AI platform governance leads and agent release approvers deciding whether a new agent version can enter production. The verified v1.3 result is NEEDS_REVIEW: two deterministic HIGH findings, one implicated governed production path, and two of two governance writeback operations independently re-read and verified.

The contradiction: this agent claims to be read-only and canonical-only. In a real v1.3 run, it writes DataHub metadata and queries a deprecated revenue table. A catalog of declarations alone cannot see that runtime gap.

The governed response: this project joins a versioned claim, automatically captured tool traces, DataHub dataset status/ownership/schema/lineage, and task evals. Two deterministic detectors emit evidence-linked findings, then an explicit human approval controls a NeedsReview tag and audit-document writeback. The safety decision is deterministic by design; an optional LLM may select tools or explain facts, but cannot grade its own risk.

Current verification status

The offline compatibility loop and the live DataHub Core v1.6.0 loop both pass. The live gate re-reads five datasets, native deprecation and replacement metadata, three lineage edges, both version-scoped traces, the v1.3 runtime tag, and the human-approved NeedsReview tag/audit document. Every artifact carries source_kind: real | synthetic | fixture.

49 tests passed (real-GMS connection test executed)
81.57% branch-aware core coverage
Ruff: pass | strict mypy: pass
Claim-only eval: 12.5% | Evidence-aware eval: 100% | unsupported claims: 0%
uv run agent-evidence verify --require-real --require-eligible-integration: pass

The release snapshots use run IDs run-revenue-analyst-1.2-65883e4d49fe and run-revenue-analyst-1.3-74cc49489dc3. Both use official Agent Context Kit LangChain tools for governed DataHub reads; the v1.3 trace separately records its approved demo mutation through the SDK. See examples/real-traces and examples/release-manifest.json.

The explicitly approved real reset/reseed rehearsal is complete. It soft-deleted only the exact demo targets, restored the protected state, reactivated the seven seeded entities, replayed the approved writeback, and passed verify --require-real again.

Core demo

One Revenue Analyst Agent has two declarations:

Version	Declared	Observed plan	Deterministic result
v1.2	Read-only; canonical revenue	Read DataHub context, lineage, canonical DuckDB table	No HIGH finding
v1.3	Still read-only; canonical only	Read deprecated legacy revenue and invoke an undeclared tag-write tool	Two HIGH findings; NEEDS_REVIEW

The detector never checks for the string 1.3. It compares declared sets and flags with the tool wrapper’s actual events and DataHub metadata returned through the adapter.

Architecture

flowchart LR
    C["Versioned Agent Claim"] --> E["Evidence Resolver"]
    D["DataHub Context Graph"] --> E
    R["Automatic Runtime Trace"] --> E
    T["Task Eval Evidence"] --> E
    E --> P["Permission / Tool Drift"]
    E --> X["Deprecated Dataset Access"]
    P --> A["Structured Evidence Answer"]
    X --> A
    A --> H["Human Approval"]
    H --> W["Allowlisted DataHub Writeback"]
    W --> D

Loading

Domain logic is outside Streamlit: adapters, runtime tools, traces, evidence, detectors, approvals, and evals are separately typed and tested.

Why DataHub is indispensable here

Runtime logs can show that a table was touched, but not whether that table is deprecated, who owns it, what its schema means, or what downstream report it affects. DataHub supplies the governed context graph that turns a tool call into an impact-aware risk finding. In the dataset detector, deprecated access begins at MEDIUM and becomes HIGH only when real DataHub lineage shows that the governed replacement has an active downstream consumer. Removing that edge removes the escalation. The prototype uses official Agent Context Kit search, get_entities, and get_lineage tools inside the real agent loop. The official Python SDK supplies the allowlisted mutation and post-write verification path. Each trace event records the integration surface, tool name, tool version, source kind, and canonical hash.

Compatibility note

The current Agent Registry documentation describes a DataHub Cloud v2.1 gated feature, and acryl-datahub==1.6.0.6 installed here does not expose its documented Python entity classes. P0 therefore maps each agent version to a compatibility DataFlow entity while keeping the complete claim/trace contract in this repository. It never pretends the native UI exists. See docs/datahub-compatibility.md.

Quickstart

Prerequisites: Python 3.11, uv, and Docker Desktop/Compose v2 for real DataHub. The official Quickstart recommends at least 2 CPUs, 8 GiB RAM, 2 GiB swap, and 13 GiB disk.

uv sync
uv run agent-evidence doctor
uv run agent-evidence datahub probe

Start local DataHub (Quickstart is for local development, not production):

uv run datahub docker quickstart --version v1.6.0
uv run datahub init --username datahub --password datahub
uv run agent-evidence doctor

Seed first prints a mutation plan. The second command is the explicit authorization:

uv run agent-evidence datahub seed
uv run agent-evidence datahub seed --approve

Run the real scripted demo sequentially:

uv run agent-evidence demo run --version 1.2 --mode scripted
uv run agent-evidence demo run --version 1.3 --mode scripted --approve-mutation
uv run agent-evidence detect --agent revenue-analyst --version 1.3
uv run agent-evidence ask --agent revenue-analyst --version 1.3
uv run agent-evidence writeback prepare --agent revenue-analyst --version 1.3
uv run agent-evidence writeback approve <approval-id> --approved-by <name>
uv run agent-evidence verify --require-real --require-eligible-integration

For CI or a machine without DataHub, the offline path is intentionally labeled fixture:

uv run agent-evidence demo run --version 1.2 --mode scripted --fixture
uv run agent-evidence demo run --version 1.3 --mode scripted --fixture
uv run agent-evidence verify

Run the UI:

uv run streamlit run src/agent_evidence_graph/ui/app.py

For a mutation-disabled judge replay of the hashed release snapshot:

uv run agent-evidence release-manifest
uv run agent-evidence submission-check
uv run agent-evidence judge-demo --check-only
uv run agent-evidence judge-demo

The Hero links and readiness verdict are manifest-owned. Set AEG_PUBLIC_REPOSITORY_URL, AEG_DEMO_VIDEO_URL, and AEG_JUDGE_REPLAY_URL before generating the release manifest. AEG_CLEAN_ROOM_RELEASE_VERIFIED=true may be set only after the clean-room rehearsal; set AEG_ANONYMOUS_URLS_VERIFIED=true only after all three public URLs work without a signed-in session. Then run uv run agent-evidence submission-check --final-release. A fresh environment defaults both verification flags to false.

Deterministic policy, optional LLM

Risk adjudication must be reproducible and auditable. --mode scripted uses the same registry and trace wrapper as any optional agent runner, while every severity and top-level recommendation remains a tested deterministic policy decision. A future provider-backed mode is intentionally limited to tool selection and evidence-grounded explanation.

An llm dependency group is declared for future/provider-enabled tool selection:

uv sync --extra llm

The current CLI deliberately rejects --mode llm instead of silently simulating a model call. The verified release therefore proves governed work through real DataHub tool calls, an executed runtime mutation, and a human-approved writeback; it does not use a model to manufacture a risk decision. No provider credential or local Ollama runtime was available for this release, so model-backed tool selection is disclosed as unrun.

Evidence semantics

• real: produced by an actual DataHub response or actual versioned run against it.
• synthetic: generated eval scenario.
• fixture: deterministic offline test or sample adapter.
• No current-version runtime evidence → UNVERIFIED.
• Metadata fetch failure → UNKNOWN/not recommendable, never “safe.”
• An LLM may explain normalized facts in a future mode, but it may not decide severity or the top-level recommendation.

Each evidence item contains an ID, source reference, UTC observation time, version, fact, source kind, and SHA-256 hash of canonical serialized input.

Evals

The most important result is the baseline: a declaration-only system gets just 1/8 expected statuses because it recommends seven cases it cannot actually verify. uv run agent-evidence eval compares that baseline with the evidence-aware rule engine across eight fixed cases: healthy canonical access, hidden write, deprecated data, missing runtime, previous-version evidence, metadata failure, undeclared read tool, and synthetic-only evidence. Three cases intentionally return UNVERIFIED, demonstrating that the engine abstains when evidence is absent, stale-version, or synthetic-only.

System	Accuracy	Unsupported claim rate
Claim-only	12.5%	0% (typed output)
Evidence-aware	100%	0%

These are deterministic prototype cases, not a claim of production performance.

Safety and limitations

• DataHub mutations are off by default and restricted to exact demo entities/tags.
• writeback prepare is a dry-run; approve <id> is the authorization boundary.
• Writeback re-reads both the tag and audit document and records verification results.
• Tokens, API keys, passwords, bearer headers, and full query results are not traced.
• This demonstrates two high-value drift checks; it does not prevent every unsafe action, provide complete enterprise governance, or guarantee trust.
• Quickstart credentials are local defaults only. Do not reuse them in production.
• The checked-in real artifacts are reviewed snapshots; use verify --require-real for current-instance proof.

Reset

Reset is also dry-run by default and lists exact targets:

uv run agent-evidence demo reset
uv run agent-evidence demo reset --approve
uv run agent-evidence demo reset --real --approve

The real path soft-deletes only five demo datasets, two compatibility DataFlows, and allowlisted audit documents. It does not delete broad directories or shared generic tags.

Development

uv run ruff check .
uv run mypy src
uv run pytest
uv run agent-evidence eval
uv run agent-evidence verify
uv run agent-evidence verify --require-real
uv run agent-evidence verify --require-real --require-eligible-integration
uv run agent-evidence submission-check

See docs/architecture.md, docs/evidence-semantics.md, docs/threat-model.md, and docs/limitations.md. The proposal is available locally in docs/rfcs/agent-execution-evidence.md and has been submitted as DataHub RFC PR #18966. It is under upstream review and is not presented as accepted or implemented.

Roadmap (after the P0 real-integration gate)

• Native Agent Registry adapter when the target DataHub deployment supports it.
• Provider-enabled LLM tool selection with schema-constrained explanation only.
• A2A importer and broader detector library.
• Signed trace transport, retention controls, and multi-agent fleet views.

Licensed under Apache License 2.0. See LICENSE.

DataHub catalogs what an agent is supposed to be. Agent Evidence Graph verifies what that version actually did.