Skip to content

Commit 466cc1b

Browse files
boshu2boshu2
committed
fix(validate): honest scoped PASS — name the disclosure homes, stop paying to delete caveats
A fresh-install smoke proved a contradiction: validate.py rejects any PASS whose not_checked is nonempty, silently rewriting the verdict to NOT_PROVEN, while SKILL.md, AGENTS.md Closeout, and docs/architecture/operating-loop.md all instruct disclosing checked and not_checked. An identical honest draft passed only after its scope caveats were deleted. Decision: PASS stays strict-empty (option b), and every doc surface now defines where honest scope limits go instead. not_checked has exactly one meaning — in-scope acceptance surface that went unverified — so a PASS has none by construction; a bounded proof of a criterion goes in criteria[].reason, a declared non-goal stays in the intent source (optionally restated as an evidence-backed boundary criterion), and residual risk goes in the report. The integrity finding now names those homes instead of failing silently, so the honest draft is told where to move a caveat rather than paid to delete it. Also fixes two verified SKILL.md defects and one error message: - checkout-relative `python3 skills/validate/scripts/...` paths are now install-agnostic ($SKILL_DIR = this SKILL.md's own directory). - the `manifest` subcommand was documented flagless; a Helper commands table now documents the real flags for all five subcommands. - store-verdict's criteria-field error names the allowed set {id, result, evidence_refs, reason} and the offending keys.
1 parent fd30523 commit 466cc1b

11 files changed

Lines changed: 409 additions & 32 deletions

File tree

‎AGENTS.md‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,12 @@ AgentOps work ownership.
133133

134134
Inspect the final subject, map acceptance to evidence, disclose `checked` and
135135
`not_checked`, and obtain one fresh validation result over the exact content.
136-
Include a verdict reference only when persistence was requested. Report residual
137-
risk plainly. Git status, pushing, merging, release, and rollback are handled
138-
by the caller's repository policy, outside semantic completion.
136+
`not_checked` names in-scope acceptance surface that went unverified, so a PASS
137+
has none by construction and any entry makes the result `NOT_PROVEN`. Scope
138+
limits are disclosed, never deleted: a bounded proof of a criterion belongs in
139+
that criterion's `reason`, a declared non-goal belongs to the intent source
140+
(optionally restated as an evidence-backed boundary criterion), and residual
141+
risk belongs in this report. Include a verdict reference only when persistence
142+
was requested. Report residual risk plainly. Git status, pushing, merging,
143+
release, and rollback are handled by the caller's repository policy, outside
144+
semantic completion.

‎docs/architecture/operating-loop.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,16 @@ the evidence, and judges every acceptance criterion.
8282
- Complete evidence satisfying every criterion, with nonempty checked scope and
8383
evidence references: `PASS`.
8484

85+
`not_checked` names in-scope acceptance surface that this validation did not
86+
verify. PASS asserts that the whole declared acceptance surface was verified,
87+
so a PASS carries no `not_checked` entries and any entry makes the result
88+
`NOT_PROVEN`. That strictness never rewards deleting an honest caveat, because
89+
each kind of scope limit has a home that survives inside a PASS: a bounded
90+
proof of a criterion goes in `criteria[].reason`, a declared non-goal stays in
91+
the intent source (optionally restated as an evidence-backed boundary
92+
criterion), and residual risk goes in the caller-facing report. The full table
93+
lives in `skills/validate/SKILL.md` under Scope disclosure.
94+
8595
The validation result records criterion results, findings, evidence references,
8696
checked and not-checked surfaces, identities, and freshness. It carries no
8797
WARN, confidence, disposition, learning, owner, next action, retry, closure,

‎images/gemini/skills/validate/SKILL.md‎

Lines changed: 52 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -77,10 +77,53 @@ fresh context produced the PASS (`e9b6cdb8...37b9`). If a mutating check is
7777
genuinely required by acceptance, run it against a disposable copy or a
7878
committed subject, never the judged working tree.
7979

80+
## Scope disclosure
81+
82+
`not_checked` has exactly one meaning: **in-scope acceptance surface this
83+
validation did not verify**. PASS asserts that the whole declared acceptance
84+
surface was verified, so a PASS carries no `not_checked` entries; the helper
85+
refuses one and records a `validate.integrity` finding.
86+
87+
That rule never pays for deleting an honest caveat, because every kind of scope
88+
limit has a home that survives inside a PASS:
89+
90+
| Scope limit | Home | Example |
91+
|---|---|---|
92+
| A criterion proven by a bounded check | `criteria[].reason` on that criterion | "proven by the unit suite; the full integration matrix was not replayed" |
93+
| A declared non-goal or out-of-scope area | the intent source's non-goals, optionally restated as an evidence-backed boundary criterion in `criteria` | "`cli/**` is a declared non-goal; the diff proves it untouched" |
94+
| Residual risk or judgment caveat | the caller-facing report | "the migration path is untested against pre-3.0 stores" |
95+
| Acceptance that genuinely went unverified | `not_checked`, and the result is `NOT_PROVEN` rather than PASS | "criterion 3 needs hardware this context cannot reach" |
96+
97+
Emptying `not_checked` to obtain PASS is a contract violation, not a
98+
workaround. If acceptance really went unverified, the honest result is
99+
`NOT_PROVEN`. If the entry was never acceptance in the first place, it belongs
100+
in one of the other homes, where it stays visible in the stored artifact
101+
instead of being deleted.
102+
103+
## Helper commands
104+
105+
The helper ships beside this file. Invoke it through this skill's own
106+
directory rather than a checkout-relative path: `$SKILL_DIR` is the directory
107+
containing this `SKILL.md` — `skills/validate/` in a repository checkout,
108+
`.agents/skills/validate/` in an installed runtime.
109+
110+
| Command | Required | Optional |
111+
|---|---|---|
112+
| `manifest` | `--root <dir>`, `--include <path>` (repeatable, at least one) | `--exclude <path-or-glob>` (repeatable), `--base-manifest <file>`, `--git-metadata-json <json>`, `--output <file>` |
113+
| `verify-manifest` | `--root <dir>`, `--manifest <file>` | `--base-manifest <file>` |
114+
| `snapshot-intent` | `--source <file>` (`-` reads stdin) | `--workspace <dir>`, `--intent-dir <dir>` |
115+
| `digest` | `<json-file>` positional | none |
116+
| `store-verdict` | `--draft`, `--intent-source`, `--subject-manifest`, `--author-context-id`, `--validator-context-id`, `--freshness-source <runtime\|caller>`, `--freshness-attester-id`, `--scope-result <PASS\|FAIL\|NOT_PROVEN>` | `--workspace <dir>`, `--verdict-dir <dir>` |
117+
118+
```sh
119+
python3 "$SKILL_DIR/scripts/validate.py" manifest \
120+
--root . --include skills/validate --exclude '**/*.log' --output manifest.json
121+
```
122+
80123
## Workflow
81124

82-
1. Recompute and compare `subject-manifest.v1` using
83-
`python3 skills/validate/scripts/validate.py manifest`. The helper uses only
125+
1. Recompute and compare `subject-manifest.v1` with the `manifest` command
126+
above (`--root` plus at least one `--include`). The helper uses only
84127
filesystem content; Git commit/tree IDs are optional metadata. Derive the
85128
manifest at the start of validation and re-derive it at the end; any
86129
mismatch between the two is subject mutation and returns `NOT_PROVEN`.
@@ -95,15 +138,19 @@ committed subject, never the judged working tree.
95138
claims, not evidence: re-execute the claimed proofs that bear on acceptance
96139
(see the freshness rules below for when a digest-bound receipt suffices).
97140
Judge every acceptance criterion and record criterion-level results,
98-
findings, evidence references, `checked`, and `not_checked`.
141+
findings, evidence references, `checked`, and any acceptance surface that
142+
went unverified in `not_checked` (see Scope disclosure).
99143
5. Choose exactly one semantic result: `PASS`, `FAIL`, or `NOT_PROVEN`. Return
100144
it with criterion results, findings, evidence references, `checked`,
101145
`not_checked`, the acceptance and subject identities, distinct author and
102146
validator context IDs, and the freshness attestation. PASS requires distinct
103147
identities, explicit freshness, nonempty checked scope, top-level evidence,
104-
and evidence for every criterion.
148+
evidence for every criterion, and an empty `not_checked`; route bounded
149+
proofs, declared non-goals, and residual risk to the homes named in Scope
150+
disclosure rather than deleting them or downgrading a proven result.
105151
6. Only when the caller requests machine-readable evidence or a declared
106-
downstream consumer requires it, persist canonical `verdict.v2` with
152+
downstream consumer requires it, persist canonical `verdict.v2` with the
153+
helper's
107154
`store-verdict --draft <draft.json> --intent-source <resolved-intent>
108155
--subject-manifest <manifest.json> --author-context-id <id>
109156
--validator-context-id <id> --freshness-source <runtime|caller>

‎skills-codex/.agentops-manifest.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -639,8 +639,8 @@
639639
{
640640
"name": "validate",
641641
"source_skill": "skills/validate",
642-
"source_hash": "3f8fd13efb33dabd114dbb66deb0eea5d3762de888b081c753a27c1fba557e03",
643-
"generated_hash": "93b79b850fe0c6581305e1357ad2e83923d3a83cd3d3c23be342e70c9e0902ca"
642+
"source_hash": "4cad5b18f30ecba7944b1502624e38bb75fc329666464c614fc9051d4d3bd10c",
643+
"generated_hash": "1938a59144b5aaf9de48e0c3e03e00750107b60904d517742dfcedf63d011304"
644644
},
645645
{
646646
"name": "workflow-builder",

‎skills-codex/validate/.agentops-generated.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
"generator": "codex-sync",
33
"source_skill": "skills/validate",
44
"layout": "modular",
5-
"source_hash": "3f8fd13efb33dabd114dbb66deb0eea5d3762de888b081c753a27c1fba557e03",
6-
"generated_hash": "93b79b850fe0c6581305e1357ad2e83923d3a83cd3d3c23be342e70c9e0902ca"
5+
"source_hash": "4cad5b18f30ecba7944b1502624e38bb75fc329666464c614fc9051d4d3bd10c",
6+
"generated_hash": "1938a59144b5aaf9de48e0c3e03e00750107b60904d517742dfcedf63d011304"
77
}

‎skills-codex/validate/SKILL.md‎

Lines changed: 52 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -49,10 +49,53 @@ fresh context produced the PASS (`e9b6cdb8...37b9`). If a mutating check is
4949
genuinely required by acceptance, run it against a disposable copy or a
5050
committed subject, never the judged working tree.
5151

52+
## Scope disclosure
53+
54+
`not_checked` has exactly one meaning: **in-scope acceptance surface this
55+
validation did not verify**. PASS asserts that the whole declared acceptance
56+
surface was verified, so a PASS carries no `not_checked` entries; the helper
57+
refuses one and records a `validate.integrity` finding.
58+
59+
That rule never pays for deleting an honest caveat, because every kind of scope
60+
limit has a home that survives inside a PASS:
61+
62+
| Scope limit | Home | Example |
63+
|---|---|---|
64+
| A criterion proven by a bounded check | `criteria[].reason` on that criterion | "proven by the unit suite; the full integration matrix was not replayed" |
65+
| A declared non-goal or out-of-scope area | the intent source's non-goals, optionally restated as an evidence-backed boundary criterion in `criteria` | "`cli/**` is a declared non-goal; the diff proves it untouched" |
66+
| Residual risk or judgment caveat | the caller-facing report | "the migration path is untested against pre-3.0 stores" |
67+
| Acceptance that genuinely went unverified | `not_checked`, and the result is `NOT_PROVEN` rather than PASS | "criterion 3 needs hardware this context cannot reach" |
68+
69+
Emptying `not_checked` to obtain PASS is a contract violation, not a
70+
workaround. If acceptance really went unverified, the honest result is
71+
`NOT_PROVEN`. If the entry was never acceptance in the first place, it belongs
72+
in one of the other homes, where it stays visible in the stored artifact
73+
instead of being deleted.
74+
75+
## Helper commands
76+
77+
The helper ships beside this file. Invoke it through this skill's own
78+
directory rather than a checkout-relative path: `$SKILL_DIR` is the directory
79+
containing this `SKILL.md` — `skills/validate/` in a repository checkout,
80+
`.agents/skills/validate/` in an installed runtime.
81+
82+
| Command | Required | Optional |
83+
|---|---|---|
84+
| `manifest` | `--root <dir>`, `--include <path>` (repeatable, at least one) | `--exclude <path-or-glob>` (repeatable), `--base-manifest <file>`, `--git-metadata-json <json>`, `--output <file>` |
85+
| `verify-manifest` | `--root <dir>`, `--manifest <file>` | `--base-manifest <file>` |
86+
| `snapshot-intent` | `--source <file>` (`-` reads stdin) | `--workspace <dir>`, `--intent-dir <dir>` |
87+
| `digest` | `<json-file>` positional | none |
88+
| `store-verdict` | `--draft`, `--intent-source`, `--subject-manifest`, `--author-context-id`, `--validator-context-id`, `--freshness-source <runtime\|caller>`, `--freshness-attester-id`, `--scope-result <PASS\|FAIL\|NOT_PROVEN>` | `--workspace <dir>`, `--verdict-dir <dir>` |
89+
90+
```sh
91+
python3 "$SKILL_DIR/scripts/validate.py" manifest \
92+
--root . --include skills/validate --exclude '**/*.log' --output manifest.json
93+
```
94+
5295
## Workflow
5396

54-
1. Recompute and compare `subject-manifest.v1` using
55-
`python3 skills/validate/scripts/validate.py manifest`. The helper uses only
97+
1. Recompute and compare `subject-manifest.v1` with the `manifest` command
98+
above (`--root` plus at least one `--include`). The helper uses only
5699
filesystem content; Git commit/tree IDs are optional metadata. Derive the
57100
manifest at the start of validation and re-derive it at the end; any
58101
mismatch between the two is subject mutation and returns `NOT_PROVEN`.
@@ -67,15 +110,19 @@ committed subject, never the judged working tree.
67110
claims, not evidence: re-execute the claimed proofs that bear on acceptance
68111
(see the freshness rules below for when a digest-bound receipt suffices).
69112
Judge every acceptance criterion and record criterion-level results,
70-
findings, evidence references, `checked`, and `not_checked`.
113+
findings, evidence references, `checked`, and any acceptance surface that
114+
went unverified in `not_checked` (see Scope disclosure).
71115
5. Choose exactly one semantic result: `PASS`, `FAIL`, or `NOT_PROVEN`. Return
72116
it with criterion results, findings, evidence references, `checked`,
73117
`not_checked`, the acceptance and subject identities, distinct author and
74118
validator context IDs, and the freshness attestation. PASS requires distinct
75119
identities, explicit freshness, nonempty checked scope, top-level evidence,
76-
and evidence for every criterion.
120+
evidence for every criterion, and an empty `not_checked`; route bounded
121+
proofs, declared non-goals, and residual risk to the homes named in Scope
122+
disclosure rather than deleting them or downgrading a proven result.
77123
6. Only when the caller requests machine-readable evidence or a declared
78-
downstream consumer requires it, persist canonical `verdict.v2` with
124+
downstream consumer requires it, persist canonical `verdict.v2` with the
125+
helper's
79126
`store-verdict --draft <draft.json> --intent-source <resolved-intent>
80127
--subject-manifest <manifest.json> --author-context-id <id>
81128
--validator-context-id <id> --freshness-source <runtime|caller>

‎skills-codex/validate/scripts/test_validate.py‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,68 @@ def test_runtime_facts_override_model_authored_digests(self):
278278
self.assertEqual(artifact["subject_manifest_digest"], manifest["canonical_manifest_digest"])
279279
self.assertEqual(artifact["verdict"], "PASS")
280280

281+
def test_honest_scoped_pass_round_trips_through_documented_homes(self):
282+
"""An honest draft with declared non-goals is representable as PASS.
283+
284+
Both drafts below carry the same honest content. Draft A parks the
285+
declared non-goals in ``not_checked``, which is reserved for unverified
286+
in-scope acceptance: the result is NOT_PROVEN and the finding names
287+
where each caveat belongs. Draft B moves the same caveats into the
288+
documented homes and stores PASS with every caveat still readable in
289+
the persisted artifact. Nothing is deleted to earn the PASS.
290+
"""
291+
bounded = "proven by the unit suite; the full integration matrix was not replayed"
292+
boundary = "declared non-goal; the diff proves cli/** untouched"
293+
294+
with tempfile.TemporaryDirectory() as raw:
295+
draft_a = self.draft()
296+
draft_a["not_checked"] = [
297+
"cli/** (declared non-goal)",
298+
"Windows runners (declared non-goal)",
299+
]
300+
artifact_a, _path, _existed = self.store_bound(draft_a, Path(raw))
301+
self.assertEqual(artifact_a["verdict"], "NOT_PROVEN")
302+
summary = artifact_a["findings"][-1]["summary"]
303+
self.assertIn("PASS cannot contain not_checked items", summary)
304+
for home in ("criteria[].reason", "non-goal", "report"):
305+
self.assertIn(home, summary)
306+
self.assert_schema_valid(artifact_a)
307+
308+
with tempfile.TemporaryDirectory() as raw:
309+
draft_b = self.draft()
310+
draft_b["criteria"][0]["reason"] = bounded
311+
draft_b["criteria"].append(
312+
{
313+
"id": "non-goal:cli-untouched",
314+
"result": "PASS",
315+
"evidence_refs": ["git-diff:cli"],
316+
"reason": boundary,
317+
}
318+
)
319+
draft_b["evidence_refs"] = ["e1", "git-diff:cli"]
320+
draft_b["not_checked"] = []
321+
artifact_b, path, _existed = self.store_bound(draft_b, Path(raw))
322+
self.assertEqual(artifact_b["verdict"], "PASS")
323+
self.assert_schema_valid(artifact_b)
324+
# Round-trip: the caveats survive in the persisted PASS artifact.
325+
stored = json.loads(path.read_text(encoding="utf-8"))
326+
reasons = [criterion.get("reason") for criterion in stored["criteria"]]
327+
self.assertIn(bounded, reasons)
328+
self.assertIn(boundary, reasons)
329+
self.assertEqual(stored["not_checked"], [])
330+
self.assertEqual(stored["verdict"], "PASS")
331+
332+
def test_criteria_field_error_names_the_allowed_set(self):
333+
with tempfile.TemporaryDirectory() as raw:
334+
draft = self.draft()
335+
draft["criteria"][0]["confidence"] = "high"
336+
with self.assertRaisesRegex(
337+
tool.ContractError,
338+
r"unknown confidence.*allowed fields are \{id, result, evidence_refs, reason\}",
339+
):
340+
self.store_bound(draft, Path(raw))
341+
self.assertEqual(list(Path(raw).iterdir()), [])
342+
281343
def test_runtime_scope_failure_forces_fail(self):
282344
with tempfile.TemporaryDirectory() as raw:
283345
artifact, _path, _existed = self.store_bound(self.draft(), Path(raw), scope="FAIL")

0 commit comments

Comments
 (0)