Improve hero-scenarios guidelines: spec fidelity, narrative consistency, convention matching#42641
Open
deyaaeldeen wants to merge 3 commits intomainfrom
Open
Improve hero-scenarios guidelines: spec fidelity, narrative consistency, convention matching#42641deyaaeldeen wants to merge 3 commits intomainfrom
deyaaeldeen wants to merge 3 commits intomainfrom
Conversation
…es to hero-scenarios guidelines Addresses three root causes of issues found during hero scenario review: 1. Spec Fidelity Rules: Example values must match spec types (uuid fields need real UUIDs, url fields need schemes, response shapes must match operation return types, no batching single-item APIs) 2. Narrative-Step Consistency: Every claim in the story/success criteria must have a corresponding API call in the steps 3. Matching Existing Document Conventions: When appending to an existing hero-scenarios.md, adopt its URL path style, ID format, and section structure instead of introducing inconsistencies Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
deyaaeldeen
requested review from
benbp,
mikeharder and
weshaggard
as code owners
Adds a mandatory verification gate between scenario generation (Step 3) and posting (Step 4). The agent must re-read the actual spec files and verify each HTTP call against the definitions — not rely on memory. Per-step checks: operation existence, request/response shape, property type validation (uuid/url/datetime/enum), no invented batching. Per-scenario checks: data flow continuity across steps, narrative completeness (every claim backed by an API call), existing document convention matching. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Next Steps to MergeNext steps that must be taken to merge this PR:
Comment generated by summarize-checks workflow run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During review of hero scenario suggestions (e.g., #42605), the reviewer caught 8 issues that trace back to 3 gaps in the agent guidelines:
No spec-schema validation — The agent invented plausible-looking IDs (
cde-...,col-...) instead of checking thatuuid-typed fields need real UUIDs. Similarly,url-typed fields got bare hostnames instead of full URLs with schemes.No narrative–step consistency check — The story said "tags both columns" but only one tagging API call appeared in the steps.
No instruction to match existing document conventions — The agent used a different URL path style than the existing scenarios in the same file.
Changes
Adds three new rule sections to
.github/prompts/hero-scenarios-guidelines.md:uuid→ valid UUID,url→ includes scheme, response shapes match operation return types, no batching single-item APIs.hero-scenarios.md, adopt its URL path style, ID format, and section structure.