Repository navigation
Refine AI test prompts to prefer high-value, non-duplicate coverage - #427
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e13bf0b2ba
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "If any program code changes, run unit-tests and follow up on the test-result (fix code if there is an error). If the tests use random values (for example random numbers or UUIDs), make them reproducible by fixing the random seed or replacing them with deterministic fixtures." | ||
| (concat | ||
| "If any program code changes, run unit-tests and follow up on the test-result (fix code if there is an error). " | ||
| ai-code--high-value-tests-instruction |
There was a problem hiding this comment.
Inline this autoloaded default text
Because this defcustom is marked with ;;;###autoload, MELPA/melpazoid-style package builds regenerate an autoload defvar from its initializer and load that autoload file before ai-code-agile has been required. Referencing the private ai-code--high-value-tests-instruction here therefore makes regenerated autoloads depend on an unbound symbol, so package activation can fail with void-variable even though the checked-in autoload was manually expanded; keep the default self-contained or autoload the constant first.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR refines the repository’s shared AI test-writing guidance to bias generated tests toward a small set of distinct, high-value behaviors (and away from low-value/duplicative coverage). It threads the same instruction through interactive TDD flows, the auto-test harness suffix, and the packaged prompt/snippet assets, with ERT assertions to prevent drift.
Changes:
- Introduces a reusable “high-value, non-duplicate tests” instruction constant in
ai-code-agile.eland reuses it in the harness/TDD suffix assembly. - Updates bundled prompt assets (
prompt/*.v1.md) and yasnippet prompt snippets to include the same guidance (with readability line breaks where applicable). - Adds regression tests to ensure assembled prompts, bundled prompts, and snippets all contain the new guidance.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
ai-code-agile.el |
Adds shared high-value test guidance constant and injects it into TDD prompt tail assembly. |
ai-code-harness.el |
Reuses the shared guidance in the test-after-change suffix so auto-test flows match TDD behavior. |
ai-code-autoloads.el |
Updates the checked-in autoload default suffix text to match the new guidance. |
prompt/test-after-change.v1.md |
Adds the high-value/non-duplicate guidance and splits into shorter lines for readability. |
prompt/test-after-change-diagnostics.v1.md |
Adds the high-value/non-duplicate guidance while preserving diagnostics instructions. |
prompt/tdd.v1.md |
Extends packaged TDD prompt with the high-value/non-duplicate test guidance. |
prompt/tdd-diagnostics.v1.md |
Extends packaged TDD+diagnostics prompt with the high-value/non-duplicate test guidance. |
prompt/tdd-with-refactoring.v1.md |
Extends packaged TDD+refactoring prompt with the high-value/non-duplicate test guidance. |
prompt/tdd-with-refactoring-diagnostics.v1.md |
Extends packaged TDD+refactoring+diagnostics prompt with the high-value/non-duplicate test guidance. |
snippets/ai-code-prompt-mode/unit-tests |
Updates unit-test snippet to prefer a small, distinct set of tests and avoid duplicates. |
snippets/ai-code-prompt-mode/create-tests |
Replaces “generate two tests” phrasing with high-value/non-duplicate guidance. |
test/test_ai-code-agile.el |
Asserts agile prompt assembly includes the new guidance. |
test/test_ai-code-harness.el |
Asserts harness-generated suffixes include the new guidance. |
test/test_ai-code-package-hygiene.el |
Adds regression checks to ensure autoload defaults and packaged prompt/snippet assets contain the new guidance. |
AI test-generation prompts were biased toward producing too many tests, including low-value overlap. This updates the shared test guidance so generated coverage stays focused on the smallest useful set of distinct behaviors.
Shared prompt rules
ai-code-agile.elthat tells AI to:ai-code-harness.elso send-time auto-test flows stay aligned with TDD/test-writing flows.Bundled prompt assets
prompt/to carry the same high-value/low-duplication guidance:test-after-change.v1.mdtest-after-change-diagnostics.v1.mdtdd.v1.mdtdd-diagnostics.v1.mdtdd-with-refactoring.v1.mdtdd-with-refactoring-diagnostics.v1.mdtest-after-changeprompt files into shorter lines for readability.Prompt snippets
snippets/ai-code-prompt-mode/so ad hoc test-generation requests no longer push quantity-first behavior.Generated/autoloaded defaults
ai-code-autoloads.eldefault text so the user-visible default suffix stays consistent with the source prompt logic.Regression coverage
Example of the new instruction shape: