Skip to content

Commit cea0c4b

Browse files
myakovemyakove
committed
feat: add /qualify AI qualification workflow
Full qualification workflow: test plan → write tests → verify on cluster → PR with proof. Components: - /qualify prompt template (orchestrator) - test-planner agent (reads feature/bug docs → test plans) - cluster-verifier agent (independent cluster state verification) - proof-generator skill (assembles proof.md reports) - Templates for test plans and proof reports
1 parent ab0bfd6 commit cea0c4b

14 files changed

Lines changed: 1214 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -174,3 +174,6 @@ jira.cfg
174174
# ENV
175175
.env
176176
.envrc
177+
178+
# Qualify workflow
179+
.qualify/

‎CLAUDE.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -354,6 +354,8 @@ AI must NEVER run tests directly (`pytest`, `uv run pytest`). Tests require live
354354
AI can: Read/analyze/write/fix tests, suggest improvements, review structure
355355
AI cannot: Execute tests, validate by running
356356

357+
**Exception:** The `/qualify` workflow (`llm/qualify/`) may run pytest on a real cluster when a user explicitly invokes it with cluster credentials. See `llm/qualify/README.md`.
358+
357359
### No Module-Level Provider Loading (MUST)
358360

359361
`load_source_providers()` must only be called within the pytest ecosystem (fixtures, hooks).

‎README.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -985,3 +985,12 @@ uv run pytest -v \
985985

986986
# For debug options (--skip-teardown, -s -vv, etc.), see "Useful Test Options" section above
987987
```
988+
989+
---
990+
991+
## AI Qualification Workflow (Optional)
992+
993+
The `/qualify` command provides an AI-driven qualification workflow:
994+
test plan → write tests → verify on cluster → PR with proof.
995+
996+
See [llm/qualify/README.md](llm/qualify/README.md) for setup and usage.

‎llm/qualify/README.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# /qualify — AI Qualification Workflow
2+
3+
Full qualification workflow for MTV API tests: from feature design or bug report to verified PR with proof.
4+
5+
## What It Does
6+
7+
```bash
8+
/qualify --type feature --source <url> --cluster ~/kubeconfig
9+
```
10+
11+
1. **Test Plan** — AI reads feature/bug docs → produces a test plan → human reviews
12+
2. **Write Tests** — AI writes E2E customer use-case tests following project patterns
13+
3. **Verify on Cluster** — AI runs tests on a real cluster AND independently verifies cluster state
14+
4. **Code Review** — AI reviewers check the code (on pi with myk-org/pi-config: 3 parallel reviewers; elsewhere per project `AGENTS.md` / `CLAUDE.md`)
15+
5. **PR with Proof** — Creates PR with proof.md documenting test results + cluster evidence + versions
16+
17+
### Outputs
18+
19+
| Artifact | Location |
20+
| --------------- | ---------------------------------------------------------------------------------- |
21+
| Test plan | `.qualify/features/<name>/test-plan.md` or `.qualify/bugs/<id>/test-plan.md` |
22+
| Proof report | `.qualify/features/<name>/proof.md` or `.qualify/bugs/<id>/proof.md` |
23+
| Test output log | `.qualify/features/<name>/test-output.log` or `.qualify/bugs/<id>/test-output.log` |
24+
| PR | GitHub (features and bugs with permanent tests) |
25+
26+
## Arguments
27+
28+
| Argument | Required | Description |
29+
| ----------- | -------- | ---------------------------------------------------------------------- |
30+
| `--type` | Yes | `feature` or `bug` |
31+
| `--source` | Yes | URL to Jira ticket, GitHub issue, design doc, or local file path |
32+
| `--cluster` | No | Path to kubeconfig. If omitted, uses current `oc` context |
33+
| `--name` | No | Short identifier (e.g., `warm-migration-rhv`). Auto-derived if omitted |
34+
35+
## Usage Examples
36+
37+
### Qualify a New Feature
38+
39+
```bash
40+
/qualify --type feature --source https://issues.redhat.com/browse/MTV-1234 --cluster ~/kubeconfigs/test-cluster
41+
```
42+
43+
### Verify a Bug Fix
44+
45+
```bash
46+
/qualify --type bug --source https://issues.redhat.com/browse/MTV-5678 --name MTV-5678
47+
```
48+
49+
The AI will ask: "Should this bug get a permanent test in the test suite?"
50+
51+
- **Yes** → full flow: test plan → write test → PR → proof
52+
- **No** → verify-only: test plan → run throwaway test → proof.md (no PR)
53+
54+
## Human Checkpoints
55+
56+
The workflow is fully automated EXCEPT at these points:
57+
58+
| Checkpoint | When | What |
59+
| ------------------ | --------------------------- | ----------------------------------------- |
60+
| Test plan review | After Phase 1 | Approve or give feedback on the test plan |
61+
| Bug: suite or not? | Start of bug workflow | Decide if test joins permanent suite |
62+
| AI stuck | When AI can't make progress | Guide the AI on how to proceed |
63+
| PR review | After Phase 3 | Normal GitHub PR review |
64+
65+
## Setup by AI CLI
66+
67+
### pi
68+
69+
1. Add to `.pi/settings.json`:
70+
71+
```json
72+
{
73+
"prompts": ["llm/qualify/prompts"],
74+
"skills": ["llm/qualify/skills"]
75+
}
76+
```
77+
78+
2. Register agents — add to your pi-config or project agents:
79+
80+
```json
81+
{
82+
"agents": ["llm/qualify/agents"]
83+
}
84+
```
85+
86+
3. Use: type `/qualify` in pi's interactive mode.
87+
88+
### Claude Code
89+
90+
1. Copy or symlink the prompt template:
91+
92+
```bash
93+
mkdir -p .claude/commands
94+
cp llm/qualify/prompts/qualify.md .claude/commands/qualify.md
95+
```
96+
97+
2. Reference agents and skills in `CLAUDE.md`:
98+
99+
```markdown
100+
## Qualification Workflow
101+
See `llm/qualify/` for the /qualify workflow:
102+
- Agents: `llm/qualify/agents/`
103+
- Skills: `llm/qualify/skills/`
104+
- Templates: `llm/qualify/templates/`
105+
```
106+
107+
3. Use: type `/qualify` in Claude Code.
108+
109+
### Cursor
110+
111+
1. Add as a Notepad or Rule:
112+
- Copy content from `llm/qualify/prompts/qualify.md` into a Cursor Rule
113+
- Reference agent/skill files in the rule
114+
115+
2. Or use `.cursorrules` to reference the qualify workflow.
116+
117+
### Other AI CLIs
118+
119+
The workflow is plain Markdown — adapt to any AI CLI that supports:
120+
121+
- Prompt templates or system prompts
122+
- Agent/persona definitions
123+
- Tool access (file read/write, bash execution, web fetching)
124+
125+
Copy the relevant `.md` files into your CLI's configuration format.
126+
127+
## Directory Structure
128+
129+
```text
130+
llm/qualify/
131+
├── README.md # This file
132+
├── prompts/
133+
│ └── qualify.md # Main prompt template (/qualify command)
134+
├── agents/
135+
│ ├── test-planner.md # Reads docs → produces test plans
136+
│ └── cluster-verifier.md # Independently verifies cluster state
137+
├── skills/
138+
│ └── proof-generator/
139+
│ └── SKILL.md # Assembles proof.md reports
140+
└── templates/
141+
├── test-plan-template.md # Test plan skeleton
142+
└── proof-template.md # Proof report skeleton
143+
```
144+
145+
Output (gitignored):
146+
147+
```text
148+
.qualify/
149+
├── features/
150+
│ └── <name>/
151+
│ ├── test-plan.md
152+
│ ├── test-output.log
153+
│ └── proof.md
154+
└── bugs/
155+
└── <name>/
156+
├── test-plan.md
157+
├── test-output.log
158+
└── proof.md
159+
```
160+
161+
## Requirements
162+
163+
- `oc` CLI configured and authenticated to a working OpenShift cluster
164+
- MTV operator installed on the cluster
165+
- CNV installed on the cluster
166+
- Source provider configured (VMware, RHV, etc.) with test VMs available
167+
- `.providers.json` configured in the repo

‎llm/qualify/agents/.gitkeep‎

Whitespace-only changes.

0 commit comments

Comments
 (0)