-
Notifications
You must be signed in to change notification settings - Fork 11
feat: add /qualify AI qualification workflow #474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
myakove
wants to merge
13
commits into
main
Choose a base branch
from
feature/qualify-workflow
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
886a402
feat: add /qualify AI qualification workflow
myakove bcbca3c
docs: add workflow diagrams for /qualify qualification workflow
myakove 7e09553
docs: split workflow diagrams into per-phase charts for readability
myakove 50a78df
fix: address CodeRabbit review comments
myakove 7dd1bd2
fix: add version collection failure handling per CodeRabbit review
myakove 0c71048
fix: address 10 CodeRabbit review comments
myakove 07d542d
fix: address 6 CodeRabbit review comments
myakove b685f33
fix: align version collection with error capture, add tool name mappi…
myakove 9fa43d9
Add bug ID extraction/validation in qualify workflow
myakove d1b1a18
Improve qualify.md version collection and path consistency
myakove d55f794
fix: address review findings for qualify workflow
myakove 451f5d7
fix: address Qodo cycle 2 review findings
myakove cc7886e
fix: clarify error message redaction in cluster-verifier
myakove File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -174,3 +174,6 @@ jira.cfg | |
| # ENV | ||
| .env | ||
| .envrc | ||
|
|
||
| # Qualify workflow | ||
| .qualify/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,167 @@ | ||
| # /qualify — AI Qualification Workflow | ||
|
|
||
| Full qualification workflow for MTV API tests: from feature design or bug report to verified PR with proof. | ||
|
|
||
| ## What It Does | ||
|
|
||
| ```bash | ||
| /qualify --type feature --source <url> --cluster ~/kubeconfig | ||
| ``` | ||
|
|
||
| 1. **Test Plan** — AI reads feature/bug docs → produces a test plan → human reviews | ||
| 2. **Write Tests** — AI writes E2E customer use-case tests following project patterns | ||
| 3. **Verify on Cluster** — AI runs tests on a real cluster AND independently verifies cluster state | ||
| 4. **Code Review** — AI reviewers check the code (on pi with myk-org/pi-config: 3 parallel reviewers; elsewhere per project `AGENTS.md` / `CLAUDE.md`) | ||
| 5. **PR with Proof** — Creates PR with proof.md documenting test results + cluster evidence + versions | ||
|
|
||
| ### Outputs | ||
|
|
||
| | Artifact | Location | | ||
| | --------------- | ---------------------------------------------------------------------------------- | | ||
| | Test plan | `.qualify/features/<name>/test-plan.md` or `.qualify/bugs/<id>/test-plan.md` | | ||
| | Proof report | `.qualify/features/<name>/proof.md` or `.qualify/bugs/<id>/proof.md` | | ||
| | Test output log | `.qualify/features/<name>/test-output.log` or `.qualify/bugs/<id>/test-output.log` | | ||
| | PR | GitHub (features and bugs with permanent tests) | | ||
|
|
||
| ## Arguments | ||
|
|
||
| | Argument | Required | Description | | ||
| | ----------- | -------- | ---------------------------------------------------------------------- | | ||
| | `--type` | Yes | `feature` or `bug` | | ||
| | `--source` | Yes | URL to Jira ticket, GitHub issue, design doc, or local file path | | ||
| | `--cluster` | Yes | Path to kubeconfig for qualification (explicit target required) | | ||
| | `--name` | No | Short identifier (e.g., `warm-migration-rhv`). Auto-derived if omitted | | ||
|
myakove marked this conversation as resolved.
|
||
|
|
||
| ## Usage Examples | ||
|
|
||
| ### Qualify a New Feature | ||
|
|
||
| ```bash | ||
| /qualify --type feature --source https://issues.redhat.com/browse/MTV-1234 --cluster ~/kubeconfigs/test-cluster | ||
| ``` | ||
|
|
||
| ### Verify a Bug Fix | ||
|
|
||
| ```bash | ||
| /qualify --type bug --source https://issues.redhat.com/browse/MTV-5678 --cluster ~/kubeconfigs/test-cluster --name MTV-5678 | ||
| ``` | ||
|
coderabbitai[bot] marked this conversation as resolved.
|
||
|
|
||
| The AI will ask: "Should this bug get a permanent test in the test suite?" | ||
|
|
||
| - **Yes** → full flow: test plan → write test → PR → proof | ||
| - **No** → verify-only: test plan → run throwaway test → proof.md (no PR) | ||
|
|
||
| ## Human Checkpoints | ||
|
|
||
| The workflow is fully automated EXCEPT at these points: | ||
|
|
||
| | Checkpoint | When | What | | ||
| | ------------------ | --------------------------- | ----------------------------------------- | | ||
| | Test plan review | After Phase 1 | Approve or give feedback on the test plan | | ||
| | Bug: suite or not? | Start of bug workflow | Decide if test joins permanent suite | | ||
| | AI stuck | When AI can't make progress | Guide the AI on how to proceed | | ||
| | PR review | After Phase 3 | Normal GitHub PR review | | ||
|
|
||
| ## Setup by AI CLI | ||
|
|
||
| ### pi | ||
|
|
||
| 1. Add to `.pi/settings.json`: | ||
|
|
||
| ```json | ||
| { | ||
| "prompts": ["llm/qualify/prompts"], | ||
| "skills": ["llm/qualify/skills"] | ||
| } | ||
| ``` | ||
|
|
||
| 2. Register agents — add to your pi-config or project agents: | ||
|
|
||
| ```json | ||
| { | ||
| "agents": ["llm/qualify/agents"] | ||
| } | ||
| ``` | ||
|
|
||
| 3. Use: type `/qualify` in pi's interactive mode. | ||
|
|
||
| ### Claude Code | ||
|
|
||
| 1. Copy or symlink the prompt template: | ||
|
|
||
| ```bash | ||
| mkdir -p .claude/commands | ||
| cp llm/qualify/prompts/qualify.md .claude/commands/qualify.md | ||
| ``` | ||
|
|
||
| 2. Reference agents and skills in `CLAUDE.md`: | ||
|
|
||
| ```markdown | ||
| ## Qualification Workflow | ||
| See `llm/qualify/` for the /qualify workflow: | ||
| - Agents: `llm/qualify/agents/` | ||
| - Skills: `llm/qualify/skills/` | ||
| - Templates: `llm/qualify/templates/` | ||
| ``` | ||
|
|
||
| 3. Use: type `/qualify` in Claude Code. | ||
|
|
||
| ### Cursor | ||
|
|
||
| 1. Add as a Notepad or Rule: | ||
| - Copy content from `llm/qualify/prompts/qualify.md` into a Cursor Rule | ||
| - Reference agent/skill files in the rule | ||
|
|
||
| 2. Or use `.cursorrules` to reference the qualify workflow. | ||
|
|
||
| ### Other AI CLIs | ||
|
|
||
| The workflow is plain Markdown — adapt to any AI CLI that supports: | ||
|
|
||
| - Prompt templates or system prompts | ||
| - Agent/persona definitions | ||
| - Tool access (file read/write, bash execution, web fetching) | ||
|
|
||
| Copy the relevant `.md` files into your CLI's configuration format. | ||
|
|
||
| ## Directory Structure | ||
|
|
||
| ```text | ||
| llm/qualify/ | ||
| ├── README.md # This file | ||
| ├── prompts/ | ||
| │ └── qualify.md # Main prompt template (/qualify command) | ||
| ├── agents/ | ||
| │ ├── test-planner.md # Reads docs → produces test plans | ||
| │ └── cluster-verifier.md # Independently verifies cluster state | ||
| ├── skills/ | ||
| │ └── proof-generator/ | ||
| │ └── SKILL.md # Assembles proof.md reports | ||
| └── templates/ | ||
| ├── test-plan-template.md # Test plan skeleton | ||
| └── proof-template.md # Proof report skeleton | ||
| ``` | ||
|
|
||
| Output (gitignored): | ||
|
|
||
| ```text | ||
| .qualify/ | ||
| ├── features/ | ||
| │ └── <name>/ | ||
| │ ├── test-plan.md | ||
| │ ├── test-output.log | ||
| │ └── proof.md | ||
| └── bugs/ | ||
| └── <id>/ | ||
| ├── test-plan.md | ||
| ├── test-output.log | ||
| └── proof.md | ||
| ``` | ||
|
|
||
| ## Requirements | ||
|
|
||
| - `oc` CLI configured and authenticated to a working OpenShift cluster | ||
| - MTV operator installed on the cluster | ||
| - CNV installed on the cluster | ||
| - Source provider configured (VMware, RHV, etc.) with test VMs available | ||
| - `.providers.json` configured in the repo | ||
Empty file.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.