Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -174,3 +174,6 @@ jira.cfg
# ENV
.env
.envrc

# Qualify workflow
.qualify/
2 changes: 2 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,8 @@ AI must NEVER run tests directly (`pytest`, `uv run pytest`). Tests require live
AI can: Read/analyze/write/fix tests, suggest improvements, review structure
AI cannot: Execute tests, validate by running

**Exception:** The `/qualify` workflow (`llm/qualify/`) may run pytest on a real cluster when a user explicitly invokes it with cluster credentials. See `llm/qualify/README.md`.

### No Module-Level Provider Loading (MUST)

`load_source_providers()` must only be called within the pytest ecosystem (fixtures, hooks).
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1032,3 +1032,12 @@ uv run pytest -v \

# For debug options (--skip-teardown, -s -vv, etc.), see "Useful Test Options" section above
```

---

## AI Qualification Workflow (Optional)

The `/qualify` command provides an AI-driven qualification workflow:
test plan → write tests → verify on cluster → PR with proof.

See [llm/qualify/README.md](llm/qualify/README.md) for setup and usage.
167 changes: 167 additions & 0 deletions llm/qualify/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
# /qualify — AI Qualification Workflow

Full qualification workflow for MTV API tests: from feature design or bug report to verified PR with proof.

## What It Does

```bash
/qualify --type feature --source <url> --cluster ~/kubeconfig
```

1. **Test Plan** — AI reads feature/bug docs → produces a test plan → human reviews
2. **Write Tests** — AI writes E2E customer use-case tests following project patterns
3. **Verify on Cluster** — AI runs tests on a real cluster AND independently verifies cluster state
4. **Code Review** — AI reviewers check the code (on pi with myk-org/pi-config: 3 parallel reviewers; elsewhere per project `AGENTS.md` / `CLAUDE.md`)
5. **PR with Proof** — Creates PR with proof.md documenting test results + cluster evidence + versions

### Outputs

| Artifact | Location |
| --------------- | ---------------------------------------------------------------------------------- |
| Test plan | `.qualify/features/<name>/test-plan.md` or `.qualify/bugs/<id>/test-plan.md` |
| Proof report | `.qualify/features/<name>/proof.md` or `.qualify/bugs/<id>/proof.md` |
| Test output log | `.qualify/features/<name>/test-output.log` or `.qualify/bugs/<id>/test-output.log` |
Comment thread
myakove marked this conversation as resolved.
| PR | GitHub (features and bugs with permanent tests) |

## Arguments

| Argument | Required | Description |
| ----------- | -------- | ---------------------------------------------------------------------- |
| `--type` | Yes | `feature` or `bug` |
| `--source` | Yes | URL to Jira ticket, GitHub issue, design doc, or local file path |
| `--cluster` | Yes | Path to kubeconfig for qualification (explicit target required) |
| `--name` | No | Short identifier (e.g., `warm-migration-rhv`). Auto-derived if omitted |
Comment thread
myakove marked this conversation as resolved.

## Usage Examples

### Qualify a New Feature

```bash
/qualify --type feature --source https://issues.redhat.com/browse/MTV-1234 --cluster ~/kubeconfigs/test-cluster
```

### Verify a Bug Fix

```bash
/qualify --type bug --source https://issues.redhat.com/browse/MTV-5678 --cluster ~/kubeconfigs/test-cluster --name MTV-5678
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

The AI will ask: "Should this bug get a permanent test in the test suite?"

- **Yes** → full flow: test plan → write test → PR → proof
- **No** → verify-only: test plan → run throwaway test → proof.md (no PR)

## Human Checkpoints

The workflow is fully automated EXCEPT at these points:

| Checkpoint | When | What |
| ------------------ | --------------------------- | ----------------------------------------- |
| Test plan review | After Phase 1 | Approve or give feedback on the test plan |
| Bug: suite or not? | Start of bug workflow | Decide if test joins permanent suite |
| AI stuck | When AI can't make progress | Guide the AI on how to proceed |
| PR review | After Phase 3 | Normal GitHub PR review |

## Setup by AI CLI

### pi

1. Add to `.pi/settings.json`:

```json
{
"prompts": ["llm/qualify/prompts"],
"skills": ["llm/qualify/skills"]
}
```

2. Register agents — add to your pi-config or project agents:

```json
{
"agents": ["llm/qualify/agents"]
}
```

3. Use: type `/qualify` in pi's interactive mode.

### Claude Code

1. Copy or symlink the prompt template:

```bash
mkdir -p .claude/commands
cp llm/qualify/prompts/qualify.md .claude/commands/qualify.md
```

2. Reference agents and skills in `CLAUDE.md`:

```markdown
## Qualification Workflow
See `llm/qualify/` for the /qualify workflow:
- Agents: `llm/qualify/agents/`
- Skills: `llm/qualify/skills/`
- Templates: `llm/qualify/templates/`
```

3. Use: type `/qualify` in Claude Code.

### Cursor

1. Add as a Notepad or Rule:
- Copy content from `llm/qualify/prompts/qualify.md` into a Cursor Rule
- Reference agent/skill files in the rule

2. Or use `.cursorrules` to reference the qualify workflow.

### Other AI CLIs

The workflow is plain Markdown — adapt to any AI CLI that supports:

- Prompt templates or system prompts
- Agent/persona definitions
- Tool access (file read/write, bash execution, web fetching)

Copy the relevant `.md` files into your CLI's configuration format.

## Directory Structure

```text
llm/qualify/
├── README.md # This file
├── prompts/
│ └── qualify.md # Main prompt template (/qualify command)
├── agents/
│ ├── test-planner.md # Reads docs → produces test plans
│ └── cluster-verifier.md # Independently verifies cluster state
├── skills/
│ └── proof-generator/
│ └── SKILL.md # Assembles proof.md reports
└── templates/
├── test-plan-template.md # Test plan skeleton
└── proof-template.md # Proof report skeleton
```

Output (gitignored):

```text
.qualify/
├── features/
│ └── <name>/
│ ├── test-plan.md
│ ├── test-output.log
│ └── proof.md
└── bugs/
└── <id>/
├── test-plan.md
├── test-output.log
└── proof.md
```

## Requirements

- `oc` CLI configured and authenticated to a working OpenShift cluster
- MTV operator installed on the cluster
- CNV installed on the cluster
- Source provider configured (VMware, RHV, etc.) with test VMs available
- `.providers.json` configured in the repo
Empty file added llm/qualify/agents/.gitkeep
Empty file.
Loading