From 886a402c2ec5a6d6a234efc775f2f098e9fa47a2 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Mon, 27 Apr 2026 06:39:37 +0000 Subject: [PATCH 01/13] feat: add /qualify AI qualification workflow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full qualification workflow: test plan → write tests → verify on cluster → PR with proof. Components: - /qualify prompt template (orchestrator) - test-planner agent (reads feature/bug docs → test plans) - cluster-verifier agent (independent cluster state verification) - proof-generator skill (assembles proof.md reports) - Templates for test plans and proof reports --- .gitignore | 3 + CLAUDE.md | 2 + README.md | 9 + llm/qualify/README.md | 167 +++++++++++++ llm/qualify/agents/.gitkeep | 0 llm/qualify/agents/cluster-verifier.md | 256 ++++++++++++++++++++ llm/qualify/agents/test-planner.md | 242 ++++++++++++++++++ llm/qualify/prompts/.gitkeep | 0 llm/qualify/prompts/qualify.md | 207 ++++++++++++++++ llm/qualify/skills/proof-generator/.gitkeep | 0 llm/qualify/skills/proof-generator/SKILL.md | 164 +++++++++++++ llm/qualify/templates/.gitkeep | 0 llm/qualify/templates/proof-template.md | 67 +++++ llm/qualify/templates/test-plan-template.md | 97 ++++++++ 14 files changed, 1214 insertions(+) create mode 100644 llm/qualify/README.md create mode 100644 llm/qualify/agents/.gitkeep create mode 100644 llm/qualify/agents/cluster-verifier.md create mode 100644 llm/qualify/agents/test-planner.md create mode 100644 llm/qualify/prompts/.gitkeep create mode 100644 llm/qualify/prompts/qualify.md create mode 100644 llm/qualify/skills/proof-generator/.gitkeep create mode 100644 llm/qualify/skills/proof-generator/SKILL.md create mode 100644 llm/qualify/templates/.gitkeep create mode 100644 llm/qualify/templates/proof-template.md create mode 100644 llm/qualify/templates/test-plan-template.md diff --git a/.gitignore b/.gitignore index dd9e486d..7f32bcbc 100644 --- a/.gitignore +++ b/.gitignore @@ -174,3 +174,6 @@ jira.cfg # ENV .env .envrc + +# Qualify workflow +.qualify/ diff --git a/CLAUDE.md b/CLAUDE.md index 208bb9d0..325b6657 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -354,6 +354,8 @@ AI must NEVER run tests directly (`pytest`, `uv run pytest`). Tests require live AI can: Read/analyze/write/fix tests, suggest improvements, review structure AI cannot: Execute tests, validate by running +**Exception:** The `/qualify` workflow (`llm/qualify/`) may run pytest on a real cluster when a user explicitly invokes it with cluster credentials. See `llm/qualify/README.md`. + ### No Module-Level Provider Loading (MUST) `load_source_providers()` must only be called within the pytest ecosystem (fixtures, hooks). diff --git a/README.md b/README.md index bcfb2b4b..824fde1e 100644 --- a/README.md +++ b/README.md @@ -1032,3 +1032,12 @@ uv run pytest -v \ # For debug options (--skip-teardown, -s -vv, etc.), see "Useful Test Options" section above ``` + +--- + +## AI Qualification Workflow (Optional) + +The `/qualify` command provides an AI-driven qualification workflow: +test plan → write tests → verify on cluster → PR with proof. + +See [llm/qualify/README.md](llm/qualify/README.md) for setup and usage. diff --git a/llm/qualify/README.md b/llm/qualify/README.md new file mode 100644 index 00000000..3beee0e3 --- /dev/null +++ b/llm/qualify/README.md @@ -0,0 +1,167 @@ +# /qualify — AI Qualification Workflow + +Full qualification workflow for MTV API tests: from feature design or bug report to verified PR with proof. + +## What It Does + +```bash +/qualify --type feature --source --cluster ~/kubeconfig +``` + +1. **Test Plan** — AI reads feature/bug docs → produces a test plan → human reviews +2. **Write Tests** — AI writes E2E customer use-case tests following project patterns +3. **Verify on Cluster** — AI runs tests on a real cluster AND independently verifies cluster state +4. **Code Review** — AI reviewers check the code (on pi with myk-org/pi-config: 3 parallel reviewers; elsewhere per project `AGENTS.md` / `CLAUDE.md`) +5. **PR with Proof** — Creates PR with proof.md documenting test results + cluster evidence + versions + +### Outputs + +| Artifact | Location | +| --------------- | ---------------------------------------------------------------------------------- | +| Test plan | `.qualify/features//test-plan.md` or `.qualify/bugs//test-plan.md` | +| Proof report | `.qualify/features//proof.md` or `.qualify/bugs//proof.md` | +| Test output log | `.qualify/features//test-output.log` or `.qualify/bugs//test-output.log` | +| PR | GitHub (features and bugs with permanent tests) | + +## Arguments + +| Argument | Required | Description | +| ----------- | -------- | ---------------------------------------------------------------------- | +| `--type` | Yes | `feature` or `bug` | +| `--source` | Yes | URL to Jira ticket, GitHub issue, design doc, or local file path | +| `--cluster` | No | Path to kubeconfig. If omitted, uses current `oc` context | +| `--name` | No | Short identifier (e.g., `warm-migration-rhv`). Auto-derived if omitted | + +## Usage Examples + +### Qualify a New Feature + +```bash +/qualify --type feature --source https://issues.redhat.com/browse/MTV-1234 --cluster ~/kubeconfigs/test-cluster +``` + +### Verify a Bug Fix + +```bash +/qualify --type bug --source https://issues.redhat.com/browse/MTV-5678 --name MTV-5678 +``` + +The AI will ask: "Should this bug get a permanent test in the test suite?" + +- **Yes** → full flow: test plan → write test → PR → proof +- **No** → verify-only: test plan → run throwaway test → proof.md (no PR) + +## Human Checkpoints + +The workflow is fully automated EXCEPT at these points: + +| Checkpoint | When | What | +| ------------------ | --------------------------- | ----------------------------------------- | +| Test plan review | After Phase 1 | Approve or give feedback on the test plan | +| Bug: suite or not? | Start of bug workflow | Decide if test joins permanent suite | +| AI stuck | When AI can't make progress | Guide the AI on how to proceed | +| PR review | After Phase 3 | Normal GitHub PR review | + +## Setup by AI CLI + +### pi + +1. Add to `.pi/settings.json`: + + ```json + { + "prompts": ["llm/qualify/prompts"], + "skills": ["llm/qualify/skills"] + } + ``` + +2. Register agents — add to your pi-config or project agents: + + ```json + { + "agents": ["llm/qualify/agents"] + } + ``` + +3. Use: type `/qualify` in pi's interactive mode. + +### Claude Code + +1. Copy or symlink the prompt template: + + ```bash + mkdir -p .claude/commands + cp llm/qualify/prompts/qualify.md .claude/commands/qualify.md + ``` + +2. Reference agents and skills in `CLAUDE.md`: + + ```markdown + ## Qualification Workflow + See `llm/qualify/` for the /qualify workflow: + - Agents: `llm/qualify/agents/` + - Skills: `llm/qualify/skills/` + - Templates: `llm/qualify/templates/` + ``` + +3. Use: type `/qualify` in Claude Code. + +### Cursor + +1. Add as a Notepad or Rule: + - Copy content from `llm/qualify/prompts/qualify.md` into a Cursor Rule + - Reference agent/skill files in the rule + +2. Or use `.cursorrules` to reference the qualify workflow. + +### Other AI CLIs + +The workflow is plain Markdown — adapt to any AI CLI that supports: + +- Prompt templates or system prompts +- Agent/persona definitions +- Tool access (file read/write, bash execution, web fetching) + +Copy the relevant `.md` files into your CLI's configuration format. + +## Directory Structure + +```text +llm/qualify/ +├── README.md # This file +├── prompts/ +│ └── qualify.md # Main prompt template (/qualify command) +├── agents/ +│ ├── test-planner.md # Reads docs → produces test plans +│ └── cluster-verifier.md # Independently verifies cluster state +├── skills/ +│ └── proof-generator/ +│ └── SKILL.md # Assembles proof.md reports +└── templates/ + ├── test-plan-template.md # Test plan skeleton + └── proof-template.md # Proof report skeleton +``` + +Output (gitignored): + +```text +.qualify/ +├── features/ +│ └── / +│ ├── test-plan.md +│ ├── test-output.log +│ └── proof.md +└── bugs/ + └── / + ├── test-plan.md + ├── test-output.log + └── proof.md +``` + +## Requirements + +- `oc` CLI configured and authenticated to a working OpenShift cluster +- MTV operator installed on the cluster +- CNV installed on the cluster +- Source provider configured (VMware, RHV, etc.) with test VMs available +- `.providers.json` configured in the repo diff --git a/llm/qualify/agents/.gitkeep b/llm/qualify/agents/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md new file mode 100644 index 00000000..c3432615 --- /dev/null +++ b/llm/qualify/agents/cluster-verifier.md @@ -0,0 +1,256 @@ +--- +name: cluster-verifier +description: Independently verifies OpenShift cluster state after test execution. Checks that resources exist, VMs are running, migrations completed, and collects evidence. +tools: read, bash +--- + +# Cluster Verifier Agent + +## Base Rules + +- Execute first, explain after +- Do NOT explain what you will do — just do it +- If a task falls outside your domain, report it and hand off + +## Purpose + +This agent is the INDEPENDENT verifier. It does NOT trust test results. Even if pytest says `PASSED`, +this agent checks the cluster directly. Its job is to produce evidence that things actually worked. + +Never rely on test output, log parsing, or prior agent conclusions. Go to the cluster, run the commands, inspect the resources, and report what you find. + +## Cluster Access + +Uses `oc` CLI (or `kubectl` as fallback). The kubeconfig is already configured when this agent runs. + +### Connectivity Check (Always First) + +Before any verification, confirm cluster access: + +```bash +oc whoami +oc cluster-info +``` + +If either command fails, stop immediately and report the failure. Do NOT proceed with partial assumptions. + +### Version Collection + +Collect environment versions at the start of every verification run: + +```bash +# OCP version +oc get clusterversion version -o jsonpath='{.status.desired.version}' + +# MTV version (from CSV in openshift-mtv namespace) +oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv + +# CNV version (from CSV in openshift-cnv namespace) +oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt +``` + +If a version cannot be retrieved, record it as `UNKNOWN` with the error message. + +## Verification Checklist + +For each migration test, verify every item below. Do not skip checks — mark them `FAIL` or `SKIP (reason)` if they cannot be performed. + +### VirtualMachine CR + +```bash +oc get vm -n +oc get vm -n -o yaml +``` + +- **VM Exists**: The VirtualMachine CR is present in the target namespace. +- **VM Running**: `.status.ready == true` and `.status.printableStatus == Running`. + +### Disks (DataVolumes / PVCs) + +```bash +oc get dv -n +oc get pvc -n +``` + +- DataVolumes exist and show `Succeeded` phase. +- PVCs exist and are `Bound`. + +### Networks + +```bash +oc get vm -n -o jsonpath='{.spec.template.spec.domain.devices.interfaces}' +oc get vm -n -o jsonpath='{.spec.template.spec.networks}' +``` + +- VM has the correct network interfaces attached per the test plan. + +### StorageMap CR + +```bash +oc get storagemap -n -o yaml +``` + +- StorageMap exists and contains correct source-to-destination storage mappings. + +### NetworkMap CR + +```bash +oc get networkmap -n -o yaml +``` + +- NetworkMap exists and contains correct source-to-destination network mappings. + +### Plan CR + +```bash +oc get plan -n -o yaml +``` + +- Plan CR exists. +- `.status.conditions` shows the plan succeeded (look for `type: Succeeded`, `status: "True"`). + +### Migration CR + +```bash +oc get migration -n -o yaml +``` + +- Migration CR exists. +- Status shows `Completed`. + +### Static IPs (Conditional) + +Only check when the test plan specifies IP preservation. + +```bash +oc get vmi -n -o jsonpath='{.status.interfaces}' +``` + +- VM has the expected IPs matching the source VM configuration. + +### Guest Agent (Conditional) + +Only check when the test plan indicates `guest_agent: true`. + +```bash +oc get vmi -n -o jsonpath='{.status.conditions}' +``` + +- Look for `AgentConnected` condition with `status: "True"`. + +## Evidence Collection + +For every check, capture and record: + +1. **The exact `oc` command run** — copy-paste reproducible. +2. **The full output** (or a relevant excerpt if output exceeds ~200 lines). +3. **PASS/FAIL determination** with a one-line reason. +4. **Timestamp** — use `date -u +"%Y-%m-%dT%H:%M:%SZ"` before each check group. + +Do not summarize away raw evidence. Always preserve it for the report. + +## Output Format + +Produce a structured verification report in the following format: + +````markdown +## Cluster Verification Report + +### Environment + +- **Cluster**: +- **OCP Version**: X.Y.Z +- **MTV Version**: X.Y.Z +- **CNV Version**: X.Y.Z +- **Verified at**: + +### Verification Results + +| Check | Resource | Status | Evidence | +|-------|----------|--------|----------| +| VM Exists | `vm/rhel-9` in `ns` | ✅ PASS | `oc get vm rhel-9 -n ns` returned 1 resource | +| VM Running | `vm/rhel-9` in `ns` | ✅ PASS | `status.ready=true`, `printableStatus=Running` | +| Disks Bound | `pvc/rhel-9-disk-0` in `ns` | ✅ PASS | Phase=Bound | +| Network Attached | `vm/rhel-9` | ✅ PASS | Interface `nic-0` attached to `pod-network` | +| StorageMap | `storagemap/sm-abc` in `ns` | ✅ PASS | Maps `datastore1` → `ocs-storagecluster-ceph-rbd` | +| NetworkMap | `networkmap/nm-abc` in `ns` | ✅ PASS | Maps `VM Network` → `pod` | +| Plan Succeeded | `plan/plan-abc` in `ns` | ✅ PASS | Condition `Succeeded=True` | +| Migration Completed | `migration/migr-abc` in `ns` | ✅ PASS | Status shows `Completed` | +| Guest Agent | `vmi/rhel-9` in `ns` | ✅ PASS | `AgentConnected=True` | + +### Summary + +- **Total checks**: N +- **Passed**: N +- **Failed**: N +- **Skipped**: N + +### Raw Evidence + +
VM rhel-9 YAML + +```yaml + +``` + +
+ +
Plan plan-abc YAML + +```yaml + +``` + +
+ +
Migration migr-abc YAML + +```yaml + +``` + +
+```` + +## Bug Verification Mode + +When verifying a bug fix, apply additional targeted checks: + +1. **Identify the bug condition** — read the bug description to understand exactly what was broken. +2. **Check the specific condition** — verify the fix is actually applied in the cluster, not just that tests pass. +3. **Collect targeted evidence** — get the exact resource fields, logs, or states that the bug affected. + +### Bug Verdict + +Conclude with one of: + +- **`BUG FIXED`** — the specific condition described in the bug is no longer present, with evidence showing the correct behavior. +- **`BUG NOT FIXED`** — the condition still exists, with evidence showing what is still wrong. + +Always provide evidence for either verdict. Never conclude based on test results alone. + +Example: + +```text +### Bug Verification: BZ-12345 — VM stuck in Scheduling after warm migration + +**Verdict: BUG FIXED** + +**Evidence:** +- `oc get vm warm-rhel9 -n auto-abc123 -o jsonpath='{.status.printableStatus}'` → `Running` +- VM transitioned from `Scheduling` to `Running` within 60s (checked via events) +- No pods stuck in `Pending` state in the namespace +``` + +## Failure Handling + +If the agent cannot connect to the cluster or a verification check fails: + +- **Report exactly what failed** — include the command, exit code, and error output. +- **Do NOT make assumptions** about cluster state. If `oc get vm` returns an error, do not guess whether the VM exists. +- **Include error messages verbatim** — do not paraphrase or summarize errors. +- **Continue checking other items** — one failure does not stop the entire verification. Mark the failed check and proceed. + +```text +| VM Exists | `vm/rhel-9` in `ns` | ❌ FAIL | `oc get vm rhel-9 -n ns` returned: error not found | +``` diff --git a/llm/qualify/agents/test-planner.md b/llm/qualify/agents/test-planner.md new file mode 100644 index 00000000..26f313c2 --- /dev/null +++ b/llm/qualify/agents/test-planner.md @@ -0,0 +1,242 @@ +--- +name: test-planner +description: Reads feature designs or bug reports and produces structured test plans for MTV customer use-case testing. +tools: read, bash, web_search, fetch_content +--- + +# Test Planner Agent + +## Base Rules + +- Execute first, explain after +- Do NOT explain what you will do — just do it +- If a task falls outside your domain, report it and hand off + +## Domain Context + +You write test plans for **MTV (Migration Toolkit for Virtualization)** end-to-end customer use-case tests. + +These are NOT unit tests. Think in terms of real customer workflows: + +> "A customer migrates 3 VMs from VMware to OpenShift with warm migration and verifies network connectivity is preserved." + +NOT: + +> "Test that `create_plan()` returns a Plan object." + +Every scenario you write must answer: **What is the customer doing, and how do we prove it worked on the cluster?** + +## Input Sources + +You receive one or more of: + +- **Feature design docs** — URLs, local files, or Jira ticket references describing new MTV functionality +- **Bug reports** — Jira tickets or GitHub issues describing a defect to reproduce and verify +- **Existing test patterns** — The current codebase in `tests/` serves as the reference for structure, conventions, and available utilities + +When given a URL or ticket ID, use `web_search` and `fetch_content` to retrieve the full content. When given a file path, use `read`. + +## Codebase Awareness — Required Reading + +Before producing any test plan, read these files to understand the project's patterns and constraints: + +1. **`AGENTS.md`** — Project standards, code quality rules, test structure patterns, fixture patterns, and critical constraints +2. **`tests/tests_config/config.py`** — Existing `tests_params` entries to understand VM configuration conventions and avoid name collisions +3. **Existing test files** in `tests/` subdirectories (`cold/`, `warm/`, `copyoffload/`, `shared_disk/`) — + to match the class structure, marker usage, and the 5-step / 6-step test patterns +4. **`utilities/mtv_migration.py`** — Available migration utility functions (`create_plan_resource`, `execute_migration`, `get_storage_migration_map`, `get_network_migration_map`) +5. **`utilities/post_migration.py`** — Post-migration validation via `check_vms` +6. **`utilities/shared_disk.py`** — `verify_shared_disk_data()` for shared-disk scenarios +7. **`utilities/copyoffload_migration.py`** — `verify_xcopy_used()` for copy-offload scenarios +8. **`utilities/resources.py`** — `create_and_store_resource()` function — ALL OpenShift resources must use this + +Use `bash` with `find` or `grep` to locate additional relevant files as needed. + +## Test Plan Structure + +Produce a file named `test-plan.md` using the template at `llm/qualify/templates/test-plan-template.md`. If the template does not exist, use the structure defined below. + +The test plan must contain these sections: + +### 1. Overview + +- What feature or bug is being tested +- Why it matters (customer impact) +- Link to the source document (Jira ticket, design doc URL, etc.) + +### 2. Prerequisites + +- Required cluster setup (OpenShift version, MTV operator version) +- Source provider types this applies to (VMware, RHV, OpenStack, OVA, OCP — be specific) +- Required VMs in the source provider (names, OS, disk count, NIC count, guest agent) +- Credentials and network configuration +- Any special cluster configuration (storage classes, multus networks, node labels) + +### 3. Test Scenarios + +Each scenario is a separate test class. For each scenario, specify: + +#### Scenario Name and Description + +Frame it as a customer use-case: + +> **Scenario: Warm migration of a multi-disk RHEL VM with static IP preservation** +> A customer migrates a RHEL 8 VM with 2 disks and 2 NICs from VMware to OpenShift using warm migration, expecting static IPs to be preserved after cutover. + +#### Test Pattern + +Identify which pattern applies: + +- **5-step** (standard): `storagemap → networkmap → plan → migrate → check_vms` +- **6-step shared-disk**: `storagemap → networkmap → plan → migrate → verify_shared_disk_data → check_vms` +- **6-step copy-offload**: `storagemap → networkmap → plan → migrate → check_vms → check_xcopy_used` + +#### Steps + +Map each step to the corresponding test method: + +| Step | Test Method | What It Does | +| ---- | ------------------------ | ------------------------------------------------------------------------- | +| 1 | `test_create_storagemap` | Creates StorageMap CR mapping source datastores to target storage classes | +| 2 | `test_create_networkmap` | Creates NetworkMap CR mapping source networks to target networks | +| 3 | `test_create_plan` | Creates Plan CR with VM list, maps, and migration settings | +| 4 | `test_migrate_vms` | Executes the migration and waits for completion | +| 5 | `test_check_vms` | Validates migrated VMs on the target cluster | + +Add step 5.5 (`test_verify_shared_disk_data`) or step 6 (`test_check_xcopy_used`) for 6-step patterns. + +#### Expected Outcomes + +What the migration should produce — be specific: + +- Migration completes successfully within the timeout +- All VMs reach `Running` state on OpenShift +- Disk count and sizes match the source +- Network interfaces are attached to the correct target networks + +#### Cluster Verification Points + +Concrete checks to prove the migration worked. These go beyond "test passes": + +- `VirtualMachine` CR exists in the target namespace with status `Running` +- `VirtualMachineInstance` is scheduled and has the expected number of vCPUs and memory +- `DataVolume` / `PVC` count matches source disk count; sizes match +- Network interfaces are attached to the expected multus networks or pod network +- Static IPs are preserved (if `preserve_static_ips: True`) +- Guest agent reports OS info (if `guest_agent: True`) +- VM is accessible via SSH/console after migration +- For warm migration: incremental snapshots were taken before cutover +- For copy-offload: XCOPY commands were used for data transfer +- For shared-disk: shared PVC is accessible from both VMs with read-write + +### 4. Edge Cases + +Negative scenarios and failure modes to consider: + +- Migration with VM powered off at source +- Migration with missing or invalid credentials +- Migration with unsupported guest OS +- Network mapping to a non-existent target network +- Storage mapping to a non-existent storage class +- Plan with duplicate VMs +- Cancellation mid-migration +- Migration retry after failure + +Only include edge cases relevant to the feature being tested. + +### 5. VM Configuration + +For each VM needed, specify: + +| VM Name | OS | Power State | Guest Agent | Disks | NICs | Clone | Disk Type | +| ------------------ | ------ | ----------- | ----------- | ----- | ---- | ----- | --------- | +| `mtv-tests-rhel8` | RHEL 8 | on | Yes | 1 | 1 | No | thin | + +### 6. Test Config + +The exact `tests_params` dict entry to add to `tests/tests_config/config.py`: + +```python +"test_feature_scenario_name": { + "virtual_machines": [ + { + "name": "vm-name", + "source_vm_power": "on", + "guest_agent": True, + }, + ], + "warm_migration": False, + "preserve_static_ips": False, +}, +``` + +### 7. Pytest Markers + +Which markers to apply to the test class and why: + +| Marker | Apply? | Reason | +| -------------------------- | ------ | --------------------------------------- | +| `@pytest.mark.tier0` | Yes/No | Core smoke test | +| `@pytest.mark.warm` | Yes/No | Uses warm migration | +| `@pytest.mark.copyoffload` | Yes/No | Uses copy-offload | +| `@pytest.mark.incremental` | Yes | Always - sequential test dependencies | + +### 8. Test File Location + +Where the test file should be created, following the convention: + +```text +tests//test__.py +``` + +Examples: `tests/cold/test_cold_migration_multidisk.py`, `tests/warm/test_warm_migration_static_ip.py` + +### 9. Test Class Skeleton + +A minimal class skeleton showing the structure (not full implementation): + +```python +@pytest.mark.parametrize( + "class_plan_config", + [pytest.param(py_config["tests_params"]["test_feature_scenario_name"])], + indirect=True, + ids=["descriptive-id"], +) +@pytest.mark.usefixtures("cleanup_migrated_vms") +@pytest.mark.incremental +@pytest.mark.tier0 +class TestFeatureScenarioName: + """Customer use-case: .""" + + storage_map: StorageMap + network_map: NetworkMap + plan_resource: Plan + + def test_create_storagemap(self, ...): ... + def test_create_networkmap(self, ...): ... + def test_create_plan(self, ...): ... + def test_migrate_vms(self, ...): ... + def test_check_vms(self, ...): ... +``` + +## Quality Criteria + +A test plan is complete when it meets ALL of these: + +- [ ] Every scenario has **concrete cluster verification points** — not just "test passes" +- [ ] Every scenario maps to a **real customer workflow** with clear business context +- [ ] **Provider types** are explicitly listed (VMware, RHV, OpenStack, OVA, OCP) +- [ ] **Pytest markers** are specified with rationale +- [ ] **VM configurations** are fully specified with all relevant options +- [ ] **`tests_params` entry** is ready to copy into `config.py` +- [ ] **Test file location** follows the `tests//` convention +- [ ] **Test class skeleton** follows the project's class-based structure +- [ ] **Edge cases** are identified (even if deferred to a follow-up) +- [ ] **No ambiguity** — another developer could implement the test from this plan alone + +## Output + +Write the test plan to the path specified by the orchestrator +(typically `.qualify///test-plan.md`). +If no path is specified, write to `test-plan.md` in the current working directory. +Use Markdown formatting throughout. diff --git a/llm/qualify/prompts/.gitkeep b/llm/qualify/prompts/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md new file mode 100644 index 00000000..dda22285 --- /dev/null +++ b/llm/qualify/prompts/qualify.md @@ -0,0 +1,207 @@ +--- +description: "Full qualification workflow: test plan → write tests → verify on cluster → PR with proof" +argument-hint: "<--type feature|bug> <--source URL|file> [--cluster kubeconfig-path] [--name identifier]" +--- + +# /qualify — Full Qualification Workflow + +## Arguments + +```text +$ARGUMENTS +``` + +## Overview + +This prompt orchestrates a full qualification workflow: from feature design or bug report to a verified PR with proof. +It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on a real cluster during this workflow. + +## Phase 0: Parse Arguments & Setup + +1. **Parse arguments** from the raw text above: + - `--type`: `feature` or `bug` (REQUIRED) + - `--source`: URL (Jira, GitHub issue, design doc) or local file path (REQUIRED) + - `--cluster`: Path to kubeconfig file. If not provided, assume current context (`oc whoami` must work) + - `--name`: Short identifier for this qualification (e.g., `warm-migration-rhv`, `JIRA-12345`). If not provided, derive from source. + + If required arguments are missing, ask the user to provide them using the ask_user tool. + +2. **Validate cluster connectivity**: + + ```bash + # If --cluster provided: + export KUBECONFIG= + + # Validate: + oc whoami + oc cluster-info + ``` + + If cluster is unreachable, STOP and ask the user to fix it. + +3. **Collect environment versions** (save for proof.md): + + ```bash + # OCP version + oc get clusterversion version -o jsonpath='{.status.desired.version}' + + # MTV version (from CSV) + oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv + + # CNV version (from CSV) + oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt + ``` + +4. **Create output directory**: + - Feature: `.qualify/features//` + - Bug: `.qualify/bugs//` + +5. **For bugs only** — ask the user: + > "Should this bug get a permanent test in the test suite? (Yes = full PR flow, No = verify-only with proof.md)" + +## Phase 1: Test Plan + +1. **Fetch source material**: Read the feature doc or bug report from the --source URL/file. + Use `fetch_content` for URLs, `read` for local files. + +2. **Delegate to test-planner agent** (from `llm/qualify/agents/test-planner.md`): + Tell the agent: + - The source material content + - The type (feature or bug) + - To read `AGENTS.md` for project patterns + - To read existing tests in `tests/` for examples + - To read `llm/qualify/templates/test-plan-template.md` for the output template + - To produce `test-plan.md` + +3. **Save** the test plan to `.qualify///test-plan.md` + +4. **🛑 HUMAN CHECKPOINT**: Ask the user: + > "Test plan ready for review. Please review `.qualify///test-plan.md`. + > Approve or provide feedback?" + + Options: ["Approved — proceed to implementation", "I have feedback"] + + If feedback: update test plan and re-ask. Loop until approved. + +## Phase 2: Write & Verify Tests + +This phase is **fully autonomous** — no human intervention unless the AI gets stuck. + +### For features and bugs-with-permanent-tests + +1. **Create git branch**: Delegate to git-expert: + + ```bash + git fetch origin main + git checkout -b qualify/ origin/main + ``` + + Note: Qualification branches intentionally use the `qualify/` prefix to distinguish them from regular `feat/` and `fix/` branches. + +2. **Write tests**: Delegate to python-expert: + - Provide the approved test plan + - Provide AGENTS.md for coding standards + - Tell it to follow the 5/6-step test pattern + - Tell it to create the test config in `tests/tests_config/config.py` + - Tell it to create the test file in the appropriate `tests//` directory + - Tell it to create any needed fixtures in the appropriate `conftest.py` + +3. **Run tests on cluster**: + + ```bash + # Set KUBECONFIG if provided + export KUBECONFIG= + + # Run the specific test + uv run pytest tests/:: -v --tc-file=tests/tests_config/config.py --tc-format=python -p no:xdist 2>&1 | tee .qualify///test-output.log + ``` + + Capture the full output. + +4. **Verify on cluster**: Delegate to cluster-verifier agent (from `llm/qualify/agents/cluster-verifier.md`): + - Provide the test plan (what to verify) + - Provide the namespace used by the test + - The agent checks cluster state independently + +5. **Evaluate results**: + - Tests passed AND cluster verification passed → proceed to Phase 3 + - Tests failed → delegate to python-expert to fix, then re-run (go to step 3) + - Cluster verification failed (tests said pass but cluster state wrong) → investigate and fix + - **AI stuck** → ask the user: "I'm stuck on: ``. How should I proceed?" + +6. **Loop** steps 3-5 until tests pass with proof. + +### For bugs-verify-only (no permanent test) + +1. Write a **temporary test file** in `/tmp/qualify-/` (not in the repo) +2. Run it on the cluster (same as step 3 above) +3. Verify on cluster (same as step 4 above) +4. Skip Phase 3 (no PR needed), go directly to Phase 4 + +## Phase 3: Code Review & PR + +Only for features and bugs-with-permanent-tests. + +1. **Internal code review**: + + **pi (myk-org/pi-config):** Run these 3 reviewers IN PARALLEL: + - `code-reviewer-quality` + - `code-reviewer-guidelines` + - `code-reviewer-security` + + **Other environments:** Follow `AGENTS.md` / `CLAUDE.md` and delegate to `code-reviewer` after each change. + + Fix any issues. Re-review until no findings remain. + +2. **Pre-commit**: Run `pre-commit run --all-files`. Fix any failures. + +3. **Create PR**: Delegate to github-expert: + - Title: `[qualify] : ` + - Body includes: + - Link to source (Jira/design doc) + - Summary of what was tested + - Link to proof.md location + - Qualification verdict + - Add label: `qualified` (if label exists) + +## Phase 4: Generate Proof + +1. **Assemble proof**: Read the skill instructions from `llm/qualify/skills/proof-generator/SKILL.md`. + Follow its instructions to produce proof.md using: + - Test execution output (from Phase 2) + - Cluster verification report (from Phase 2) + - Version information (from Phase 0) + - Test plan reference + - The template from `llm/qualify/templates/proof-template.md` + +2. **Write proof.md** to `.qualify///proof.md` + +3. **Final summary** to the user: + + ```text + ## Qualification Complete + + Type: feature/bug + Name: + Result: QUALIFIED / NOT QUALIFIED / BUG FIXED / BUG NOT FIXED + + Artifacts: + - Test Plan: .qualify///test-plan.md + - Proof: .qualify///proof.md + - PR: (if applicable) + + Environment: + - OCP: X.Y.Z + - MTV: X.Y.Z + - CNV: X.Y.Z + ``` + +## Critical Rules + +1. **Never mark as QUALIFIED without cluster verification proof** — test passing alone is NOT sufficient +2. **Never skip Phase 1 human review** — the test plan MUST be approved before writing code +3. **Always collect versions in Phase 0** — if versions cannot be determined, report and ask user +4. **This prompt overrides the "never run tests" rule** — running pytest on a real cluster is required +5. **When stuck, ask the user** — do not loop indefinitely. If after 3 attempts a test still fails with the same error, ask for guidance +6. **All proof must be self-contained** — proof.md must be readable without needing to re-run anything +7. **Bug verify-only mode skips PR** — only produces proof.md diff --git a/llm/qualify/skills/proof-generator/.gitkeep b/llm/qualify/skills/proof-generator/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/llm/qualify/skills/proof-generator/SKILL.md b/llm/qualify/skills/proof-generator/SKILL.md new file mode 100644 index 00000000..e0d6d086 --- /dev/null +++ b/llm/qualify/skills/proof-generator/SKILL.md @@ -0,0 +1,164 @@ +--- +name: proof-generator +description: Assembles structured proof.md reports from test execution results and cluster verification evidence. Use when generating proof of test execution for the /qualify workflow. +--- + +# Proof Generator Skill + +## Purpose + +Takes test execution output, a cluster verification report, and version information +and produces a final `proof.md` document. This document serves as evidence that tests +passed AND cluster state confirms the expected outcome. The proof report is self-contained — +a human reading it must understand what was tested and what the evidence shows without +needing to re-run anything. + +## Inputs Expected + +Collect all of the following before generating the report: + +| Input | Source | Required | +| ------------------------------- | ------------------------------------------------------ | -------------------------------------------- | +| **Test execution output** | pytest stdout/stderr, exit code, per-test results | Yes | +| **Cluster verification report** | Output from the `cluster-verifier` agent | Yes | +| **Version information** | OCP version, MTV version, CNV version | Yes — report is NOT QUALIFIED if any missing | +| **Test plan reference** | Path or link to the test-plan.md that was executed | Yes | +| **Type** | `feature` or `bug` | Yes | +| **Bug details** (bugs only) | Bug ID/URL and whether the bug was FIXED or NOT FIXED | Yes when type is `bug` | +| **Source provider info** | Provider type and version (e.g., vSphere 8.0) | Yes | +| **Cluster API URL** | API endpoint of the target cluster | Yes | + +## Proof Report Template + +Generate `proof.md` using this exact structure. Replace every `` with real data. + +````markdown +# Qualification Proof Report + +## Summary +- **Type**: Feature / Bug Verification +- **Name**: +- **Result**: ✅ QUALIFIED / ❌ NOT QUALIFIED / 🐛 BUG FIXED / 🐛 BUG NOT FIXED +- **Date**: +- **Test Plan**: [test-plan.md]() + +## Environment +| Component | Version | +|-----------|---------| +| OpenShift | X.Y.Z | +| MTV | X.Y.Z | +| CNV | X.Y.Z | +| Source Provider | | +| Cluster API | | + +## Test Execution Results +| Test | Result | Duration | Notes | +|------|--------|----------|-------| +| test_create_storagemap | ✅ PASSED | 2.3s | | +| test_create_networkmap | ✅ PASSED | 1.8s | | +| test_create_plan | ✅ PASSED | 3.1s | | +| test_migrate_vms | ✅ PASSED | 120.5s | | +| test_check_vms | ✅ PASSED | 45.2s | | + +### Test Output +
Full pytest output + +``` + +``` + +
+ +## Cluster Verification +Independent verification performed after test execution. + +| Check | Resource | Status | Evidence | +|-------|----------|--------|----------| +| VM Exists | vm/ | ✅ PASS | | +| VM Running | vm/ | ✅ PASS | status.ready=true | +| Disks Bound | pvc/ | ✅ PASS | phase=Bound | +| Plan Succeeded | plan/ | ✅ PASS | status=Succeeded | +| Migration Completed | migration/ | ✅ PASS | | + +### Raw Evidence +
VM YAML + +```yaml + +``` + +
+ +
Plan Status + +```yaml + +``` + +
+ +## Qualification Decision + +### Criteria Met +- [x] All tests passed (exit code 0) +- [x] All cluster verifications passed +- [x] Test scenarios match test plan expectations +- [x] Evidence collected for all verification points + +### Verdict +**✅ QUALIFIED** — All tests passed with cluster verification proof. +```` + +## Rules + +### Qualification Logic + +1. **NEVER** mark as `✅ QUALIFIED` if any test failed (exit code ≠ 0 or any individual test result is not PASSED). +2. **NEVER** mark as `✅ QUALIFIED` if cluster verification has any `❌ FAIL` check. +3. If both tests and cluster verification pass → `✅ QUALIFIED`. +4. If any test or verification fails → `❌ NOT QUALIFIED`. Include a clear **Reason** line under the verdict explaining which checks failed. + +### Bug Verification Logic + +- For `bug` type reports, the verdict is `🐛 BUG FIXED` or `🐛 BUG NOT FIXED` instead of QUALIFIED/NOT QUALIFIED. +- `🐛 BUG FIXED` — all tests pass, cluster verification confirms the fix, and the behavior described in the bug no longer reproduces. +- `🐛 BUG NOT FIXED` — tests fail or cluster verification shows the buggy behavior still present. +- Always include the bug ID/URL in the Summary section and reference the specific evidence that proves or disproves the fix. + +### Evidence Requirements + +- **Always** include raw evidence (YAML, logs) in collapsible `
` sections. +- **Versions are MANDATORY.** If any version (OCP, MTV, or CNV) is missing, mark the report as `❌ NOT QUALIFIED` with reason: `"Missing version information"`. +- The report must be **self-contained**. A reader must understand what was tested, what passed or failed, and what the cluster state looked like — all from the proof.md alone. + +### Formatting + +- Use `✅ PASSED` / `❌ FAILED` for individual test results. +- Use `✅ PASS` / `❌ FAIL` for cluster verification checks. +- Durations should be in seconds with one decimal place (e.g., `2.3s`). +- The Date field must be ISO 8601 format with timezone. +- Unchecked criteria boxes (`- [ ]`) must appear for any criteria not met, with an explanation. + +### Handling Failures + +When the verdict is NOT QUALIFIED or BUG NOT FIXED, add a `### Failure Details` section before the Verdict: + +```markdown +### Failure Details +| Failed Item | Type | Details | +|-------------|------|---------| +| test_migrate_vms | Test | TimeoutError after 3600s | +| VM Running | Cluster Check | status.ready=false, phase=Scheduling | + +### Verdict +**❌ NOT QUALIFIED** — 1 test failed, 1 cluster verification failed. See Failure Details above. +``` + +## Output Location + +Write the generated proof report to: + +- **Features**: `.qualify/features//proof.md` +- **Bugs**: `.qualify/bugs//proof.md` + +The directory must match the directory used by the test plan. If the test plan lives at `.qualify/features/cold-migration/test-plan.md`, then the proof goes to `.qualify/features/cold-migration/proof.md`. diff --git a/llm/qualify/templates/.gitkeep b/llm/qualify/templates/.gitkeep new file mode 100644 index 00000000..e69de29b diff --git a/llm/qualify/templates/proof-template.md b/llm/qualify/templates/proof-template.md new file mode 100644 index 00000000..0187fe2b --- /dev/null +++ b/llm/qualify/templates/proof-template.md @@ -0,0 +1,67 @@ +# Qualification Proof Report + +## Summary + +- **Type**: `` +- **Name**: `` +- **Result**: <✅ QUALIFIED / ❌ NOT QUALIFIED / 🐛 BUG FIXED / 🐛 BUG NOT FIXED> +- **Date**: +- **Test Plan**: `` + +## Environment + +| Component | Version | +| --------------- | ------------------ | +| OpenShift | | +| MTV | | +| CNV | | +| Source Provider | `` | +| Cluster API | | + +## Test Execution Results + +| Test | Result | Duration | Notes | +| ---- | ------ | -------- | ----- | +| | | | | + +### Test Output + +
Full pytest output + +```text + +``` + +
+ +## Cluster Verification + +Independent verification performed after test execution. + +| Check | Resource | Status | Evidence | +| ----- | -------- | ------ | -------- | +| | | | | + +### Raw Evidence + +
Resource details + +```yaml + +``` + +
+ +## Qualification Decision + +### Criteria Met + +- [ ] All tests passed (exit code 0) +- [ ] All cluster verifications passed +- [ ] Test scenarios match test plan expectations +- [ ] Evidence collected for all verification points +- [ ] Versions recorded + +### Verdict + + diff --git a/llm/qualify/templates/test-plan-template.md b/llm/qualify/templates/test-plan-template.md new file mode 100644 index 00000000..f46d6112 --- /dev/null +++ b/llm/qualify/templates/test-plan-template.md @@ -0,0 +1,97 @@ +# Test Plan: + +## Overview + +**Type**: Feature / Bug Verification +**Source**: `` +**Date**: `` +**Author**: AI-generated, human-reviewed + +### Description + +`` + +## Prerequisites + +### Cluster Requirements + +- OpenShift version: `` +- MTV version: `` +- CNV installed: Yes/No + +### Provider Requirements + +- Source provider type: `` +- Source provider version: `` +- Provider credentials: `` + +### VM Requirements + +| VM Name | OS | Power State | Guest Agent | Disk Type | Special Config | +| -------- | ----- | ----------- | ----------- | ---------- | -------------- | +| `` | ``| on/off | Yes/No | thin/thick | `` | + +## Test Scenarios + +### Scenario 1: + +**Description**: `` + +**Test Pattern**: 5-step / 6-step shared-disk / 6-step copy-offload + +**Steps**: + +1. Create StorageMap with `` +2. Create NetworkMap with `` +3. Create Plan with `` +4. Execute migration +5. Verify migrated VMs + +**Expected Outcomes**: + +- `` +- `` + +**Cluster Verification Points**: + +| What to Check | How to Check | Expected Value | +| ------------------ | ---------------------------------------------------------------------------------------- | -------------------------- | +| VM exists, running | `oc get vm -n ` | status.ready=true | +| Disks attached | `oc get pvc -n ` | All PVCs Bound | +| Network configured | `oc get vm -n -o jsonpath='{.spec.template.spec.domain.devices.interfaces}'` | Correct network interfaces | +| `` | `` | `` | + +### Scenario 2: `` + +... + +## Test Configuration + +### tests_params entry + +```python +"test_": { + "virtual_machines": [ + { + "name": "", + "source_vm_power": "on", + "guest_agent": True, + }, + ], + "warm_migration": False, +}, +``` + +### Pytest Markers + +- `@pytest.mark.` — `` + +### Test File Location + +`tests//test__migration.py` + +## Risk Assessment + +| Risk | Impact | Mitigation | +| -------------------- | ---------- | ------------------ | +| `` | `` | `` | From bcbca3cd7698fd5c22f19f0638a4238af468a6b4 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Wed, 6 May 2026 16:21:24 +0300 Subject: [PATCH 02/13] docs: add workflow diagrams for /qualify qualification workflow --- llm/qualify/workflow-diagrams.md | 470 +++++++++++++++++++++++++++++++ 1 file changed, 470 insertions(+) create mode 100644 llm/qualify/workflow-diagrams.md diff --git a/llm/qualify/workflow-diagrams.md b/llm/qualify/workflow-diagrams.md new file mode 100644 index 00000000..8c91983b --- /dev/null +++ b/llm/qualify/workflow-diagrams.md @@ -0,0 +1,470 @@ +# `/qualify` — AI Qualification Workflow Diagrams + +The `/qualify` workflow automates end-to-end qualification of MTV features and bug fixes: +from reading a design doc or bug report, through writing and executing tests on a real OpenShift cluster, +to producing a verified proof report and PR. +This document visualizes the workflow's phases, component relationships, and inter-agent communication. + +--- + +## 1. Main Workflow Flowchart + +Four phases from setup to proof, showing the feature vs. bug split, human checkpoints, agent delegations, decision points, and generated artifacts. + +```mermaid +flowchart TD + classDef human fill:#ff9f43,stroke:#e17055,color:#2d3436,font-weight:bold + classDef agent fill:#74b9ff,stroke:#0984e3,color:#2d3436 + classDef decision fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,font-weight:bold + classDef artifact fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,font-style:italic + classDef phase fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold + classDef fail fill:#ff7675,stroke:#d63031,color:#fff,font-weight:bold + classDef success fill:#55efc4,stroke:#00b894,color:#2d3436,font-weight:bold + + START(["/qualify --type --source --cluster --name"]):::success + + subgraph P0["Phase 0 — Parse Arguments & Setup"] + direction TB + P0_TITLE["⚙️ PHASE 0: SETUP"]:::phase + PARSE["Parse CLI arguments\n--type, --source, --cluster, --name"]:::agent + VALIDATE_ARGS{{"Required args\npresent?"}}:::decision + ASK_ARGS{{{"🛑 Ask user\nfor missing args"}}}:::human + CLUSTER_CHECK["Validate cluster connectivity\noc whoami · oc cluster-info"]:::agent + CLUSTER_OK{{"Cluster\nreachable?"}}:::decision + CLUSTER_FAIL{{{"🛑 Ask user\nto fix cluster"}}}:::human + VERSIONS["Collect environment versions\nOCP · MTV · CNV"]:::agent + MKDIR["Create output directory\n.qualify/‹type›/‹name›/"]:::agent + IS_BUG{{"--type\n== bug?"}}:::decision + BUG_ASK{{{"🛑 Permanent test\nor verify-only?"}}}:::human + SET_PERM["Set mode:\npermanent-test"]:::agent + SET_VERIFY["Set mode:\nverify-only"]:::agent + + P0_TITLE ~~~ PARSE + PARSE --> VALIDATE_ARGS + VALIDATE_ARGS -- "No" --> ASK_ARGS --> PARSE + VALIDATE_ARGS -- "Yes" --> CLUSTER_CHECK + CLUSTER_CHECK --> CLUSTER_OK + CLUSTER_OK -- "No" --> CLUSTER_FAIL --> CLUSTER_CHECK + CLUSTER_OK -- "Yes" --> VERSIONS --> MKDIR --> IS_BUG + IS_BUG -- "Yes" --> BUG_ASK + IS_BUG -- "No (feature)" --> P0_END + BUG_ASK -- "Permanent test" --> SET_PERM --> P0_END + BUG_ASK -- "Verify-only" --> SET_VERIFY --> P0_END + end + + P0_END((" ")) + + subgraph P1["Phase 1 — Test Plan"] + direction TB + P1_TITLE["📋 PHASE 1: TEST PLAN"]:::phase + FETCH["Fetch source material\nfetch_content (URL) · read (file)"]:::agent + DELEGATE_TP["Delegate to\ntest-planner agent"]:::agent + TP_READ["test-planner reads:\nAGENTS.md · tests/ · config.py\nutilities/ · templates"]:::agent + TP_PRODUCE["Produce structured\ntest-plan.md"]:::agent + TP_ARTIFACT[/"💾 .qualify/‹type›/‹name›/test-plan.md"/]:::artifact + HUMAN_REVIEW{{{"🛑 HUMAN CHECKPOINT\nReview test plan"}}}:::human + PLAN_OK{{"Plan\napproved?"}}:::decision + PLAN_FEEDBACK["Incorporate feedback\nupdate test-plan.md"]:::agent + + P1_TITLE ~~~ FETCH + FETCH --> DELEGATE_TP --> TP_READ --> TP_PRODUCE --> TP_ARTIFACT --> HUMAN_REVIEW + HUMAN_REVIEW --> PLAN_OK + PLAN_OK -- "Feedback" --> PLAN_FEEDBACK --> TP_PRODUCE + PLAN_OK -- "Approved ✅" --> P1_END + end + + P1_END((" ")) + + subgraph P2["Phase 2 — Write & Verify Tests"] + direction TB + P2_TITLE["🧪 PHASE 2: WRITE & VERIFY (Autonomous)"]:::phase + MODE_CHECK{{"Test\nmode?"}}:::decision + + subgraph PERM["Feature / Bug-Permanent-Test Path"] + direction TB + GIT_BRANCH["Create branch\nqualify/‹name›"]:::agent + WRITE_TESTS["Delegate to python-expert\nWrite tests (5/6-step pattern)\nConfig + fixtures + test file"]:::agent + RUN_TESTS["Run tests on real cluster\nuv run pytest … 2>&1 | tee test-output.log"]:::agent + TEST_LOG[/"💾 .qualify/‹type›/‹name›/test-output.log"/]:::artifact + DELEGATE_CV["Delegate to\ncluster-verifier agent"]:::agent + CV_CHECK["cluster-verifier independently\nchecks cluster state\nVM · PVC · Plan · Migration · Network"]:::agent + EVAL{{"Tests passed\nAND verification\npassed?"}}:::decision + ATTEMPT_COUNT{{"Attempt\n≤ 3?"}}:::decision + FIX_TESTS["python-expert\nfixes tests"]:::agent + STUCK_ASK{{{"🛑 AI stuck\nAsk user for guidance"}}}:::human + end + + subgraph VONLY["Bug Verify-Only Path"] + direction TB + WRITE_TEMP["Write temp test\nin /tmp/qualify-‹name›/"]:::agent + RUN_TEMP["Run temp test on cluster\nuv run pytest …"]:::agent + TEMP_LOG[/"💾 .qualify/bugs/‹name›/test-output.log"/]:::artifact + CV_TEMP["Delegate to\ncluster-verifier agent"]:::agent + EVAL_TEMP{{"Passed?"}}:::decision + ATTEMPT_TEMP{{"Attempt\n≤ 3?"}}:::decision + FIX_TEMP["Fix temp test"]:::agent + STUCK_TEMP{{{"🛑 AI stuck\nAsk user"}}}:::human + end + + P2_TITLE ~~~ MODE_CHECK + MODE_CHECK -- "feature / permanent" --> GIT_BRANCH + MODE_CHECK -- "verify-only" --> WRITE_TEMP + + GIT_BRANCH --> WRITE_TESTS --> RUN_TESTS --> TEST_LOG --> DELEGATE_CV --> CV_CHECK --> EVAL + EVAL -- "Yes ✅" --> P2_PASS_PERM + EVAL -- "No ❌" --> ATTEMPT_COUNT + ATTEMPT_COUNT -- "Yes" --> FIX_TESTS --> RUN_TESTS + ATTEMPT_COUNT -- "No (3 failures)" --> STUCK_ASK --> FIX_TESTS + + WRITE_TEMP --> RUN_TEMP --> TEMP_LOG --> CV_TEMP --> EVAL_TEMP + EVAL_TEMP -- "Yes ✅" --> P2_PASS_VONLY + EVAL_TEMP -- "No ❌" --> ATTEMPT_TEMP + ATTEMPT_TEMP -- "Yes" --> FIX_TEMP --> RUN_TEMP + ATTEMPT_TEMP -- "No (3 failures)" --> STUCK_TEMP --> FIX_TEMP + end + + P2_PASS_PERM((" ")) + P2_PASS_VONLY((" ")) + + subgraph P3["Phase 3 — Code Review & PR"] + direction TB + P3_TITLE["🔍 PHASE 3: CODE REVIEW & PR"]:::phase + REVIEW_PARALLEL["Run 3 parallel code reviewers\n quality · guidelines · security"]:::agent + REVIEW_ISSUES{{"Issues\nfound?"}}:::decision + FIX_ISSUES["Fix review findings"]:::agent + PRECOMMIT["Run pre-commit\npre-commit run --all-files"]:::agent + PRECOMMIT_OK{{"Pre-commit\npassed?"}}:::decision + FIX_PRECOMMIT["Fix formatting/linting"]:::agent + CREATE_PR["Delegate to github-expert\nCreate PR: [qualify] ‹type›: ‹name›"]:::agent + PR_ARTIFACT[/"💾 GitHub PR with proof link"/]:::artifact + PR_REVIEW{{{"🛑 HUMAN CHECKPOINT\nPR review"}}}:::human + + P3_TITLE ~~~ REVIEW_PARALLEL + REVIEW_PARALLEL --> REVIEW_ISSUES + REVIEW_ISSUES -- "Yes" --> FIX_ISSUES --> REVIEW_PARALLEL + REVIEW_ISSUES -- "No ✅" --> PRECOMMIT + PRECOMMIT --> PRECOMMIT_OK + PRECOMMIT_OK -- "No" --> FIX_PRECOMMIT --> PRECOMMIT + PRECOMMIT_OK -- "Yes ✅" --> CREATE_PR --> PR_ARTIFACT --> PR_REVIEW + end + + subgraph P4["Phase 4 — Generate Proof"] + direction TB + P4_TITLE["📜 PHASE 4: GENERATE PROOF"]:::phase + INVOKE_SKILL["Invoke proof-generator skill\nRead SKILL.md + proof-template.md"]:::agent + ASSEMBLE["Assemble proof.md\nTest results · Cluster evidence\nVersions · Raw YAML"]:::agent + PROOF_ARTIFACT[/"💾 .qualify/‹type›/‹name›/proof.md"/]:::artifact + VERDICT{{"Determine\nverdict"}}:::decision + V_QUAL["✅ QUALIFIED"]:::success + V_NOTQUAL["❌ NOT QUALIFIED"]:::fail + V_FIXED["🐛 BUG FIXED"]:::success + V_NOTFIXED["🐛 BUG NOT FIXED"]:::fail + SUMMARY["Print final summary\nType · Name · Result · Artifacts · Versions"]:::agent + + P4_TITLE ~~~ INVOKE_SKILL + INVOKE_SKILL --> ASSEMBLE --> PROOF_ARTIFACT --> VERDICT + VERDICT -- "Feature pass" --> V_QUAL --> SUMMARY + VERDICT -- "Feature fail" --> V_NOTQUAL --> SUMMARY + VERDICT -- "Bug pass" --> V_FIXED --> SUMMARY + VERDICT -- "Bug fail" --> V_NOTFIXED --> SUMMARY + end + + DONE(["🏁 Qualification Complete"]):::success + + START --> P0 + P0_END --> P1 + P1_END --> P2 + P2_PASS_PERM --> P3 + P2_PASS_VONLY --> P4 + PR_REVIEW --> P4 + SUMMARY --> DONE +``` + +--- + +## 2. Component Relationship Diagram + +How the prompt template, agents, skill, templates, and output artifacts relate to each other. + +```mermaid +flowchart LR + classDef prompt fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold + classDef agent fill:#74b9ff,stroke:#0984e3,color:#2d3436,font-weight:bold + classDef skill fill:#55efc4,stroke:#00b894,color:#2d3436,font-weight:bold + classDef template fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436 + classDef artifact fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,font-style:italic + classDef external fill:#fab1a0,stroke:#e17055,color:#2d3436 + classDef codebase fill:#fd79a8,stroke:#e84393,color:#fff + + subgraph ORCHESTRATOR["llm/qualify/prompts/"] + QUALIFY["qualify.md\n(Main Prompt Template)\nOrchestrates all 4 phases"]:::prompt + end + + subgraph AGENTS["llm/qualify/agents/"] + TP["test-planner.md\nReads docs → test plans"]:::agent + CV["cluster-verifier.md\nIndependent cluster verification"]:::agent + end + + subgraph SKILLS["llm/qualify/skills/"] + PG["proof-generator\nSKILL.md\nAssembles proof.md"]:::skill + end + + subgraph TEMPLATES["llm/qualify/templates/"] + TPL_PLAN["test-plan-template.md\nTest plan skeleton"]:::template + TPL_PROOF["proof-template.md\nProof report skeleton"]:::template + end + + subgraph OUTPUT[".qualify/‹type›/‹name›/"] + OUT_PLAN[/"test-plan.md"/]:::artifact + OUT_LOG[/"test-output.log"/]:::artifact + OUT_PROOF[/"proof.md"/]:::artifact + end + + subgraph EXTERNAL_AGENTS["External Agents\n(from pi-config / project)"] + PE["python-expert\nWrites test code"]:::external + GE["github-expert\nCreates PR"]:::external + CR["code-reviewers ×3\nquality · guidelines · security"]:::external + GITE["git-expert\nBranch management"]:::external + end + + subgraph CODEBASE["Project Codebase"] + AGENTS_MD["AGENTS.md"]:::codebase + TESTS["tests/‹feature›/"]:::codebase + CONFIG["tests/tests_config/config.py"]:::codebase + UTILS["utilities/"]:::codebase + end + + %% Orchestrator delegates to agents & skill + QUALIFY -- "delegates\n(Phase 1)" --> TP + QUALIFY -- "delegates\n(Phase 2)" --> CV + QUALIFY -- "invokes\n(Phase 4)" --> PG + QUALIFY -- "delegates\n(Phase 2)" --> PE + QUALIFY -- "delegates\n(Phase 3)" --> CR + QUALIFY -- "delegates\n(Phase 3)" --> GE + QUALIFY -- "delegates\n(Phase 2)" --> GITE + + %% Agents use templates + TP -- "uses as\noutput format" --> TPL_PLAN + PG -- "uses as\noutput format" --> TPL_PROOF + + %% Agents read codebase + TP -. "reads" .-> AGENTS_MD + TP -. "reads" .-> TESTS + TP -. "reads" .-> CONFIG + TP -. "reads" .-> UTILS + PE -. "reads" .-> AGENTS_MD + PE -. "reads" .-> TESTS + + %% Agents produce artifacts + TP -- "produces" --> OUT_PLAN + CV -- "feeds into" --> PG + PG -- "produces" --> OUT_PROOF + + %% Test run produces log + PE -- "test run\nproduces" --> OUT_LOG + + %% Data flows + OUT_PLAN -. "input to" .-> PE + OUT_LOG -. "input to" .-> PG + OUT_LOG -. "input to" .-> CV +``` + +--- + +## 3. Sequence Diagram + +Interaction timeline between the User, Orchestrator (`qualify.md`), and all agents/skills across the four phases. + +```mermaid +sequenceDiagram + box rgb(255, 245, 235) Human + actor User + end + box rgb(230, 240, 255) Orchestrator + participant Orch as qualify.md
(Orchestrator) + end + box rgb(220, 245, 255) Agents + participant TP as test-planner + participant PE as python-expert + participant CV as cluster-verifier + participant CR as code-reviewers
(×3 parallel) + participant GE as github-expert + end + box rgb(220, 255, 235) Skills + participant PG as proof-generator + end + box rgb(255, 230, 230) Cluster + participant K8s as OpenShift
Cluster + end + + Note over User,K8s: Phase 0 — Parse Arguments & Setup + + User ->>+ Orch: /qualify --type feature --source --cluster + Orch ->> Orch: Parse CLI arguments + alt Missing required args + Orch -->> User: Ask for missing arguments + User -->> Orch: Provide arguments + end + Orch ->>+ K8s: oc whoami · oc cluster-info + K8s -->>- Orch: Cluster identity & status + alt Cluster unreachable + Orch -->> User: 🛑 Cluster unreachable — please fix + User -->> Orch: Cluster fixed + Orch ->> K8s: Retry connectivity + end + Orch ->>+ K8s: Collect versions (OCP, MTV, CNV) + K8s -->>- Orch: Version strings + Orch ->> Orch: Create .qualify/‹type›/‹name›/ + opt type == bug + Orch -->> User: 🛑 Permanent test or verify-only? + User -->> Orch: Decision (permanent / verify-only) + end + + Note over User,K8s: Phase 1 — Test Plan + + Orch ->> Orch: Fetch source material (URL or file) + Orch ->>+ TP: Delegate: produce test plan + TP ->> TP: Read AGENTS.md, tests/, config.py,
utilities/, test-plan-template.md + TP ->> TP: Analyze source material + TP -->>- Orch: test-plan.md + + Orch ->> Orch: Save .qualify/‹type›/‹name›/test-plan.md + Orch -->> User: 🛑 HUMAN CHECKPOINT: Review test plan + + loop Until approved + User -->> Orch: Feedback or Approved + alt Feedback provided + Orch ->> TP: Update plan with feedback + TP -->> Orch: Revised test-plan.md + Orch -->> User: Updated plan — please re-review + end + end + + User -->> Orch: ✅ Plan approved + + Note over User,K8s: Phase 2 — Write & Verify Tests (Autonomous) + + alt Feature or Bug-Permanent-Test + Orch ->> Orch: git checkout -b qualify/‹name› + Orch ->>+ PE: Delegate: write tests per approved plan + PE ->> PE: Read AGENTS.md, follow 5/6-step pattern + PE ->> PE: Create config entry, fixtures, test file + PE -->>- Orch: Test code ready + + loop Until pass or stuck (max 3 retries) + Orch ->>+ K8s: uv run pytest … | tee test-output.log + K8s -->>- Orch: Test results + output + + Orch ->>+ CV: Delegate: verify cluster state + CV ->>+ K8s: oc get vm, pvc, plan, migration … + K8s -->>- CV: Resource states + YAML + CV ->> CV: Compare against test plan expectations + CV -->>- Orch: Verification report (PASS/FAIL per check) + + alt Tests PASS + Verification PASS + Note over Orch: ✅ Proceed to Phase 3 + else Tests FAIL or Verification FAIL + alt Attempt ≤ 3 + Orch ->>+ PE: Fix failing tests + PE -->>- Orch: Updated test code + else Attempt > 3 + Orch -->> User: 🛑 Stuck on: ‹problem› + User -->> Orch: Guidance + Orch ->> PE: Apply user guidance + end + end + end + + else Bug Verify-Only + Orch ->>+ PE: Write temp test in /tmp/qualify-‹name›/ + PE -->>- Orch: Temp test ready + + loop Until pass or stuck + Orch ->>+ K8s: Run temp test on cluster + K8s -->>- Orch: Test results + Orch ->>+ CV: Verify cluster state + CV ->>+ K8s: oc get … + K8s -->>- CV: Resource states + CV -->>- Orch: Verification report + alt FAIL & retries remain + Orch ->> PE: Fix temp test + else FAIL & stuck + Orch -->> User: 🛑 Stuck — need guidance + User -->> Orch: Guidance + end + end + Note over Orch: Skip Phase 3 → go to Phase 4 + end + + Note over User,K8s: Phase 3 — Code Review & PR (permanent tests only) + + alt Feature or Bug-Permanent-Test + par Quality Review + Orch ->>+ CR: code-reviewer-quality + CR -->>- Orch: Quality findings + and Guidelines Review + Orch ->>+ CR: code-reviewer-guidelines + CR -->>- Orch: Guidelines findings + and Security Review + Orch ->>+ CR: code-reviewer-security + CR -->>- Orch: Security findings + end + + loop Until no findings + alt Issues found + Orch ->> PE: Fix review issues + PE -->> Orch: Fixes applied + Orch ->> CR: Re-review + CR -->> Orch: Updated findings + end + end + + Orch ->> Orch: pre-commit run --all-files + loop Until pre-commit passes + alt Failures + Orch ->> Orch: Fix formatting/linting + end + end + + Orch ->>+ GE: Create PR: [qualify] ‹type›: ‹name› + GE -->>- Orch: PR URL + Orch -->> User: 🛑 PR ready for review + end + + Note over User,K8s: Phase 4 — Generate Proof + + Orch ->>+ PG: Assemble proof report + Note right of PG: Inputs:
• test-output.log
• cluster verification report
• OCP/MTV/CNV versions
• test-plan.md reference + PG ->> PG: Apply proof-template.md structure + PG ->> PG: Determine verdict:
QUALIFIED / NOT QUALIFIED /
BUG FIXED / BUG NOT FIXED + PG -->>- Orch: proof.md + + Orch ->> Orch: Write .qualify/‹type›/‹name›/proof.md + + Orch -->>- User: 🏁 Qualification Complete
Result + Artifacts + Versions +``` + +--- + +## Legend + +| Shape | Meaning | +| ------- | --------- | +| 🟪 Purple rounded | Phase header | +| 🟦 Blue rectangle | Agent / automated action | +| 🟧 Orange hexagon | 🛑 Human checkpoint — requires user input | +| 🟨 Yellow diamond | Decision point | +| ⬜ Gray parallelogram | Output artifact (file) | +| 🟩 Green rounded | Start / success outcome | +| 🟥 Red rounded | Failure outcome | +| 🟤 Pink | Codebase reference | +| 🔴 Coral | External agent (not in qualify/) | + +## Key Takeaways + +1. **Four distinct phases** with clear handoff boundaries. +2. **Human stays in the loop** at test-plan review, bug-mode decision, stuck escalation, and PR review — everything else is autonomous. +3. **Dual verification** — pytest execution alone is never sufficient; the `cluster-verifier` agent independently confirms cluster state. +4. **Bug workflows fork early** (Phase 0) into permanent-test vs. verify-only, rejoining at proof generation (Phase 4). +5. **Three parallel code reviewers** in Phase 3 ensure quality, guideline compliance, and security before any PR is created. +6. **Self-contained proof** — `proof.md` captures test results, cluster evidence, version info, and raw YAML so the qualification can be audited without re-running anything. From 7e0955384a3c112f7c032346b413c37a9f0527c5 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Wed, 6 May 2026 16:31:54 +0300 Subject: [PATCH 03/13] docs: split workflow diagrams into per-phase charts for readability --- llm/qualify/workflow-diagrams.md | 667 +++++++++++++------------------ 1 file changed, 286 insertions(+), 381 deletions(-) diff --git a/llm/qualify/workflow-diagrams.md b/llm/qualify/workflow-diagrams.md index 8c91983b..cf4cd419 100644 --- a/llm/qualify/workflow-diagrams.md +++ b/llm/qualify/workflow-diagrams.md @@ -7,184 +7,230 @@ This document visualizes the workflow's phases, component relationships, and int --- -## 1. Main Workflow Flowchart +## 1. High-Level Overview -Four phases from setup to proof, showing the feature vs. bug split, human checkpoints, agent delegations, decision points, and generated artifacts. +Five phases from setup to proof, showing the bug verify-only shortcut and human checkpoints. ```mermaid flowchart TD classDef human fill:#ff9f43,stroke:#e17055,color:#2d3436,font-weight:bold + classDef phase fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold + classDef success fill:#55efc4,stroke:#00b894,color:#2d3436,font-weight:bold + classDef decision fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,font-weight:bold + + START(["▶ /qualify"]):::success + P0["⚙️ Phase 0\nSetup"]:::phase + P1["📋 Phase 1\nTest Plan"]:::phase + P2["🧪 Phase 2\nWrite & Verify"]:::phase + P3["🔍 Phase 3\nReview & PR"]:::phase + P4["📜 Phase 4\nGenerate Proof"]:::phase + DONE(["🏁 Complete"]):::success + + H1{{"🛑 Human\nPlan Review"}}:::human + H2{{"🛑 Human\nPR Review"}}:::human + MODE{{"Bug\nverify-only?"}}:::decision + + START --> P0 --> P1 --> H1 --> P2 --> MODE + MODE -- "No" --> P3 --> H2 --> P4 + MODE -- "Yes\n(skip PR)" --> P4 + P4 --> DONE +``` + +--- + +## 2. Phase 0 — Setup + +Parse arguments, validate cluster, collect versions, determine bug mode. + +```mermaid +flowchart TD classDef agent fill:#74b9ff,stroke:#0984e3,color:#2d3436 + classDef human fill:#ff9f43,stroke:#e17055,color:#2d3436,font-weight:bold + classDef decision fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,font-weight:bold + classDef phase fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold + + TITLE["⚙️ PHASE 0: SETUP"]:::phase + PARSE["Parse CLI args"]:::agent + VALID{{"Args OK?"}}:::decision + ASK_ARGS{{"🛑 Ask user\nfor missing args"}}:::human + CLUSTER["Validate cluster"]:::agent + C_OK{{"Cluster\nreachable?"}}:::decision + C_FIX{{"🛑 Ask user\nto fix cluster"}}:::human + VERSIONS["Collect versions\nOCP · MTV · CNV"]:::agent + MKDIR["Create output dir"]:::agent + IS_BUG{{"--type\n== bug?"}}:::decision + BUG_ASK{{"🛑 Permanent\nor verify-only?"}}:::human + DONE((" ")) + + TITLE ~~~ PARSE + PARSE --> VALID + VALID -- "No" --> ASK_ARGS --> PARSE + VALID -- "Yes" --> CLUSTER --> C_OK + C_OK -- "No" --> C_FIX --> CLUSTER + C_OK -- "Yes" --> VERSIONS --> MKDIR --> IS_BUG + IS_BUG -- "No (feature)" --> DONE + IS_BUG -- "Yes" --> BUG_ASK --> DONE +``` + +--- + +## 3. Phase 1 — Test Plan + +Fetch source, delegate to test-planner, human review loop. + +```mermaid +flowchart TD + classDef agent fill:#74b9ff,stroke:#0984e3,color:#2d3436 + classDef human fill:#ff9f43,stroke:#e17055,color:#2d3436,font-weight:bold + classDef decision fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,font-weight:bold + classDef artifact fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,font-style:italic + classDef phase fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold + + TITLE["📋 PHASE 1: TEST PLAN"]:::phase + FETCH["Fetch source\nmaterial"]:::agent + DELEGATE["Delegate to\ntest-planner"]:::agent + PRODUCE["Produce\ntest-plan.md"]:::agent + SAVE[/"💾 test-plan.md"/]:::artifact + REVIEW{{"🛑 Human\nReview plan"}}:::human + OK{{"Approved?"}}:::decision + FEEDBACK["Incorporate\nfeedback"]:::agent + DONE((" ")) + + TITLE ~~~ FETCH + FETCH --> DELEGATE --> PRODUCE --> SAVE --> REVIEW --> OK + OK -- "Feedback" --> FEEDBACK --> PRODUCE + OK -- "Approved ✅" --> DONE +``` + +--- + +## 4. Phase 2 — Write & Verify Tests + +Two paths: feature/permanent-test (creates branch) vs. bug verify-only (temp test). Both run on a real cluster with cluster-verifier validation. + +```mermaid +flowchart TD + classDef agent fill:#74b9ff,stroke:#0984e3,color:#2d3436 + classDef human fill:#ff9f43,stroke:#e17055,color:#2d3436,font-weight:bold classDef decision fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,font-weight:bold classDef artifact fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,font-style:italic classDef phase fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold - classDef fail fill:#ff7675,stroke:#d63031,color:#fff,font-weight:bold classDef success fill:#55efc4,stroke:#00b894,color:#2d3436,font-weight:bold - START(["/qualify --type --source --cluster --name"]):::success - - subgraph P0["Phase 0 — Parse Arguments & Setup"] - direction TB - P0_TITLE["⚙️ PHASE 0: SETUP"]:::phase - PARSE["Parse CLI arguments\n--type, --source, --cluster, --name"]:::agent - VALIDATE_ARGS{{"Required args\npresent?"}}:::decision - ASK_ARGS{{{"🛑 Ask user\nfor missing args"}}}:::human - CLUSTER_CHECK["Validate cluster connectivity\noc whoami · oc cluster-info"]:::agent - CLUSTER_OK{{"Cluster\nreachable?"}}:::decision - CLUSTER_FAIL{{{"🛑 Ask user\nto fix cluster"}}}:::human - VERSIONS["Collect environment versions\nOCP · MTV · CNV"]:::agent - MKDIR["Create output directory\n.qualify/‹type›/‹name›/"]:::agent - IS_BUG{{"--type\n== bug?"}}:::decision - BUG_ASK{{{"🛑 Permanent test\nor verify-only?"}}}:::human - SET_PERM["Set mode:\npermanent-test"]:::agent - SET_VERIFY["Set mode:\nverify-only"]:::agent - - P0_TITLE ~~~ PARSE - PARSE --> VALIDATE_ARGS - VALIDATE_ARGS -- "No" --> ASK_ARGS --> PARSE - VALIDATE_ARGS -- "Yes" --> CLUSTER_CHECK - CLUSTER_CHECK --> CLUSTER_OK - CLUSTER_OK -- "No" --> CLUSTER_FAIL --> CLUSTER_CHECK - CLUSTER_OK -- "Yes" --> VERSIONS --> MKDIR --> IS_BUG - IS_BUG -- "Yes" --> BUG_ASK - IS_BUG -- "No (feature)" --> P0_END - BUG_ASK -- "Permanent test" --> SET_PERM --> P0_END - BUG_ASK -- "Verify-only" --> SET_VERIFY --> P0_END - end + TITLE["🧪 PHASE 2: WRITE & VERIFY"]:::phase + MODE{{"Test mode?"}}:::decision + + TITLE ~~~ MODE + + %% Feature / permanent path + BRANCH["Create branch\nqualify/‹name›"]:::agent + WRITE["python-expert\nwrites tests"]:::agent + RUN["Run pytest\non cluster"]:::agent + LOG1[/"💾 test-output.log"/]:::artifact + CV1["cluster-verifier\nchecks state"]:::agent + PASS1{{"Passed?"}}:::decision + RETRY1{{"Attempt\n≤ 3?"}}:::decision + FIX1["Fix tests"]:::agent + STUCK1{{"🛑 Ask user\nfor guidance"}}:::human + DONE1(["→ Phase 3"]):::success + + MODE -- "Feature /\npermanent" --> BRANCH --> WRITE --> RUN --> LOG1 --> CV1 --> PASS1 + PASS1 -- "Yes ✅" --> DONE1 + PASS1 -- "No ❌" --> RETRY1 + RETRY1 -- "Yes" --> FIX1 --> RUN + RETRY1 -- "No" --> STUCK1 --> FIX1 + + %% Verify-only path + WTEMP["Write temp test\nin /tmp/"]:::agent + RTEMP["Run temp test\non cluster"]:::agent + LOG2[/"💾 test-output.log"/]:::artifact + CV2["cluster-verifier\nchecks state"]:::agent + PASS2{{"Passed?"}}:::decision + RETRY2{{"Attempt\n≤ 3?"}}:::decision + FIX2["Fix temp test"]:::agent + STUCK2{{"🛑 Ask user\nfor guidance"}}:::human + DONE2(["→ Phase 4\n(skip PR)"]):::success + + MODE -- "Verify-only" --> WTEMP --> RTEMP --> LOG2 --> CV2 --> PASS2 + PASS2 -- "Yes ✅" --> DONE2 + PASS2 -- "No ❌" --> RETRY2 + RETRY2 -- "Yes" --> FIX2 --> RTEMP + RETRY2 -- "No" --> STUCK2 --> FIX2 +``` - P0_END((" ")) - - subgraph P1["Phase 1 — Test Plan"] - direction TB - P1_TITLE["📋 PHASE 1: TEST PLAN"]:::phase - FETCH["Fetch source material\nfetch_content (URL) · read (file)"]:::agent - DELEGATE_TP["Delegate to\ntest-planner agent"]:::agent - TP_READ["test-planner reads:\nAGENTS.md · tests/ · config.py\nutilities/ · templates"]:::agent - TP_PRODUCE["Produce structured\ntest-plan.md"]:::agent - TP_ARTIFACT[/"💾 .qualify/‹type›/‹name›/test-plan.md"/]:::artifact - HUMAN_REVIEW{{{"🛑 HUMAN CHECKPOINT\nReview test plan"}}}:::human - PLAN_OK{{"Plan\napproved?"}}:::decision - PLAN_FEEDBACK["Incorporate feedback\nupdate test-plan.md"]:::agent - - P1_TITLE ~~~ FETCH - FETCH --> DELEGATE_TP --> TP_READ --> TP_PRODUCE --> TP_ARTIFACT --> HUMAN_REVIEW - HUMAN_REVIEW --> PLAN_OK - PLAN_OK -- "Feedback" --> PLAN_FEEDBACK --> TP_PRODUCE - PLAN_OK -- "Approved ✅" --> P1_END - end +--- - P1_END((" ")) - - subgraph P2["Phase 2 — Write & Verify Tests"] - direction TB - P2_TITLE["🧪 PHASE 2: WRITE & VERIFY (Autonomous)"]:::phase - MODE_CHECK{{"Test\nmode?"}}:::decision - - subgraph PERM["Feature / Bug-Permanent-Test Path"] - direction TB - GIT_BRANCH["Create branch\nqualify/‹name›"]:::agent - WRITE_TESTS["Delegate to python-expert\nWrite tests (5/6-step pattern)\nConfig + fixtures + test file"]:::agent - RUN_TESTS["Run tests on real cluster\nuv run pytest … 2>&1 | tee test-output.log"]:::agent - TEST_LOG[/"💾 .qualify/‹type›/‹name›/test-output.log"/]:::artifact - DELEGATE_CV["Delegate to\ncluster-verifier agent"]:::agent - CV_CHECK["cluster-verifier independently\nchecks cluster state\nVM · PVC · Plan · Migration · Network"]:::agent - EVAL{{"Tests passed\nAND verification\npassed?"}}:::decision - ATTEMPT_COUNT{{"Attempt\n≤ 3?"}}:::decision - FIX_TESTS["python-expert\nfixes tests"]:::agent - STUCK_ASK{{{"🛑 AI stuck\nAsk user for guidance"}}}:::human - end - - subgraph VONLY["Bug Verify-Only Path"] - direction TB - WRITE_TEMP["Write temp test\nin /tmp/qualify-‹name›/"]:::agent - RUN_TEMP["Run temp test on cluster\nuv run pytest …"]:::agent - TEMP_LOG[/"💾 .qualify/bugs/‹name›/test-output.log"/]:::artifact - CV_TEMP["Delegate to\ncluster-verifier agent"]:::agent - EVAL_TEMP{{"Passed?"}}:::decision - ATTEMPT_TEMP{{"Attempt\n≤ 3?"}}:::decision - FIX_TEMP["Fix temp test"]:::agent - STUCK_TEMP{{{"🛑 AI stuck\nAsk user"}}}:::human - end - - P2_TITLE ~~~ MODE_CHECK - MODE_CHECK -- "feature / permanent" --> GIT_BRANCH - MODE_CHECK -- "verify-only" --> WRITE_TEMP - - GIT_BRANCH --> WRITE_TESTS --> RUN_TESTS --> TEST_LOG --> DELEGATE_CV --> CV_CHECK --> EVAL - EVAL -- "Yes ✅" --> P2_PASS_PERM - EVAL -- "No ❌" --> ATTEMPT_COUNT - ATTEMPT_COUNT -- "Yes" --> FIX_TESTS --> RUN_TESTS - ATTEMPT_COUNT -- "No (3 failures)" --> STUCK_ASK --> FIX_TESTS - - WRITE_TEMP --> RUN_TEMP --> TEMP_LOG --> CV_TEMP --> EVAL_TEMP - EVAL_TEMP -- "Yes ✅" --> P2_PASS_VONLY - EVAL_TEMP -- "No ❌" --> ATTEMPT_TEMP - ATTEMPT_TEMP -- "Yes" --> FIX_TEMP --> RUN_TEMP - ATTEMPT_TEMP -- "No (3 failures)" --> STUCK_TEMP --> FIX_TEMP - end +## 5. Phase 3 — Code Review & PR - P2_PASS_PERM((" ")) - P2_PASS_VONLY((" ")) - - subgraph P3["Phase 3 — Code Review & PR"] - direction TB - P3_TITLE["🔍 PHASE 3: CODE REVIEW & PR"]:::phase - REVIEW_PARALLEL["Run 3 parallel code reviewers\n quality · guidelines · security"]:::agent - REVIEW_ISSUES{{"Issues\nfound?"}}:::decision - FIX_ISSUES["Fix review findings"]:::agent - PRECOMMIT["Run pre-commit\npre-commit run --all-files"]:::agent - PRECOMMIT_OK{{"Pre-commit\npassed?"}}:::decision - FIX_PRECOMMIT["Fix formatting/linting"]:::agent - CREATE_PR["Delegate to github-expert\nCreate PR: [qualify] ‹type›: ‹name›"]:::agent - PR_ARTIFACT[/"💾 GitHub PR with proof link"/]:::artifact - PR_REVIEW{{{"🛑 HUMAN CHECKPOINT\nPR review"}}}:::human - - P3_TITLE ~~~ REVIEW_PARALLEL - REVIEW_PARALLEL --> REVIEW_ISSUES - REVIEW_ISSUES -- "Yes" --> FIX_ISSUES --> REVIEW_PARALLEL - REVIEW_ISSUES -- "No ✅" --> PRECOMMIT - PRECOMMIT --> PRECOMMIT_OK - PRECOMMIT_OK -- "No" --> FIX_PRECOMMIT --> PRECOMMIT - PRECOMMIT_OK -- "Yes ✅" --> CREATE_PR --> PR_ARTIFACT --> PR_REVIEW - end +Three parallel reviewers, fix loop, pre-commit, then PR creation. - subgraph P4["Phase 4 — Generate Proof"] - direction TB - P4_TITLE["📜 PHASE 4: GENERATE PROOF"]:::phase - INVOKE_SKILL["Invoke proof-generator skill\nRead SKILL.md + proof-template.md"]:::agent - ASSEMBLE["Assemble proof.md\nTest results · Cluster evidence\nVersions · Raw YAML"]:::agent - PROOF_ARTIFACT[/"💾 .qualify/‹type›/‹name›/proof.md"/]:::artifact - VERDICT{{"Determine\nverdict"}}:::decision - V_QUAL["✅ QUALIFIED"]:::success - V_NOTQUAL["❌ NOT QUALIFIED"]:::fail - V_FIXED["🐛 BUG FIXED"]:::success - V_NOTFIXED["🐛 BUG NOT FIXED"]:::fail - SUMMARY["Print final summary\nType · Name · Result · Artifacts · Versions"]:::agent - - P4_TITLE ~~~ INVOKE_SKILL - INVOKE_SKILL --> ASSEMBLE --> PROOF_ARTIFACT --> VERDICT - VERDICT -- "Feature pass" --> V_QUAL --> SUMMARY - VERDICT -- "Feature fail" --> V_NOTQUAL --> SUMMARY - VERDICT -- "Bug pass" --> V_FIXED --> SUMMARY - VERDICT -- "Bug fail" --> V_NOTFIXED --> SUMMARY - end +```mermaid +flowchart TD + classDef agent fill:#74b9ff,stroke:#0984e3,color:#2d3436 + classDef human fill:#ff9f43,stroke:#e17055,color:#2d3436,font-weight:bold + classDef decision fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,font-weight:bold + classDef artifact fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,font-style:italic + classDef phase fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold + + TITLE["🔍 PHASE 3: REVIEW & PR"]:::phase + REVIEW["3 parallel reviewers\nquality · guidelines\n· security"]:::agent + ISSUES{{"Issues\nfound?"}}:::decision + FIX["Fix findings"]:::agent + PRECOMMIT["pre-commit\nrun --all-files"]:::agent + PC_OK{{"Passed?"}}:::decision + FIX_PC["Fix lint/format"]:::agent + PR["github-expert\ncreates PR"]:::agent + PR_ART[/"💾 GitHub PR"/]:::artifact + HUMAN{{"🛑 Human\nPR review"}}:::human + DONE((" ")) + + TITLE ~~~ REVIEW + REVIEW --> ISSUES + ISSUES -- "Yes" --> FIX --> REVIEW + ISSUES -- "No ✅" --> PRECOMMIT --> PC_OK + PC_OK -- "No" --> FIX_PC --> PRECOMMIT + PC_OK -- "Yes ✅" --> PR --> PR_ART --> HUMAN --> DONE +``` + +--- + +## 6. Phase 4 — Generate Proof + +Assemble proof report, determine verdict, print summary. - DONE(["🏁 Qualification Complete"]):::success +```mermaid +flowchart TD + classDef agent fill:#74b9ff,stroke:#0984e3,color:#2d3436 + classDef decision fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436,font-weight:bold + classDef artifact fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,font-style:italic + classDef phase fill:#a29bfe,stroke:#6c5ce7,color:#fff,font-weight:bold + classDef success fill:#55efc4,stroke:#00b894,color:#2d3436,font-weight:bold + classDef fail fill:#ff7675,stroke:#d63031,color:#fff,font-weight:bold - START --> P0 - P0_END --> P1 - P1_END --> P2 - P2_PASS_PERM --> P3 - P2_PASS_VONLY --> P4 - PR_REVIEW --> P4 - SUMMARY --> DONE + TITLE["📜 PHASE 4: PROOF"]:::phase + INVOKE["Invoke\nproof-generator"]:::agent + ASSEMBLE["Assemble\nproof.md"]:::agent + PROOF[/"💾 proof.md"/]:::artifact + VERDICT{{"Verdict?"}}:::decision + QUAL["✅ QUALIFIED"]:::success + NOTQUAL["❌ NOT QUALIFIED"]:::fail + FIXED["🐛 BUG FIXED"]:::success + NOTFIXED["🐛 NOT FIXED"]:::fail + SUMMARY["Print summary"]:::agent + + TITLE ~~~ INVOKE + INVOKE --> ASSEMBLE --> PROOF --> VERDICT + VERDICT -- "Feature pass" --> QUAL --> SUMMARY + VERDICT -- "Feature fail" --> NOTQUAL --> SUMMARY + VERDICT -- "Bug pass" --> FIXED --> SUMMARY + VERDICT -- "Bug fail" --> NOTFIXED --> SUMMARY ``` --- -## 2. Component Relationship Diagram +## 7. Component Relationship Diagram -How the prompt template, agents, skill, templates, and output artifacts relate to each other. +How the orchestrator, agents, skills, templates, and artifacts relate to each other. ```mermaid flowchart LR @@ -194,254 +240,114 @@ flowchart LR classDef template fill:#ffeaa7,stroke:#fdcb6e,color:#2d3436 classDef artifact fill:#dfe6e9,stroke:#b2bec3,color:#2d3436,font-style:italic classDef external fill:#fab1a0,stroke:#e17055,color:#2d3436 - classDef codebase fill:#fd79a8,stroke:#e84393,color:#fff - subgraph ORCHESTRATOR["llm/qualify/prompts/"] - QUALIFY["qualify.md\n(Main Prompt Template)\nOrchestrates all 4 phases"]:::prompt + subgraph ORCH["Orchestrator"] + Q["qualify.md"]:::prompt end - subgraph AGENTS["llm/qualify/agents/"] - TP["test-planner.md\nReads docs → test plans"]:::agent - CV["cluster-verifier.md\nIndependent cluster verification"]:::agent + subgraph AGENTS["Qualify Agents"] + TP["test-planner"]:::agent + CV["cluster-verifier"]:::agent end - subgraph SKILLS["llm/qualify/skills/"] - PG["proof-generator\nSKILL.md\nAssembles proof.md"]:::skill + subgraph EXT["External Agents"] + PE["python-expert"]:::external + CR["code-reviewers ×3"]:::external + GE["github-expert"]:::external end - subgraph TEMPLATES["llm/qualify/templates/"] - TPL_PLAN["test-plan-template.md\nTest plan skeleton"]:::template - TPL_PROOF["proof-template.md\nProof report skeleton"]:::template + subgraph SKILL["Qualify Skills"] + PG["proof-generator"]:::skill end - subgraph OUTPUT[".qualify/‹type›/‹name›/"] - OUT_PLAN[/"test-plan.md"/]:::artifact - OUT_LOG[/"test-output.log"/]:::artifact - OUT_PROOF[/"proof.md"/]:::artifact + subgraph TPL["Templates"] + T1["test-plan-\ntemplate.md"]:::template + T2["proof-\ntemplate.md"]:::template end - subgraph EXTERNAL_AGENTS["External Agents\n(from pi-config / project)"] - PE["python-expert\nWrites test code"]:::external - GE["github-expert\nCreates PR"]:::external - CR["code-reviewers ×3\nquality · guidelines · security"]:::external - GITE["git-expert\nBranch management"]:::external + subgraph OUT["Output Artifacts"] + O1[/"test-plan.md"/]:::artifact + O2[/"test-output.log"/]:::artifact + O3[/"proof.md"/]:::artifact end - subgraph CODEBASE["Project Codebase"] - AGENTS_MD["AGENTS.md"]:::codebase - TESTS["tests/‹feature›/"]:::codebase - CONFIG["tests/tests_config/config.py"]:::codebase - UTILS["utilities/"]:::codebase - end + Q -- "Phase 1" --> TP + Q -- "Phase 2" --> PE + Q -- "Phase 2" --> CV + Q -- "Phase 3" --> CR + Q -- "Phase 3" --> GE + Q -- "Phase 4" --> PG - %% Orchestrator delegates to agents & skill - QUALIFY -- "delegates\n(Phase 1)" --> TP - QUALIFY -- "delegates\n(Phase 2)" --> CV - QUALIFY -- "invokes\n(Phase 4)" --> PG - QUALIFY -- "delegates\n(Phase 2)" --> PE - QUALIFY -- "delegates\n(Phase 3)" --> CR - QUALIFY -- "delegates\n(Phase 3)" --> GE - QUALIFY -- "delegates\n(Phase 2)" --> GITE - - %% Agents use templates - TP -- "uses as\noutput format" --> TPL_PLAN - PG -- "uses as\noutput format" --> TPL_PROOF - - %% Agents read codebase - TP -. "reads" .-> AGENTS_MD - TP -. "reads" .-> TESTS - TP -. "reads" .-> CONFIG - TP -. "reads" .-> UTILS - PE -. "reads" .-> AGENTS_MD - PE -. "reads" .-> TESTS - - %% Agents produce artifacts - TP -- "produces" --> OUT_PLAN - CV -- "feeds into" --> PG - PG -- "produces" --> OUT_PROOF - - %% Test run produces log - PE -- "test run\nproduces" --> OUT_LOG - - %% Data flows - OUT_PLAN -. "input to" .-> PE - OUT_LOG -. "input to" .-> PG - OUT_LOG -. "input to" .-> CV + TP --> T1 + PG --> T2 + + TP --> O1 + PE --> O2 + PG --> O3 + + O1 --> PE + O2 --> CV + O2 --> PG ``` --- -## 3. Sequence Diagram +## 8. Sequence Diagrams + +Happy-path interaction split into two diagrams for readability. -Interaction timeline between the User, Orchestrator (`qualify.md`), and all agents/skills across the four phases. +### 8a. Setup, Plan & Write (Phases 0–2) ```mermaid sequenceDiagram - box rgb(255, 245, 235) Human - actor User - end - box rgb(230, 240, 255) Orchestrator - participant Orch as qualify.md
(Orchestrator) - end - box rgb(220, 245, 255) Agents - participant TP as test-planner - participant PE as python-expert - participant CV as cluster-verifier - participant CR as code-reviewers
(×3 parallel) - participant GE as github-expert - end - box rgb(220, 255, 235) Skills - participant PG as proof-generator - end - box rgb(255, 230, 230) Cluster - participant K8s as OpenShift
Cluster - end - - Note over User,K8s: Phase 0 — Parse Arguments & Setup - - User ->>+ Orch: /qualify --type feature --source --cluster - Orch ->> Orch: Parse CLI arguments - alt Missing required args - Orch -->> User: Ask for missing arguments - User -->> Orch: Provide arguments - end - Orch ->>+ K8s: oc whoami · oc cluster-info - K8s -->>- Orch: Cluster identity & status - alt Cluster unreachable - Orch -->> User: 🛑 Cluster unreachable — please fix - User -->> Orch: Cluster fixed - Orch ->> K8s: Retry connectivity - end - Orch ->>+ K8s: Collect versions (OCP, MTV, CNV) - K8s -->>- Orch: Version strings - Orch ->> Orch: Create .qualify/‹type›/‹name›/ - opt type == bug - Orch -->> User: 🛑 Permanent test or verify-only? - User -->> Orch: Decision (permanent / verify-only) - end - - Note over User,K8s: Phase 1 — Test Plan - - Orch ->> Orch: Fetch source material (URL or file) - Orch ->>+ TP: Delegate: produce test plan - TP ->> TP: Read AGENTS.md, tests/, config.py,
utilities/, test-plan-template.md - TP ->> TP: Analyze source material - TP -->>- Orch: test-plan.md - - Orch ->> Orch: Save .qualify/‹type›/‹name›/test-plan.md - Orch -->> User: 🛑 HUMAN CHECKPOINT: Review test plan - - loop Until approved - User -->> Orch: Feedback or Approved - alt Feedback provided - Orch ->> TP: Update plan with feedback - TP -->> Orch: Revised test-plan.md - Orch -->> User: Updated plan — please re-review - end - end - - User -->> Orch: ✅ Plan approved - - Note over User,K8s: Phase 2 — Write & Verify Tests (Autonomous) - - alt Feature or Bug-Permanent-Test - Orch ->> Orch: git checkout -b qualify/‹name› - Orch ->>+ PE: Delegate: write tests per approved plan - PE ->> PE: Read AGENTS.md, follow 5/6-step pattern - PE ->> PE: Create config entry, fixtures, test file - PE -->>- Orch: Test code ready - - loop Until pass or stuck (max 3 retries) - Orch ->>+ K8s: uv run pytest … | tee test-output.log - K8s -->>- Orch: Test results + output - - Orch ->>+ CV: Delegate: verify cluster state - CV ->>+ K8s: oc get vm, pvc, plan, migration … - K8s -->>- CV: Resource states + YAML - CV ->> CV: Compare against test plan expectations - CV -->>- Orch: Verification report (PASS/FAIL per check) - - alt Tests PASS + Verification PASS - Note over Orch: ✅ Proceed to Phase 3 - else Tests FAIL or Verification FAIL - alt Attempt ≤ 3 - Orch ->>+ PE: Fix failing tests - PE -->>- Orch: Updated test code - else Attempt > 3 - Orch -->> User: 🛑 Stuck on: ‹problem› - User -->> Orch: Guidance - Orch ->> PE: Apply user guidance - end - end - end - - else Bug Verify-Only - Orch ->>+ PE: Write temp test in /tmp/qualify-‹name›/ - PE -->>- Orch: Temp test ready - - loop Until pass or stuck - Orch ->>+ K8s: Run temp test on cluster - K8s -->>- Orch: Test results - Orch ->>+ CV: Verify cluster state - CV ->>+ K8s: oc get … - K8s -->>- CV: Resource states - CV -->>- Orch: Verification report - alt FAIL & retries remain - Orch ->> PE: Fix temp test - else FAIL & stuck - Orch -->> User: 🛑 Stuck — need guidance - User -->> Orch: Guidance - end - end - Note over Orch: Skip Phase 3 → go to Phase 4 - end - - Note over User,K8s: Phase 3 — Code Review & PR (permanent tests only) - - alt Feature or Bug-Permanent-Test - par Quality Review - Orch ->>+ CR: code-reviewer-quality - CR -->>- Orch: Quality findings - and Guidelines Review - Orch ->>+ CR: code-reviewer-guidelines - CR -->>- Orch: Guidelines findings - and Security Review - Orch ->>+ CR: code-reviewer-security - CR -->>- Orch: Security findings - end - - loop Until no findings - alt Issues found - Orch ->> PE: Fix review issues - PE -->> Orch: Fixes applied - Orch ->> CR: Re-review - CR -->> Orch: Updated findings - end - end - - Orch ->> Orch: pre-commit run --all-files - loop Until pre-commit passes - alt Failures - Orch ->> Orch: Fix formatting/linting - end - end - - Orch ->>+ GE: Create PR: [qualify] ‹type›: ‹name› - GE -->>- Orch: PR URL - Orch -->> User: 🛑 PR ready for review - end - - Note over User,K8s: Phase 4 — Generate Proof - - Orch ->>+ PG: Assemble proof report - Note right of PG: Inputs:
• test-output.log
• cluster verification report
• OCP/MTV/CNV versions
• test-plan.md reference - PG ->> PG: Apply proof-template.md structure - PG ->> PG: Determine verdict:
QUALIFIED / NOT QUALIFIED /
BUG FIXED / BUG NOT FIXED - PG -->>- Orch: proof.md + actor User + participant Orch as Orchestrator + participant TP as test-planner + participant PE as python-expert + participant CV as cluster-verifier + + Note over User,CV: Phase 0 — Setup + User ->> Orch: /qualify --type --source + Orch ->> Orch: Validate cluster + collect versions + + Note over User,CV: Phase 1 — Test Plan + Orch ->> TP: Produce test plan + TP -->> Orch: test-plan.md + Orch -->> User: 🛑 Review plan + User -->> Orch: ✅ Approved + + Note over User,CV: Phase 2 — Write & Verify + Orch ->> PE: Write tests + PE -->> Orch: Tests ready + Orch ->> Orch: Run pytest on cluster + Orch ->> CV: Verify cluster state + CV -->> Orch: Verification ✅ + Note right of Orch: Retry up to 3× on failure +``` - Orch ->> Orch: Write .qualify/‹type›/‹name›/proof.md +### 8b. Review, PR & Proof (Phases 3–4) - Orch -->>- User: 🏁 Qualification Complete
Result + Artifacts + Versions +```mermaid +sequenceDiagram + actor User + participant Orch as Orchestrator + participant CR as code-reviewers + participant GE as github-expert + participant PG as proof-generator + + Note over User,PG: Phase 3 — Review & PR + Orch ->> CR: 3 parallel reviews + CR -->> Orch: Findings + Note right of Orch: Fix & re-review until clean + Orch ->> Orch: pre-commit + Orch ->> GE: Create PR + GE -->> Orch: PR URL + Orch -->> User: 🛑 PR ready + + Note over User,PG: Phase 4 — Proof + Orch ->> PG: Assemble proof + PG -->> Orch: proof.md + verdict + Orch -->> User: 🏁 Complete ``` --- @@ -457,7 +363,6 @@ sequenceDiagram | ⬜ Gray parallelogram | Output artifact (file) | | 🟩 Green rounded | Start / success outcome | | 🟥 Red rounded | Failure outcome | -| 🟤 Pink | Codebase reference | | 🔴 Coral | External agent (not in qualify/) | ## Key Takeaways From 50a78dfacdd5810baedde2d576d7fbfdc177c40c Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Mon, 11 May 2026 11:17:19 +0300 Subject: [PATCH 04/13] fix: address CodeRabbit review comments - Clarify failure semantics in cluster-verifier agent - Add redaction rules to proof-generator skill - Fix verify-only path in qualify prompt --- llm/qualify/agents/cluster-verifier.md | 9 ++++++--- llm/qualify/prompts/qualify.md | 10 +++++++++- llm/qualify/skills/proof-generator/SKILL.md | 3 +++ 3 files changed, 18 insertions(+), 4 deletions(-) diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md index c3432615..33a7f470 100644 --- a/llm/qualify/agents/cluster-verifier.md +++ b/llm/qualify/agents/cluster-verifier.md @@ -32,7 +32,7 @@ oc whoami oc cluster-info ``` -If either command fails, stop immediately and report the failure. Do NOT proceed with partial assumptions. +If either command fails, stop immediately and report the failure. Do NOT proceed with any verification checks. ### Version Collection @@ -143,11 +143,14 @@ oc get vmi -n -o jsonpath='{.status.conditions}' For every check, capture and record: 1. **The exact `oc` command run** — copy-paste reproducible. -2. **The full output** (or a relevant excerpt if output exceeds ~200 lines). +2. **The full output** (or a relevant excerpt if output exceeds ~200 lines), with sensitive values redacted. 3. **PASS/FAIL determination** with a one-line reason. 4. **Timestamp** — use `date -u +"%Y-%m-%dT%H:%M:%SZ"` before each check group. Do not summarize away raw evidence. Always preserve it for the report. +Before storing evidence, redact sensitive fields/tokens +(for example: `token`, `password`, `secret`, `clientSecret`, `Authorization`, +private keys, kubeconfig credentials). Keep resource names, states, and condition fields intact. ## Output Format @@ -249,7 +252,7 @@ If the agent cannot connect to the cluster or a verification check fails: - **Report exactly what failed** — include the command, exit code, and error output. - **Do NOT make assumptions** about cluster state. If `oc get vm` returns an error, do not guess whether the VM exists. - **Include error messages verbatim** — do not paraphrase or summarize errors. -- **Continue checking other items** — one failure does not stop the entire verification. Mark the failed check and proceed. +- **Continue checking other items** — applies only after connectivity is confirmed. One check failure does not stop the entire verification; mark failed checks and proceed. ```text | VM Exists | `vm/rhel-9` in `ns` | ❌ FAIL | `oc get vm rhel-9 -n ns` returned: error not found | diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index dda22285..504cde41 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -134,7 +134,15 @@ This phase is **fully autonomous** — no human intervention unless the AI gets ### For bugs-verify-only (no permanent test) 1. Write a **temporary test file** in `/tmp/qualify-/` (not in the repo) -2. Run it on the cluster (same as step 3 above) +2. Run it on the cluster using the temporary path, for example: + + ```bash + export KUBECONFIG= + uv run pytest /tmp/qualify-/.py -v \ + --tc-file=tests/tests_config/config.py --tc-format=python -p no:xdist \ + 2>&1 | tee .qualify///test-output.log + ``` + 3. Verify on cluster (same as step 4 above) 4. Skip Phase 3 (no PR needed), go directly to Phase 4 diff --git a/llm/qualify/skills/proof-generator/SKILL.md b/llm/qualify/skills/proof-generator/SKILL.md index e0d6d086..b75542af 100644 --- a/llm/qualify/skills/proof-generator/SKILL.md +++ b/llm/qualify/skills/proof-generator/SKILL.md @@ -128,6 +128,9 @@ Independent verification performed after test execution. ### Evidence Requirements - **Always** include raw evidence (YAML, logs) in collapsible `
` sections. +- **Always redact sensitive values** before writing proof artifacts + (tokens, passwords, secrets, auth headers, private keys, kubeconfig credentials). + Preserve only fields needed to validate behavior. - **Versions are MANDATORY.** If any version (OCP, MTV, or CNV) is missing, mark the report as `❌ NOT QUALIFIED` with reason: `"Missing version information"`. - The report must be **self-contained**. A reader must understand what was tested, what passed or failed, and what the cluster state looked like — all from the proof.md alone. From 7dd1bd28e505797475f7f688a152d0a353217239 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Mon, 11 May 2026 12:05:46 +0300 Subject: [PATCH 05/13] fix: add version collection failure handling per CodeRabbit review --- llm/qualify/prompts/qualify.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index 504cde41..ad0f3d75 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -52,6 +52,8 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt ``` + If any version cannot be retrieved, record it as `UNKNOWN` with the error message. The final proof report will reflect missing versions. + 4. **Create output directory**: - Feature: `.qualify/features//` - Bug: `.qualify/bugs//` From 0c71048e0ac356f666a05113755624b857b21c6f Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Mon, 11 May 2026 13:06:11 +0300 Subject: [PATCH 06/13] fix: address 10 CodeRabbit review comments Safety, redaction, retry bounds, and doc consistency. --- llm/qualify/README.md | 4 ++-- llm/qualify/agents/cluster-verifier.md | 4 ++-- llm/qualify/prompts/qualify.md | 20 +++++++++++++++----- llm/qualify/skills/proof-generator/SKILL.md | 2 ++ llm/qualify/templates/proof-template.md | 2 ++ llm/qualify/templates/test-plan-template.md | 1 + llm/qualify/workflow-diagrams.md | 6 +++--- 7 files changed, 27 insertions(+), 12 deletions(-) diff --git a/llm/qualify/README.md b/llm/qualify/README.md index 3beee0e3..576e6a0d 100644 --- a/llm/qualify/README.md +++ b/llm/qualify/README.md @@ -29,7 +29,7 @@ Full qualification workflow for MTV API tests: from feature design or bug report | ----------- | -------- | ---------------------------------------------------------------------- | | `--type` | Yes | `feature` or `bug` | | `--source` | Yes | URL to Jira ticket, GitHub issue, design doc, or local file path | -| `--cluster` | No | Path to kubeconfig. If omitted, uses current `oc` context | +| `--cluster` | Yes | Path to kubeconfig for qualification (explicit target required) | | `--name` | No | Short identifier (e.g., `warm-migration-rhv`). Auto-derived if omitted | ## Usage Examples @@ -152,7 +152,7 @@ Output (gitignored): │ ├── test-output.log │ └── proof.md └── bugs/ - └── / + └── / ├── test-plan.md ├── test-output.log └── proof.md diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md index 33a7f470..0f22fba5 100644 --- a/llm/qualify/agents/cluster-verifier.md +++ b/llm/qualify/agents/cluster-verifier.md @@ -43,10 +43,10 @@ Collect environment versions at the start of every verification run: oc get clusterversion version -o jsonpath='{.status.desired.version}' # MTV version (from CSV in openshift-mtv namespace) -oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv +oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv || true # CNV version (from CSV in openshift-cnv namespace) -oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt +oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt || true ``` If a version cannot be retrieved, record it as `UNKNOWN` with the error message. diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index ad0f3d75..0d0e85ed 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -24,9 +24,16 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on - `--cluster`: Path to kubeconfig file. If not provided, assume current context (`oc whoami` must work) - `--name`: Short identifier for this qualification (e.g., `warm-migration-rhv`, `JIRA-12345`). If not provided, derive from source. +- Normalize `name` to a safe slug before any use: + - allowed chars: `a-z`, `0-9`, `-` + - replace all other chars with `-` + - collapse repeated `-`, trim leading/trailing `-` + - max length 63 + - reject values containing `..`, `/`, `\` + If required arguments are missing, ask the user to provide them using the ask_user tool. -2. **Validate cluster connectivity**: +1. **Validate cluster connectivity**: ```bash # If --cluster provided: @@ -39,7 +46,7 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on If cluster is unreachable, STOP and ask the user to fix it. -3. **Collect environment versions** (save for proof.md): +2. **Collect environment versions** (save for proof.md): ```bash # OCP version @@ -54,11 +61,11 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on If any version cannot be retrieved, record it as `UNKNOWN` with the error message. The final proof report will reflect missing versions. -4. **Create output directory**: +3. **Create output directory**: - Feature: `.qualify/features//` - Bug: `.qualify/bugs//` -5. **For bugs only** — ask the user: +4. **For bugs only** — ask the user: > "Should this bug get a permanent test in the test suite? (Yes = full PR flow, No = verify-only with proof.md)" ## Phase 1: Test Plan @@ -131,7 +138,10 @@ This phase is **fully autonomous** — no human intervention unless the AI gets - Cluster verification failed (tests said pass but cluster state wrong) → investigate and fix - **AI stuck** → ask the user: "I'm stuck on: ``. How should I proceed?" -6. **Loop** steps 3-5 until tests pass with proof. +6. **Loop with bounded retries**: + - Track `attempt_count` for the current failing error signature. + - If the same failure persists for 3 attempts, STOP autonomous retries and ask the user for guidance. + - Resume only after user guidance; reset counter when error signature changes. ### For bugs-verify-only (no permanent test) diff --git a/llm/qualify/skills/proof-generator/SKILL.md b/llm/qualify/skills/proof-generator/SKILL.md index b75542af..5d8d81d1 100644 --- a/llm/qualify/skills/proof-generator/SKILL.md +++ b/llm/qualify/skills/proof-generator/SKILL.md @@ -113,6 +113,8 @@ Independent verification performed after test execution. ### Qualification Logic +Applies to `type=feature`. For `type=bug`, use **Bug Verification Logic** verdict labels. + 1. **NEVER** mark as `✅ QUALIFIED` if any test failed (exit code ≠ 0 or any individual test result is not PASSED). 2. **NEVER** mark as `✅ QUALIFIED` if cluster verification has any `❌ FAIL` check. 3. If both tests and cluster verification pass → `✅ QUALIFIED`. diff --git a/llm/qualify/templates/proof-template.md b/llm/qualify/templates/proof-template.md index 0187fe2b..6d6f59c7 100644 --- a/llm/qualify/templates/proof-template.md +++ b/llm/qualify/templates/proof-template.md @@ -44,6 +44,8 @@ Independent verification performed after test execution. ### Raw Evidence +> Redact sensitive values before pasting evidence (`token`, `password`, `secret`, `Authorization`, kubeconfig credentials, private keys, emails/IPs as required by policy). +
Resource details ```yaml diff --git a/llm/qualify/templates/test-plan-template.md b/llm/qualify/templates/test-plan-template.md index f46d6112..4703ceed 100644 --- a/llm/qualify/templates/test-plan-template.md +++ b/llm/qualify/templates/test-plan-template.md @@ -46,6 +46,7 @@ 3. Create Plan with `` 4. Execute migration 5. Verify migrated VMs +6. `` **Expected Outcomes**: diff --git a/llm/qualify/workflow-diagrams.md b/llm/qualify/workflow-diagrams.md index cf4cd419..4091489b 100644 --- a/llm/qualify/workflow-diagrams.md +++ b/llm/qualify/workflow-diagrams.md @@ -138,7 +138,7 @@ flowchart TD PASS1 -- "Yes ✅" --> DONE1 PASS1 -- "No ❌" --> RETRY1 RETRY1 -- "Yes" --> FIX1 --> RUN - RETRY1 -- "No" --> STUCK1 --> FIX1 + RETRY1 -- "No" --> STUCK1 --> ESCALATE1["⛔ Escalate / stop autonomous run"]:::human %% Verify-only path WTEMP["Write temp test\nin /tmp/"]:::agent @@ -155,7 +155,7 @@ flowchart TD PASS2 -- "Yes ✅" --> DONE2 PASS2 -- "No ❌" --> RETRY2 RETRY2 -- "Yes" --> FIX2 --> RTEMP - RETRY2 -- "No" --> STUCK2 --> FIX2 + RETRY2 -- "No" --> STUCK2 --> ESCALATE2["⛔ Escalate / stop autonomous run"]:::human ``` --- @@ -367,7 +367,7 @@ sequenceDiagram ## Key Takeaways -1. **Four distinct phases** with clear handoff boundaries. +1. **Five distinct phases** with clear handoff boundaries. 2. **Human stays in the loop** at test-plan review, bug-mode decision, stuck escalation, and PR review — everything else is autonomous. 3. **Dual verification** — pytest execution alone is never sufficient; the `cluster-verifier` agent independently confirms cluster state. 4. **Bug workflows fork early** (Phase 0) into permanent-test vs. verify-only, rejoining at proof generation (Phase 4). From 07d542d21fc3910149bc63671740aa5b2845f21a Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Mon, 11 May 2026 14:03:26 +0300 Subject: [PATCH 07/13] fix: address 6 CodeRabbit review comments - require --cluster, normalize placeholders, capture version errors --- llm/qualify/README.md | 2 +- llm/qualify/agents/cluster-verifier.md | 8 ++++++-- llm/qualify/prompts/qualify.md | 10 +++++----- llm/qualify/skills/proof-generator/SKILL.md | 2 +- 4 files changed, 13 insertions(+), 9 deletions(-) diff --git a/llm/qualify/README.md b/llm/qualify/README.md index 576e6a0d..25a78f57 100644 --- a/llm/qualify/README.md +++ b/llm/qualify/README.md @@ -43,7 +43,7 @@ Full qualification workflow for MTV API tests: from feature design or bug report ### Verify a Bug Fix ```bash -/qualify --type bug --source https://issues.redhat.com/browse/MTV-5678 --name MTV-5678 +/qualify --type bug --source https://issues.redhat.com/browse/MTV-5678 --cluster ~/kubeconfigs/test-cluster --name MTV-5678 ``` The AI will ask: "Should this bug get a permanent test in the test suite?" diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md index 0f22fba5..8069c283 100644 --- a/llm/qualify/agents/cluster-verifier.md +++ b/llm/qualify/agents/cluster-verifier.md @@ -43,10 +43,14 @@ Collect environment versions at the start of every verification run: oc get clusterversion version -o jsonpath='{.status.desired.version}' # MTV version (from CSV in openshift-mtv namespace) -oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv || true +MTV_VERSION_RAW="$(oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' 2>&1)" +MTV_VERSION="$(printf '%s\n' "$MTV_VERSION_RAW" | grep mtv || true)" +# if empty -> record: UNKNOWN: # CNV version (from CSV in openshift-cnv namespace) -oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt || true +CNV_VERSION_RAW="$(oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' 2>&1)" +CNV_VERSION="$(printf '%s\n' "$CNV_VERSION_RAW" | grep kubevirt || true)" +# if empty -> record: UNKNOWN: ``` If a version cannot be retrieved, record it as `UNKNOWN` with the error message. diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index 0d0e85ed..45e7c110 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -1,6 +1,6 @@ --- description: "Full qualification workflow: test plan → write tests → verify on cluster → PR with proof" -argument-hint: "<--type feature|bug> <--source URL|file> [--cluster kubeconfig-path] [--name identifier]" +argument-hint: "<--type feature|bug> <--source URL|file> <--cluster kubeconfig-path> [--name identifier]" --- # /qualify — Full Qualification Workflow @@ -21,7 +21,7 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on 1. **Parse arguments** from the raw text above: - `--type`: `feature` or `bug` (REQUIRED) - `--source`: URL (Jira, GitHub issue, design doc) or local file path (REQUIRED) - - `--cluster`: Path to kubeconfig file. If not provided, assume current context (`oc whoami` must work) + - `--cluster`: Path to kubeconfig file (REQUIRED). Do not use implicit current context. - `--name`: Short identifier for this qualification (e.g., `warm-migration-rhv`, `JIRA-12345`). If not provided, derive from source. - Normalize `name` to a safe slug before any use: @@ -53,17 +53,17 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on oc get clusterversion version -o jsonpath='{.status.desired.version}' # MTV version (from CSV) - oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv + oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv || true # CNV version (from CSV) - oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt + oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt || true ``` If any version cannot be retrieved, record it as `UNKNOWN` with the error message. The final proof report will reflect missing versions. 3. **Create output directory**: - Feature: `.qualify/features//` - - Bug: `.qualify/bugs//` + - Bug: `.qualify/bugs//` 4. **For bugs only** — ask the user: > "Should this bug get a permanent test in the test suite? (Yes = full PR flow, No = verify-only with proof.md)" diff --git a/llm/qualify/skills/proof-generator/SKILL.md b/llm/qualify/skills/proof-generator/SKILL.md index 5d8d81d1..be7b5bc2 100644 --- a/llm/qualify/skills/proof-generator/SKILL.md +++ b/llm/qualify/skills/proof-generator/SKILL.md @@ -164,6 +164,6 @@ When the verdict is NOT QUALIFIED or BUG NOT FIXED, add a `### Failure Details` Write the generated proof report to: - **Features**: `.qualify/features//proof.md` -- **Bugs**: `.qualify/bugs//proof.md` +- **Bugs**: `.qualify/bugs//proof.md` The directory must match the directory used by the test plan. If the test plan lives at `.qualify/features/cold-migration/test-plan.md`, then the proof goes to `.qualify/features/cold-migration/proof.md`. From b685f33052a10e60abf6b4eb52651b2c8d386b52 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Mon, 11 May 2026 15:10:14 +0300 Subject: [PATCH 08/13] fix: align version collection with error capture, add tool name mapping note --- llm/qualify/agents/cluster-verifier.md | 1 + llm/qualify/prompts/qualify.md | 8 ++++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md index 8069c283..9d0dcdd6 100644 --- a/llm/qualify/agents/cluster-verifier.md +++ b/llm/qualify/agents/cluster-verifier.md @@ -2,6 +2,7 @@ name: cluster-verifier description: Independently verifies OpenShift cluster state after test execution. Checks that resources exist, VMs are running, migrations completed, and collects evidence. tools: read, bash + --- # Cluster Verifier Agent diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index 45e7c110..38b6c679 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -53,10 +53,14 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on oc get clusterversion version -o jsonpath='{.status.desired.version}' # MTV version (from CSV) - oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep mtv || true + MTV_VERSION_RAW="$(oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' 2>&1)" + MTV_VERSION="$(printf '%s\n' "$MTV_VERSION_RAW" | grep mtv || true)" + # if empty -> record: UNKNOWN: # CNV version (from CSV) - oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' | grep kubevirt || true + CNV_VERSION_RAW="$(oc get csv -n openshift-cnv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' 2>&1)" + CNV_VERSION="$(printf '%s\n' "$CNV_VERSION_RAW" | grep kubevirt || true)" + # if empty -> record: UNKNOWN: ``` If any version cannot be retrieved, record it as `UNKNOWN` with the error message. The final proof report will reflect missing versions. From 9fa43d935fc2f1b4e1f036bc3a7e8f21c5e518ce Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Tue, 19 May 2026 10:45:10 +0300 Subject: [PATCH 09/13] Add bug ID extraction/validation in qualify workflow Add explicit step in Phase 0 to extract canonical bug identifiers from --source URLs (Jira/GitHub patterns), support --name override, and prompt user on extraction failure. --- llm/qualify/prompts/qualify.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index 38b6c679..f4f92a25 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -69,7 +69,14 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on - Feature: `.qualify/features//` - Bug: `.qualify/bugs//` -4. **For bugs only** — ask the user: +4. **For bugs only — extract bug ID**: + - Extract bug ID from `--source` URL (e.g., Jira ticket key from `https://issues.redhat.com/browse/MTV-1234`, GitHub issue number from `https://github.com/org/repo/issues/42`) + - If `--name` was provided, use it as the bug ID (user override) + - If extraction fails and `--name` was not provided, ask the user: "Could not extract bug ID from source. Please provide bug ID using --name (e.g., `MTV-1234`, `BZ-67890`, `42`)" + - Normalize the extracted/provided bug ID using the same slug rules from step 1 (lowercase, safe chars, max 63) + - Use the normalized bug ID as `` for directory creation in step 3 + +5. **For bugs only** — ask the user: > "Should this bug get a permanent test in the test suite? (Yes = full PR flow, No = verify-only with proof.md)" ## Phase 1: Test Plan From d1b1a1819299d22c4cf6eb305f6d604256aeb4d6 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Tue, 19 May 2026 11:39:17 +0300 Subject: [PATCH 10/13] Improve qualify.md version collection and path consistency - Add UNKNOWN+diagnostics pattern for OCP version retrieval, matching the existing MTV/CNV error-capture approach - Introduce artifact_key concept (name for features, id for bugs) and unify all downstream path references to use it consistently --- llm/qualify/prompts/qualify.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index f4f92a25..ebe475ce 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -50,7 +50,9 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on ```bash # OCP version - oc get clusterversion version -o jsonpath='{.status.desired.version}' + OCP_VERSION_RAW="$(oc get clusterversion version -o jsonpath='{.status.desired.version}' 2>&1)" + OCP_VERSION="$(printf '%s\n' "$OCP_VERSION_RAW")" + # if empty or command error -> record: UNKNOWN: # MTV version (from CSV) MTV_VERSION_RAW="$(oc get csv -n openshift-mtv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.version}{"\n"}{end}' 2>&1)" @@ -65,9 +67,9 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on If any version cannot be retrieved, record it as `UNKNOWN` with the error message. The final proof report will reflect missing versions. -3. **Create output directory**: - - Feature: `.qualify/features//` - - Bug: `.qualify/bugs//` +3. **Create output directory** and define `artifact_key` for all later paths: + - Feature: `.qualify/features//` — `artifact_key` = `` + - Bug: `.qualify/bugs//` — `artifact_key` = `` 4. **For bugs only — extract bug ID**: - Extract bug ID from `--source` URL (e.g., Jira ticket key from `https://issues.redhat.com/browse/MTV-1234`, GitHub issue number from `https://github.com/org/repo/issues/42`) @@ -93,10 +95,10 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on - To read `llm/qualify/templates/test-plan-template.md` for the output template - To produce `test-plan.md` -3. **Save** the test plan to `.qualify///test-plan.md` +3. **Save** the test plan to `.qualify///test-plan.md` 4. **🛑 HUMAN CHECKPOINT**: Ask the user: - > "Test plan ready for review. Please review `.qualify///test-plan.md`. + > "Test plan ready for review. Please review `.qualify///test-plan.md`. > Approve or provide feedback?" Options: ["Approved — proceed to implementation", "I have feedback"] @@ -133,7 +135,7 @@ This phase is **fully autonomous** — no human intervention unless the AI gets export KUBECONFIG= # Run the specific test - uv run pytest tests/:: -v --tc-file=tests/tests_config/config.py --tc-format=python -p no:xdist 2>&1 | tee .qualify///test-output.log + uv run pytest tests/:: -v --tc-file=tests/tests_config/config.py --tc-format=python -p no:xdist 2>&1 | tee .qualify///test-output.log ``` Capture the full output. @@ -163,7 +165,7 @@ This phase is **fully autonomous** — no human intervention unless the AI gets export KUBECONFIG= uv run pytest /tmp/qualify-/.py -v \ --tc-file=tests/tests_config/config.py --tc-format=python -p no:xdist \ - 2>&1 | tee .qualify///test-output.log + 2>&1 | tee .qualify///test-output.log ``` 3. Verify on cluster (same as step 4 above) @@ -205,7 +207,7 @@ Only for features and bugs-with-permanent-tests. - Test plan reference - The template from `llm/qualify/templates/proof-template.md` -2. **Write proof.md** to `.qualify///proof.md` +2. **Write proof.md** to `.qualify///proof.md` 3. **Final summary** to the user: @@ -217,8 +219,8 @@ Only for features and bugs-with-permanent-tests. Result: QUALIFIED / NOT QUALIFIED / BUG FIXED / BUG NOT FIXED Artifacts: - - Test Plan: .qualify///test-plan.md - - Proof: .qualify///proof.md + - Test Plan: .qualify///test-plan.md + - Proof: .qualify///proof.md - PR: (if applicable) Environment: From d55f7942993437f1ed095d8f6758a94f162e0b0d Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Tue, 2 Jun 2026 10:27:31 +0300 Subject: [PATCH 11/13] fix: address review findings for qualify workflow - Add set -o pipefail and PIPESTATUS to preserve pytest exit codes through tee - Fix ask_user tool reference for cross-CLI compatibility - Clarify UNKNOWN version handling links to NOT QUALIFIED verdict - Clarify AGENTS.md scope in workflow references - Move HTML comment outside YAML frontmatter in cluster-verifier - Make step 6 explicitly conditional in test-plan template - Add preserve_static_ips to tests_params example - Add --cluster arg to sequence diagram - Add Failure Details section to proof template --- llm/qualify/agents/cluster-verifier.md | 3 ++- llm/qualify/prompts/qualify.md | 16 +++++++++++----- llm/qualify/templates/proof-template.md | 6 ++++++ llm/qualify/templates/test-plan-template.md | 3 ++- llm/qualify/workflow-diagrams.md | 2 +- 5 files changed, 22 insertions(+), 8 deletions(-) diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md index 9d0dcdd6..459c9132 100644 --- a/llm/qualify/agents/cluster-verifier.md +++ b/llm/qualify/agents/cluster-verifier.md @@ -2,9 +2,10 @@ name: cluster-verifier description: Independently verifies OpenShift cluster state after test execution. Checks that resources exist, VMs are running, migrations completed, and collects evidence. tools: read, bash - --- + + # Cluster Verifier Agent ## Base Rules diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index ebe475ce..4e7ace2e 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -31,7 +31,7 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on - max length 63 - reject values containing `..`, `/`, `\` - If required arguments are missing, ask the user to provide them using the ask_user tool. + If required arguments are missing, ask the user to provide them (use `ask_user` if available, otherwise ask in chat). 1. **Validate cluster connectivity**: @@ -65,7 +65,9 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on # if empty -> record: UNKNOWN: ``` - If any version cannot be retrieved, record it as `UNKNOWN` with the error message. The final proof report will reflect missing versions. + If any version cannot be retrieved, record it as `UNKNOWN` with the error message and continue the workflow. + **Note:** `UNKNOWN` versions will force the final verdict to `❌ NOT QUALIFIED` or `🐛 BUG NOT FIXED` + per proof-generator rules (versions are mandatory). The workflow still proceeds to collect all other evidence. 3. **Create output directory** and define `artifact_key` for all later paths: - Feature: `.qualify/features//` — `artifact_key` = `` @@ -90,7 +92,7 @@ It overrides the normal "AI must NEVER run tests" rule — tests ARE executed on Tell the agent: - The source material content - The type (feature or bug) - - To read `AGENTS.md` for project patterns + - To read `AGENTS.md` for project standards (includes coding patterns, test structure, and the `/qualify` exception to the test execution prohibition) - To read existing tests in `tests/` for examples - To read `llm/qualify/templates/test-plan-template.md` for the output template - To produce `test-plan.md` @@ -122,7 +124,7 @@ This phase is **fully autonomous** — no human intervention unless the AI gets 2. **Write tests**: Delegate to python-expert: - Provide the approved test plan - - Provide AGENTS.md for coding standards + - Provide `AGENTS.md` for coding standards (all project constraints are in this file) - Tell it to follow the 5/6-step test pattern - Tell it to create the test config in `tests/tests_config/config.py` - Tell it to create the test file in the appropriate `tests//` directory @@ -134,8 +136,10 @@ This phase is **fully autonomous** — no human intervention unless the AI gets # Set KUBECONFIG if provided export KUBECONFIG= - # Run the specific test + # Run the specific test (pipefail preserves pytest exit code through tee) + set -o pipefail uv run pytest tests/:: -v --tc-file=tests/tests_config/config.py --tc-format=python -p no:xdist 2>&1 | tee .qualify///test-output.log + PYTEST_EXIT=${PIPESTATUS[0]} ``` Capture the full output. @@ -163,9 +167,11 @@ This phase is **fully autonomous** — no human intervention unless the AI gets ```bash export KUBECONFIG= + set -o pipefail uv run pytest /tmp/qualify-/.py -v \ --tc-file=tests/tests_config/config.py --tc-format=python -p no:xdist \ 2>&1 | tee .qualify///test-output.log + PYTEST_EXIT=${PIPESTATUS[0]} ``` 3. Verify on cluster (same as step 4 above) diff --git a/llm/qualify/templates/proof-template.md b/llm/qualify/templates/proof-template.md index 6d6f59c7..86aed080 100644 --- a/llm/qualify/templates/proof-template.md +++ b/llm/qualify/templates/proof-template.md @@ -64,6 +64,12 @@ Independent verification performed after test execution. - [ ] Evidence collected for all verification points - [ ] Versions recorded +### Failure Details (required when result is ❌ NOT QUALIFIED or 🐛 BUG NOT FIXED) + +| Failed Item | Type | Details | +|--------------------------|----------------------|--------------------------------| +| `` | Test / Cluster Check | `` | + ### Verdict diff --git a/llm/qualify/templates/test-plan-template.md b/llm/qualify/templates/test-plan-template.md index 4703ceed..22405973 100644 --- a/llm/qualify/templates/test-plan-template.md +++ b/llm/qualify/templates/test-plan-template.md @@ -46,7 +46,7 @@ 3. Create Plan with `` 4. Execute migration 5. Verify migrated VMs -6. `` +6. `` **Expected Outcomes**: @@ -80,6 +80,7 @@ }, ], "warm_migration": False, + "preserve_static_ips": False, }, ``` diff --git a/llm/qualify/workflow-diagrams.md b/llm/qualify/workflow-diagrams.md index 4091489b..a56e0ca6 100644 --- a/llm/qualify/workflow-diagrams.md +++ b/llm/qualify/workflow-diagrams.md @@ -307,7 +307,7 @@ sequenceDiagram participant CV as cluster-verifier Note over User,CV: Phase 0 — Setup - User ->> Orch: /qualify --type --source + User ->> Orch: /qualify --type --source --cluster ~/kubeconfig Orch ->> Orch: Validate cluster + collect versions Note over User,CV: Phase 1 — Test Plan From 451f5d74f5e5403a49ebc4d86680eff3b67e0b20 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Tue, 2 Jun 2026 10:40:04 +0300 Subject: [PATCH 12/13] fix: address Qodo cycle 2 review findings - Add explicit PYTEST_EXIT gating logic to enforce pass/fail branching - Add concrete argument values to sequence diagram example - Improve tool name mapping comment with actionable instructions --- llm/qualify/agents/cluster-verifier.md | 3 ++- llm/qualify/prompts/qualify.md | 9 +++++---- llm/qualify/workflow-diagrams.md | 2 +- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md index 459c9132..b35ffb48 100644 --- a/llm/qualify/agents/cluster-verifier.md +++ b/llm/qualify/agents/cluster-verifier.md @@ -4,7 +4,8 @@ description: Independently verifies OpenShift cluster state after test execution tools: read, bash --- - + + # Cluster Verifier Agent diff --git a/llm/qualify/prompts/qualify.md b/llm/qualify/prompts/qualify.md index 4e7ace2e..9ffe0e73 100644 --- a/llm/qualify/prompts/qualify.md +++ b/llm/qualify/prompts/qualify.md @@ -142,16 +142,17 @@ This phase is **fully autonomous** — no human intervention unless the AI gets PYTEST_EXIT=${PIPESTATUS[0]} ``` - Capture the full output. + **Tests passed** is defined as `PYTEST_EXIT == 0`. If `PYTEST_EXIT != 0`, tests failed — do NOT + proceed to cluster verification. Instead, go to step 5 (evaluate results) with a failed status. 4. **Verify on cluster**: Delegate to cluster-verifier agent (from `llm/qualify/agents/cluster-verifier.md`): - Provide the test plan (what to verify) - Provide the namespace used by the test - The agent checks cluster state independently -5. **Evaluate results**: - - Tests passed AND cluster verification passed → proceed to Phase 3 - - Tests failed → delegate to python-expert to fix, then re-run (go to step 3) +5. **Evaluate results** (based on `PYTEST_EXIT` from step 3, NOT log parsing): + - `PYTEST_EXIT == 0` AND cluster verification passed → proceed to Phase 3 + - `PYTEST_EXIT != 0` → tests failed, delegate to python-expert to fix, then re-run (go to step 3) - Cluster verification failed (tests said pass but cluster state wrong) → investigate and fix - **AI stuck** → ask the user: "I'm stuck on: ``. How should I proceed?" diff --git a/llm/qualify/workflow-diagrams.md b/llm/qualify/workflow-diagrams.md index a56e0ca6..aad9beb2 100644 --- a/llm/qualify/workflow-diagrams.md +++ b/llm/qualify/workflow-diagrams.md @@ -307,7 +307,7 @@ sequenceDiagram participant CV as cluster-verifier Note over User,CV: Phase 0 — Setup - User ->> Orch: /qualify --type --source --cluster ~/kubeconfig + User ->> Orch: /qualify --type feature --source --cluster ~/kubeconfig Orch ->> Orch: Validate cluster + collect versions Note over User,CV: Phase 1 — Test Plan From cc7886e8691b8b808999182e7a3902ecffeaf379 Mon Sep 17 00:00:00 2001 From: Meni Yakove Date: Tue, 2 Jun 2026 11:34:20 +0300 Subject: [PATCH 13/13] fix: clarify error message redaction in cluster-verifier Add parenthetical noting that sensitive value redaction still applies when including error messages verbatim. --- llm/qualify/agents/cluster-verifier.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/llm/qualify/agents/cluster-verifier.md b/llm/qualify/agents/cluster-verifier.md index b35ffb48..65be7f2e 100644 --- a/llm/qualify/agents/cluster-verifier.md +++ b/llm/qualify/agents/cluster-verifier.md @@ -258,7 +258,7 @@ If the agent cannot connect to the cluster or a verification check fails: - **Report exactly what failed** — include the command, exit code, and error output. - **Do NOT make assumptions** about cluster state. If `oc get vm` returns an error, do not guess whether the VM exists. -- **Include error messages verbatim** — do not paraphrase or summarize errors. +- **Include error messages verbatim (with sensitive values redacted)** — do not paraphrase or summarize errors, but apply the same redaction rules as for evidence output. - **Continue checking other items** — applies only after connectivity is confirmed. One check failure does not stop the entire verification; mark failed checks and proceed. ```text