feat: add Tier 3 testing classification for extended validation (#123)

* feat: add Tier 3 testing classification

Add Tier 3 (Extended Validation) as a new testing tier for tests
with higher execution cost — longer runtime, heavier resource
usage, or complex setup — that typically run in dedicated test
cycles rather than standard CI lanes.

Updated files: AGENTS.md, docs/testing-tiers.md, docs/stp-guide.md,
README.md, stps/stp-template/stp.md, stps/stp-template/child-stp.md

Signed-off-by: rnetser <rnetser@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>

* fix: make Tier 3 coverage conditional in checklists

Tier 3 is optional extended validation, not mandatory for every STP.
Updated sign-off checklist and process milestone to say 'where applicable'.

Signed-off-by: rnetser <rnetser@redhat.com>

Co-authored-by: PI (claude-opus-4-6) <noreply@pi.dev>
Signed-off-by: rnetser <rnetser@redhat.com>

---------

Signed-off-by: rnetser <rnetser@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: PI (claude-opus-4-6) <noreply@pi.dev>

Author: 3 people

AGENTS.md

@@ -228,7 +228,7 @@ Assisted-by: Claude <noreply@anthropic.com>
   In the scenarios table, multiple test scenarios mapped to the same requirement
   may leave the Requirement ID cell blank in continuation rows.
 - [ ] Requirement summaries use user story format ("As a [role], I want...")
-- [ ] Each scenario has Tier (1 or 2) and Priority (P0/P1/P2)
+- [ ] Each scenario has Tier (1, 2, or 3) and Priority (P0/P1/P2)
 - [ ] Every Testing Goal from Section II.1 has a matching scenario here
 - [ ] Every Acceptance Criterion from Section I.1 is traceable to a scenario
 - [ ] Granularity: if one scenario can fail while another passes, they are separate items
@@ -237,6 +237,7 @@ Assisted-by: Claude <noreply@anthropic.com>
 **Tier classification:**
 - **Tier 1**: Single feature, isolated — API validation, basic CRUD, single operation
 - **Tier 2**: End-to-end user workflows, multi-feature integration, upgrade paths
+- **Tier 3**: Extended validation — tests with higher execution cost (longer runtime, heavier resource usage, or complex setup) that typically run in dedicated test cycles rather than standard CI lanes
 
 **Common rejection reasons:**
 - All requirement IDs are "[TBD]" — blocks approval

README.md

@@ -32,7 +32,7 @@ Centralize QE documentation and test planning to ensure:
 | Guide                                          | Description                                                                        |
 |:-----------------------------------------------|:-----------------------------------------------------------------------------------|
 | [STP Guide](./docs/stp-guide.md)               | Complete guide to Software Test Plans: structure, lifecycle, and best practices    |
-| [Testing Tiers Guide](./docs/testing-tiers.md) | Differences between Unit Tests, Tier 1 (Functional), and Tier 2 (End-to-End) tests |
+| [Testing Tiers Guide](./docs/testing-tiers.md) | Differences between Unit Tests, Tier 1 (Functional), Tier 2 (End-to-End), and Tier 3 (Extended) tests |
 
 ### Templates
 
@@ -50,13 +50,14 @@ Centralize QE documentation and test planning to ensure:
 | **NFR**    | Non-Functional Requirements (performance, security, monitoring, etc.) |
 | **Tier 1** | Functional tests covering individual features                         |
 | **Tier 2** | End-to-end tests covering complete workflows                          |
+| **Tier 3** | Extended validation tests with higher execution cost                   |
 
 ### Testing Tiers
 
 ```text
-Unit Tests → Tier 1 (Functional) → Tier 2 (End-to-End)
-  Many          Moderate              Few
-Isolated       Feature-level      Full Integration
+Unit Tests → Tier 1 (Functional) → Tier 2 (End-to-End) → Tier 3 (Extended)
+  Many          Moderate              Few                    Fewest
+Isolated       Feature-level      Full Integration      Dedicated Cycles
 ```
 
 See [Testing Tiers Guide](./docs/testing-tiers.md) for details.
@@ -82,7 +83,7 @@ graph TD
 
 2. **STP Creation**
    - Document scope, strategy, environment, risks
-   - Define Tier 1 and Tier 2 test coverage
+   - Define Tier 1 and Tier 2 test coverage (Tier 3 where applicable)
    - List untestable aspects with stakeholder agreement
 
 3. **Enhancement Freeze (EF)**

docs/stp-guide.md

@@ -163,7 +163,7 @@ The STP template uses four distinct categories for constraints. Refer to the tem
 ### IV. Sign-off and Approval
 
 **Final Sign-off Checklist:**
-- Tier 1 / Tier 2 tests defined
+- Tier 1 / Tier 2 tests defined (Tier 3 where applicable)
 - **Automation merged** (mandatory for GA)
 - Tests running in release checklist jobs
 - Documentation reviewed

docs/testing-tiers.md

@@ -2,20 +2,23 @@
 
 ## Overview
 
-OpenShift Virtualization uses a tiered testing strategy to ensure comprehensive coverage while optimizing test execution time and resource usage. Understanding the differences between **Unit Tests**, **Tier 1 (Functional Tests)**, and **Tier 2 (End-to-End Tests)** is critical for effective test planning and implementation.
+OpenShift Virtualization uses a tiered testing strategy to ensure comprehensive coverage while optimizing test execution time and resource usage. Understanding the differences between **Unit Tests**, **Tier 1 (Functional Tests)**, **Tier 2 (End-to-End Tests)**, and **Tier 3 (Extended Validation Tests)** is critical for effective test planning and implementation.
 
 ## Testing Pyramid
 
 ```text
-        /\
-       /  \      Tier 2 (E2E)
-      /____\     - Fewest tests, full system
-     /      \
-    / Tier 1 \   Tier 1 (Functional)
-   /__________\  - More tests, feature integration
-  /            \
- /  Unit Tests  \ Unit Tests
-/________________\ - Most tests, isolated components
+          /\
+         /  \      Tier 3 (Extended)
+        /____\     - Fewest tests, dedicated cycles
+       /      \
+      / Tier 2 \   Tier 2 (E2E)
+     /__________\  - Full system workflows
+    /            \
+   /    Tier 1    \   Tier 1 (Functional)
+  /________________\  - More tests, feature integration
+ /                  \
+/     Unit Tests     \  Unit Tests
+/____________________\  - Most tests, isolated components
 ```
 
 ## Unit Tests
@@ -155,13 +158,49 @@ OpenShift Virtualization uses a tiered testing strategy to ensure comprehensive
 - System metrics that users don't directly interact with
 - Internal error messages or stack traces are not shown to users
 
+## Tier 3 - Extended Validation Tests
+
+### What Are Tier 3 Tests?
+
+**Tier 3 tests** validate feature functionality with higher execution cost — longer runtime, heavier resource usage, or complex setup requirements. These tests typically run in dedicated test cycles rather than standard CI lanes.
+
+**Important:** The distinguishing factor for Tier 3 is **execution cost**, not test complexity or scope. A Tier 3 test may verify a single feature (like Tier 1) or a complete workflow (like Tier 2) — what makes it Tier 3 is that it takes longer, needs more resources, or requires complex setup.
+
+### Characteristics
+
+- **Scope:** Any feature scenario with higher execution cost
+- **Dependencies:** May require specific guest OS images, non-default storage backends, large clusters, or extended runtime
+- **Environment:** Dedicated test cycles, not standard CI gating
+
+### Purpose
+
+- Validate feature behavior in configurations that are too costly for standard CI
+- Test scenarios requiring extended runtime or heavy resource usage
+- Verify behavior with non-default guest operating systems or storage backends
+- Validate resource-intensive concurrent operations
+
+### Test Scope for Tier 3
+
+**Extended Validation Scenarios — Examples:**
+
+- Windows guest VM operations (longer setup and execution time)
+- Large-scale concurrent operations (multiple VMs restoring/migrating in parallel)
+- Tests requiring non-default storage backends (LVM, specific CSI drivers)
+- Long-running stability or soak tests
+- Scenarios requiring large cluster configurations
+
+**What's NOT in Tier 3:**
+
+- Tests that run within standard CI time and resource budgets (use Tier 1 or Tier 2)
+- Performance benchmarking (covered under Performance Testing in Test Strategy)
+
 ## Comparison Matrix
 
-| Aspect                  | Unit Tests             | Tier 1 (Functional)                  | Tier 2 (E2E)                  |
-|:------------------------|:-----------------------|:-------------------------------------|:------------------------------|
-| **Scope**               | Single function/method | Single feature                       | Complete workflows            |
-| **Dependencies**        | None (mocked)          | OpenShift + OpenShift Virtualization | Full production-like setup    |
-| **Isolation**           | Complete               | Feature-level                        | System-level integration      |
+| Aspect                  | Unit Tests             | Tier 1 (Functional)                  | Tier 2 (E2E)                  | Tier 3 (Extended)                     |
+|:------------------------|:-----------------------|:-------------------------------------|:------------------------------|:--------------------------------------|
+| **Scope**               | Single function/method | Single feature                       | Complete workflows            | Any scope, higher execution cost      |
+| **Dependencies**        | None (mocked)          | OpenShift + OpenShift Virtualization | Full production-like setup    | Extended runtime / resources / setup  |
+| **Isolation**           | Complete               | Feature-level                        | System-level integration      | Dedicated test cycles                 |
 
 ## Resources
 

stps/stp-template/child-stp.md

@@ -190,7 +190,7 @@ Regression tests are documented in Test Strategy (II.2), not in this table. -->
 
 - **[Jira-ID]** — As a [role], I want [action] so that [benefit]
   - *Test Scenario:* [Description]
-  - *Tier:* [1 or 2]
+  - *Tier:* [1, 2, or 3]
   - *Priority:* [P0/P1/P2]
 
 ---

stps/stp-template/stp.md

@@ -452,6 +452,10 @@ justification in mitigation strategy. -->
   - *Test Scenario:* [Tier 2] Verify upgrade from version X to Y preserves feature state
   - *Priority:* P2
 
+- **[Jira-127]** — As a Windows VM user...
+  - *Test Scenario:* [Tier 3] Verify feature X works on Windows guest with NTFS filesystem
+  - *Priority:* P1
+
 ---
 
 ### **IV. Sign-off and Approval**