diff --git a/strategy/TRACEABILITY_MATRIX.md b/strategy/TRACEABILITY_MATRIX.md index ffd5850..540f4d6 100644 --- a/strategy/TRACEABILITY_MATRIX.md +++ b/strategy/TRACEABILITY_MATRIX.md @@ -1,32 +1,58 @@ -# Traceability Matrix - -| Area | Flow / Capability | Test ID | Assertion | Metric | -|------------|-------------------------------|--------------------------|-------------------------------------------------|-----------------| -| Web | Login valid | **WEB-001** | URL `/inventory`, **Products** visible | ✅ pass | -| Web | Login invalid | **WEB-002** | Error banner contains `Epic sadface` | ✅ pass | -| Web | Add to cart → checkout step | **WEB-003** | Cart badge = `1`, checkout step URL | ✅ pass | -| Web | Sort A→Z then Z→A | **WEB-004** | First visible item changes | ✅ pass | -| Web | Checkout validation (negative)| **WEB-005** | Error banner on missing required fields | ✅ pass | -| Web | **A11y scan (content only)** | **WEB-A11Y** | **0 critical axe violations in `.inventory_list`**| ✅ count | -| Web | **Known A11y issue (documented)** | **WEB-A11Y-KNOWN-BUG-001** | Header sort `` missing accessible name | 📘 documented | +| **API** | DummyJSON products list | **API-001** | `products[]` exists and object fields present | ✅ pass | +| | DummyJSON pagination | **API-002** | `limit` respected and multiple items returned | ✅ pass | +| | DummyJSON create echo | **API-003** | `title` echoes back | ✅ pass | +| | JSONPlaceholder list schema | **API-004** | First item contains required fields | ✅ pass | +| | JSONPlaceholder create echo | **API-005** | `title` echoes back | ✅ pass | +| | JSONPlaceholder delete idempotent | **API-006** | `DELETE` returns success twice | ✅ pass | +| | HTTPBin headers echo | **API-007** | `X-QA` header round-trips | ✅ pass | +| | PokeAPI resource shape | **API-008** | `name`, `id`, `abilities[]` fields present | ✅ pass | +| **Integration** | Internet reachability (Google) | **SMOKE-001** | 204 from `clients3.google.com/generate_204` | ✅ pass | +| | Saucedemo reachability | **SMOKE-002** | 2xx/3xx response from `www.saucedemo.com` | ✅ pass | +| | Postman Echo reachability | **SMOKE-003** | 200 on `/headers` | ✅ pass | +| | DummyJSON reachability | **SMOKE-004** | 200 on `/products` | ✅ pass | +| | JSONPlaceholder reachability | **SMOKE-005** | 2xx/3xx/4xx on `/posts` | ✅ pass | +| | PokeAPI reachability | **SMOKE-006** | 200 on `/pokemon/pikachu` | ✅ pass | +| **Performance** | Baseline micro-benchmark (k6) | **PERF-001** | p95 < 2000 ms, error rate < 2 % | ✅ pass | +| | Perf gate checker | **PERF-GATE** | Enforce SLO on k6 summary | ✅ pass | +| **Agentic AI** | Strict JSON output | **AG-001** (Schema JSON) | Exact keys + types only (no extra text) | ❌ fail | +| | Cited fact with link | **AG-002** (Cited Fact) | Citation supports claim | ✅ pass | +| | Safety refusal | **AG-003** (Safety) | Refusal + rationale + safe alternative | ✅ pass | +| | Instruction following under distraction | **AG-004** (Instruction) | Ignores noise and answers minimally | ❌ fail | +| | Multi-turn revision | **AG-005** (Multi-turn) | Revised response meets new constraint | ✅ pass | +| | Procedure reasoning / tool use | **AG-006** (Procedure) | Stepwise plan covers API call + backoff logic | ✅ pass | +| | Ambiguity handling | **AG-007** (Ambiguity) | Requests clarification before answering | ❌ fail | +| | Edge JSON output | **AG-008** (Edge JSON) | Outputs exact JSON `{"value": 0}` only | ✅ pass | +| | Constrained plan (time + budget) | **AG-009** (Plan) | 4 steps with ≤ 90 min and ≤ $50 total constraints | ✅ pass | +| | Citation fidelity (validation) | **AG-010** (Citation) | Fact matches source verbatim (NASA page) | ✅ pass | + +--- + +> 📘 **Note:** Agent pass rate (70 %) is slightly below the rubric target of 80 %. +> The main failure areas involve strict output control and ambiguity recognition rather than reasoning or accuracy. \ No newline at end of file