CI: only run playbook tests whose behavior actually changed

[lucbruni-amd]

Problem

A PR that only edits prose in a playbook README (a typo, a reworded sentence) currently re-runs that playbook's entire GPU test matrix, because detection is path-based: any file under playbooks/<cat>/<id>/ changing triggers all of that playbook's entries. These runs are slow and sometimes flaky.

What this does

Replaces path-based detection with content-based selection at per-entry (playbook, platform, device) granularity. On a PR, an entry runs only if what it would actually execute changed.

• Prose/typo edit -> 0 entries.
• Edit a test scoped @os:linux + @device:stx -> only the (linux, stx) entry, not Windows or other devices.
• Edit a shared @required dependency -> exactly its consumer playbooks.

The nightly cron still runs the full 194-entry matrix, so anything wrongly skipped on a PR surfaces within 24h.

How

run_playbook_tests.py gains a per-entry signature helper that reuses the real test extractor (so @require/@setup/@var/device filtering are all reflected). build_test_matrix.py materialises the base revision with git archive and diffs signatures computed by the head extractor against both trees. A byte-wise harness path check (deny-by-default over .github/scripts/ and test-playbooks.yml) forces the full matrix on any change to the selector/runner/workflow, which is what makes applying one extractor to both trees safe. Every failure to establish a trustworthy base falls back to the full matrix.

Safety

The dangerous direction is skipping an entry that changed. Guards: null signature -> run; new/vanished entry -> run; harness changed -> full matrix; unresolvable base -> full matrix; empty matrix or >256 jobs -> hard fail; nightly backstop.

Tests

test_build_test_matrix.py (22 tests, run as a required selector-tests job): synthetic mutations for each selection rule, plus replay of 9 real historical commits judged by an independent diff-derived classifier (e.g. #571 "add instructions to clarify usage" -> 0 selected; a real comfyui test edit -> 8/194). Verified green under GITHUB_ACTIONS=true.

Known limitation

A composite action under .github/actions/ referenced via uses: ./ would be outside the harness check. None exist today; closable by widening the harness prefixes if one is ever added.

[Copilot]

Pull request overview

This PR updates the playbook CI workflow to avoid re-running the full GPU test matrix on PRs when only non-executing content changes, by switching from path-based change detection to per-entry, content-based signatures derived from the real test extractor.

Changes:

• Replace path-based “changed playbook” detection with a signature-based selector that chooses entries at (playbook, platform, device) granularity.
• Add a dedicated selector-tests job that regression-tests the selection logic, and gate merges on selector correctness.
• Introduce new selector/build scripts to materialize the base revision and diff per-entry execution signatures, with safety fallbacks to “run full matrix”.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
.github/workflows/test-playbooks.yml	Switches to signature-based matrix building; adds selector regression test job and tighter gating.
.github/scripts/build_test_matrix.py	New matrix builder implementing harness-change detection and per-entry signature diffing.
.github/scripts/run_playbook_tests.py	Extends extractor APIs and adds signature helpers used by the matrix builder.
.github/scripts/test_build_test_matrix.py	New regression tests for selection behavior, including synthetic mutations and commit replay.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.