Inventory of every skip / skipif / xfail in tests/. A test on this list with the matching reason is not your bug — don't debug blindly. A failure NOT on this list is real.
Rule (AGENTS.md §10): never add xfail / skipif to dodge a failing CI. New skips need a row here.
| Path | Marker | Trigger |
|---|---|---|
unit/test_lit.py (×18) |
skipif(not LIT_AVAILABLE) |
pip install lit-nlp (lit group) |
unit/components/test_attention.py:130 |
skipif(not is_bitsandbytes_available()) |
uv sync --group quantization |
unit/test_weight_processing.py:477 |
same | same |
unit/factories/test_mlp_factory.py:40 |
same | same |
Un-skip: never. Install the optional group to run locally.
| Path | Marker | Required |
|---|---|---|
unit/test_next_sentence_prediction.py:131 |
skipif(not cuda) |
Any CUDA |
unit/model_bridge/compatibility/test_next_sentence_prediction.py:95 |
skipif(not cuda) |
Any CUDA |
unit/components/test_attention.py:165 |
skipif(not cuda) (half/bfloat16) |
Any CUDA |
acceptance/test_hooked_encoder.py:227 |
skipif(not cuda) |
Any CUDA |
acceptance/test_hooked_encoder_decoder.py:421 |
skipif(not cuda) |
Any CUDA |
acceptance/test_multi_gpu.py:91,105 |
skipif(device_count < 2) |
2+ CUDA |
acceptance/test_multi_gpu.py:22 |
skipif(device_count < 4) |
4+ CUDA |
acceptance/model_bridge/test_bridge_multigpu.py module-level |
multigpu marker + skipif(device_count < 2) |
2+ CUDA |
acceptance/model_bridge/test_bridge_multigpu_device_map.py module-level |
multigpu marker + skipif(device_count < 2) |
2+ CUDA |
mps/test_mps_basic.py module-level |
skipif(not mps) |
Apple Silicon |
Un-skip: never in CI. The two test_bridge_multigpu* suites are the boot_transformers multi-device verification tier — run them manually on a >= 2-GPU box (-m multigpu, one file per pytest process) together with scripts/bridge_multi_device_parity.py before releases that touch device placement. Validated 2026-07-16 on a 2-GPU box: both suites pass, and the parity sweep (5 architectures, n_devices=2 and device_map=balanced) reported bitwise-identical activations vs single-device (worst diff 0.0 across every cached hook); HookedTransformer's legacy test_multi_gpu.py 2-GPU subset also passed. The box debugging added two fail-loud boot guards: tied-weight pairs split across map entries, and mixed CPU+GPU maps (accelerate CPU offload, whose materialization hooks param-reading Bridge components bypass). CI provides the other tiers (CUDA via compatibility-checks → CPU-only in practice; MPS via mps-checks). See tests/AGENTS.md §MPS rules and the --ignore= list in checks.yml.
skipif(os.getenv("CI")) to avoid expensive HF fetches / large loads.
| Path | Reason |
|---|---|
unit/model_bridge/supported_architectures/test_gemma2_adapter.py:49 |
"Network/disk fetch of tiny Gemma2" |
integration/model_bridge/test_bridge_integration.py:801 |
"Skip Gemma2 in CI to avoid timeout" |
acceptance/model_bridge/compatibility/test_hook_completeness.py:156 |
"Gemma2 too large for CI" |
Big-model adapter tests use @pytest.mark.slow, CI tier filters -m "not slow":
| Path | Model |
|---|---|
integration/model_bridge/test_exaone4_adapter.py:17 |
EXAONE-4.0 1.2B |
integration/model_bridge/test_bitnet_adapter.py:15 |
BitNet 2.4B |
integration/model_bridge/test_glm_adapter.py:11 |
glm-edge 1.59B |
integration/model_bridge/test_glm_asr_adapter.py:12 |
GLM-ASR-Nano 2.26B (fp32 double-load) |
integration/model_bridge/test_pegasus_adapter.py:16 |
Pegasus-XSum 568M; distilled variants are asymmetric |
integration/model_bridge/test_exaone_adapter.py:18 |
EXAONE 2.4B; no working tiny mirror (hyper-accel/tiny-random-exaone ships stale remote code) |
integration/model_bridge/test_ouro_adapter.py:35 (skipif + slow) |
ByteDance/Ouro-1.4B: 2.8GB download + ~11GB RAM |
integration/test_jacobian_lens_kurtosis.py |
Qwen3.5-0.8B (~1.6GB) + lens artifacts + wikitext validation split; CPU-feasible (#1539 Tier-1) |
Un-skip: locally with HF_TOKEN sourced (slow-marked files: run the file directly or -m slow).
| Path | Reason |
|---|---|
integration/model_bridge/test_qwen3_moe_bridge.py:155,166 |
"Requires real weights — run via verify_models" |
Un-skip: /verify-model Qwen/Qwen3-MoE-... (tools/model_registry/AGENTS.md).
| Path | Reason | Issue |
|---|---|---|
unit/model_bridge/test_bridge_generate_no_tokenizer.py:30,128 |
skipif(_MACOS_ARM64) — KV-cache NaN |
Upstream PyTorch/HF on M-series Macs |
integration/model_bridge/test_bridge_generate_stopping_criteria.py |
skipif(_MACOS_ARM64), KV-cache NaN (one use_past_kv_cache=True test) |
Upstream PyTorch/HF on M-series Macs |
Un-skip: when upstream resolves. Don't bypass — produces NaN logits.
Entire test modules quarantined via module-level pytestmark. Significant coverage gap — priority to re-enable.
| Path | Reason |
|---|---|
acceptance/test_hooked_transformer.py:19 |
"CI test pollution" |
acceptance/test_hooked_encoder.py:13 |
same |
acceptance/test_hooked_encoder_decoder.py:10 |
same |
Un-skip: root-cause the test pollution (fixture-scope or import-ordering bug). Until then, these acceptance tiers are dark.
Un-skip: debug the underlying issue and remove the marker. Each removal lands in a focused PR with a regression test.
Read AGENTS.md §10 first — default answer is "fix the bug instead."
If a quarantine is genuinely right:
- Pick the right marker —
skipif(condition)for env gates;skip(reason=)for known-bad paths;xfail(strict=True, reason=)when you expect failure and want CI to alert if it passes. - Use a
reason=descriptive enough to look up — not"flaky"or"broken". - Add a row above with path, marker, "un-skip when" line.
- Whole-module
pytestmarkskips go in the⚠️ section for visibility.