Skip to content

testing.md

Latest commit

 

History

History
287 lines (255 loc) · 15.7 KB

testing.md

File metadata and controls

287 lines (255 loc) · 15.7 KB
Raw

Testing and Benchmarking

The GPU-only proof strategy and July 2026 validation record are documented in apple-gpu-execution.md. In particular, a numerical result is not a GPU pass unless the test also observes completed Apple-GPU provenance and rejects CPU fallback/stub sources.

A skip is not a pass either. Harnesses must not downgrade a wrong answer to a skip, and a skip must be a precondition checked before the work runs, never a verdict reached after it. Read the pass count together with the skip count, and inspect any long-lived skip with ctest -V before trusting it — correctness-audit-2026-07-26.md records two cases where a skip concealed a real failure and one where a test had never executed on any machine.

CI layers

The workflow definitions are temporarily disabled by their .yml.disabled suffixes, so GitHub does not discover or run them. Rename them back to .yml to re-enable the gates described below.

.github/workflows/ci.yml.disabled defines the hosted gate for pushes and pull requests. It uses the macos-15 Apple Silicon image and covers both supported build policies:

Configuration Purpose
Release + binary shim off Shipping, source-first configuration
Debug + binary shim on Development and opt-in alias coverage

The hosted job builds the complete project, runs the CTest tests carrying the hosted label, and installs into a temporary prefix to check tools, CUDA toolchain shims, library aliases, and the conditional libcuda.dylib layout. The label contains parser, CFG/SSA, typed IR/MSL, PTX lowering, AIR container, negative-path, headers, CLI, cache, ABI registration, and packaging checks. It does not contain GPU execution tests. Hosted success is therefore a compiler and distribution claim, not an Apple-GPU execution claim.

Run the same selection locally:

bash scripts/ci_report.sh build \
  --require-tests \
  --label-regex '^hosted$'

.github/workflows/gpu-ci.yml.disabled defines the spec §10.7 hardware layer. Its runner must have the standard self-hosted labels self-hosted, macOS, and ARM64, plus the custom label ci-m1. When enabled, it runs only on pushes to main and manual dispatch, never on pull requests. Set the repository Actions variable CUMETAL_GPU_CI_ENABLED=true after the runner is commissioned to enable the push trigger.

The GPU workflow first runs a narrow proof set spanning Metal library loading, runtime launch, streams, atomics, warp masks, source-first executable linking, and numerical PTX execution. That selection uses both --require-tests and --require-no-skips. Only after every proof test executes does the workflow run the full correctness suite (excluding the separately managed benchmark gate), where optional external-project tests may still report explicit skips.

The report policy flags are intentionally distinct:

  • --require-tests rejects an empty CTest label or regex selection.
  • --require-no-skips rejects any skip in a prerequisite-complete proof gate.
  • The full suite should not use --require-no-skips, because optional llm.c, llama.cpp, PhysX, and multi-Xcode checks are environment-dependent.

Runtime execution tests

These tests compile Metal kernels with xcrun and run them through the CuMetal runtime:

ctest --test-dir build -R functional_runtime_vector_add --output-on-failure
ctest --test-dir build -R functional_runtime_vector_add_heap_alloc --output-on-failure
ctest --test-dir build -R functional_runtime_vector_add_cu --output-on-failure
ctest --test-dir build -R functional_sample_vector_add --output-on-failure
ctest --test-dir build -R functional_runtime_matrix_mul --output-on-failure
ctest --test-dir build -R functional_runtime_stream_vector_add --output-on-failure
ctest --test-dir build -R functional_runtime_null_stream_sync --output-on-failure
ctest --test-dir build -R functional_runtime_stream_per_thread --output-on-failure
ctest --test-dir build -R functional_runtime_async_memops --output-on-failure
ctest --test-dir build -R functional_runtime_event --output-on-failure
ctest --test-dir build -R functional_runtime_stream_wait_event --output-on-failure
ctest --test-dir build -R functional_runtime_stream_query --output-on-failure
ctest --test-dir build -R functional_runtime_memcpy_kind --output-on-failure
ctest --test-dir build -R functional_runtime_symbol_memcpy --output-on-failure
ctest --test-dir build -R functional_runtime_mem_get_info --output-on-failure
ctest --test-dir build -R functional_runtime_host_alloc --output-on-failure
ctest --test-dir build -R functional_runtime_host_pointer_api --output-on-failure
ctest --test-dir build -R functional_runtime_stream_flags --output-on-failure
ctest --test-dir build -R functional_runtime_stream_callback --output-on-failure
ctest --test-dir build -R functional_runtime_device_api --output-on-failure
ctest --test-dir build -R functional_runtime_device_properties --output-on-failure
ctest --test-dir build -R functional_runtime_device_attribute --output-on-failure
ctest --test-dir build -R functional_runtime_device_reset --output-on-failure
ctest --test-dir build -R functional_runtime_device_flags --output-on-failure
ctest --test-dir build -R functional_runtime_error_api --output-on-failure
ctest --test-dir build -R functional_runtime_profiler_api --output-on-failure
ctest --test-dir build -R functional_curand_uniform --output-on-failure
ctest --test-dir build -R functional_cublas_api --output-on-failure
ctest --test-dir build -R functional_driver_vector_add --output-on-failure
ctest --test-dir build -R functional_driver_matrix_mul --output-on-failure
ctest --test-dir build -R functional_driver_null_stream_sync --output-on-failure
ctest --test-dir build -R functional_driver_device_api --output-on-failure
ctest --test-dir build -R functional_driver_error_api --output-on-failure
ctest --test-dir build -R functional_driver_profiler_api --output-on-failure
ctest --test-dir build -R functional_driver_device_query --output-on-failure
ctest --test-dir build -R functional_driver_device_attribute --output-on-failure
ctest --test-dir build -R functional_driver_stream_flags --output-on-failure
ctest --test-dir build -R functional_driver_stream_per_thread --output-on-failure
ctest --test-dir build -R functional_driver_stream_callback --output-on-failure
ctest --test-dir build -R functional_driver_context_switch --output-on-failure
ctest --test-dir build -R functional_driver_context_requirements --output-on-failure
ctest --test-dir build -R functional_driver_async_memcpy --output-on-failure
ctest --test-dir build -R functional_driver_memset --output-on-failure
ctest --test-dir build -R functional_driver_mem_get_info --output-on-failure
ctest --test-dir build -R functional_driver_mem_alloc_managed --output-on-failure
ctest --test-dir build -R functional_driver_host_alloc --output-on-failure
ctest --test-dir build -R functional_driver_host_pointer_api --output-on-failure
ctest --test-dir build -R functional_driver_module_load_data --output-on-failure
ctest --test-dir build -R functional_driver_module_load_data_ptx --output-on-failure
ctest --test-dir build -R functional_driver_launch_extra --output-on-failure
ctest --test-dir build -R functional_driver_launch_extra_scalar --output-on-failure
ctest --test-dir build -R functional_driver_stream_wait_event --output-on-failure
ctest --test-dir build -R functional_runtime_axpy_offset --output-on-failure
ctest --test-dir build -R functional_runtime_atomic --output-on-failure
ctest --test-dir build -R functional_runtime_atomic_shared --output-on-failure
ctest --test-dir build -R functional_runtime_warp_shuffle --output-on-failure
ctest --test-dir build -R functional_runtime_warp_vote --output-on-failure
ctest --test-dir build -R functional_runtime_warp_size_lane --output-on-failure
ctest --test-dir build -R functional_runtime_warp_partial_mask --output-on-failure
ctest --test-dir build -R functional_runtime_fp16_ops --output-on-failure
ctest --test-dir build -R functional_runtime_fp64_ops --output-on-failure
ctest --test-dir build -R functional_runtime_shared_reduce --output-on-failure
ctest --test-dir build -R functional_runtime_grid_2d --output-on-failure
ctest --test-dir build -R functional_runtime_grid_3d --output-on-failure
ctest --test-dir build -R functional_runtime_cooperative_launch --output-on-failure
ctest --test-dir build -R functional_runtime_struct_arg --output-on-failure
ctest --test-dir build -R functional_runtime_barrier_order --output-on-failure
ctest --test-dir build -R functional_runtime_cp_async_emul --output-on-failure
ctest --test-dir build -R functional_runtime_device_properties --output-on-failure
ctest --test-dir build -R functional_runtime_occupancy --output-on-failure
ctest --test-dir build -R functional_runtime_device_limits --output-on-failure
ctest --test-dir build -R functional_runtime_printf --output-on-failure
ctest --test-dir build -R functional_cufft_c2c --output-on-failure
ctest --test-dir build -R functional_runtime_ptx_lowering_regression --output-on-failure
ctest --test-dir build -R functional_runtime_matrix_mul_tiled --output-on-failure
ctest --test-dir build -R functional_runtime_dynamic_shared --output-on-failure
ctest --test-dir build -R functional_runtime_registration_printf --output-on-failure
ctest --test-dir build -R functional_driver_extended_api --output-on-failure
# CUDA registration-path tests (always built; CUMETAL_ENABLE_CUDA_REGISTRATION=ON):
ctest --test-dir build -R functional_runtime_registration_path --output-on-failure
ctest --test-dir build -R functional_runtime_call_config_registration --output-on-failure
ctest --test-dir build -R functional_runtime_registration_fatbin_ptx --output-on-failure
ctest --test-dir build -R functional_runtime_legacy_launch_registration --output-on-failure
ctest --test-dir build -R functional_runtime_registration_fatbinary2_symbols --output-on-failure
ctest --test-dir build -R functional_runtime_registration_var_symbol --output-on-failure
ctest --test-dir build -R unit_allocation_table --output-on-failure
ctest --test-dir build -R unit_module_cache --output-on-failure
ctest --test-dir build -R unit_library_conflict --output-on-failure
ctest --test-dir build -R unit_metallib_parser --output-on-failure
ctest --test-dir build -R unit_ptx_parser --output-on-failure
ctest --test-dir build -R unit_intrinsic_lower --output-on-failure
ctest --test-dir build -R unit_printf_lower --output-on-failure
ctest --test-dir build -R unit_addrspace_pass --output-on-failure
ctest --test-dir build -R unit_metadata_pass --output-on-failure
ctest --test-dir build -R unit_phase1_pipeline --output-on-failure
ctest --test-dir build -R unit_ptx_lower_to_llvm --output-on-failure
ctest --test-dir build -R unit_cuda_fp16_host --output-on-failure
ctest --test-dir build -R unit_cuda_vector_types --output-on-failure
ctest --test-dir build -R unit_ptx_lower_to_metal --output-on-failure
ctest --test-dir build -R unit_cumetal_bench_help --output-on-failure
ctest --test-dir build -R unit_cumetal_bench_invalid_arg --output-on-failure
ctest --test-dir build -R unit_cumetal_bench_ratio_gate --output-on-failure
ctest --test-dir build -R unit_runtime_library_aliases --output-on-failure
# libcuda.dylib alias tests (`CUMETAL_ENABLE_BINARY_SHIM=ON` only):
ctest --test-dir build -R unit_binary_shim_symbol_exports --output-on-failure
ctest --test-dir build -R unit_binary_shim_library_alias --output-on-failure
ctest --test-dir build -R unit_binary_shim_link_alias --output-on-failure
ctest --test-dir build -R unit_library_link_aliases --output-on-failure
ctest --test-dir build -R ptx_sweep_supported_ops --output-on-failure
ctest --test-dir build -R ptx_sweep_unsupported_ops --output-on-failure
ctest --test-dir build -R unit_install_uninstall_scripts --output-on-failure

Conformance suite

Phase 4 conformance gate over functional tests:

ctest --test-dir build -R conformance_phase4_functional --output-on-failure
ctest --test-dir build -R conformance_llmc_gpt2fp32cu --output-on-failure
ctest --test-dir build -R functional_cuda_projects_ --output-on-failure

# Manifest-complete strict sweep with classified TSV/JSON output:
python3 tests/cuda_projects/sweep_cuda_projects.py

Notes:

  • conformance_phase4_functional now prints per-test progress ([i/N]) and applies a per-test timeout.
  • Override per-test timeout with CUMETAL_CONFORMANCE_SINGLE_TEST_TIMEOUT (seconds, default 120).
  • air_abi_xcode_matrix_regression uses CUMETAL_XCODE15_DEVELOPER_DIR/CUMETAL_XCODE16_DEVELOPER_DIR when set. If unset, it falls back to xcode-select -p for both slots (single-Xcode mode).
  • functional_cuda_projects_* compiles and runs standalone CUDA sample programs under tests/cuda_projects/ (SGEMM naive/shmem/2d, reduction, transpose). It requires Clang, xcrun, and the matching libcumetal build. Unsupported lowering is reported as an exit-77 skip.
  • The standalone sweep uses strict classification instead of CTest skip semantics and distinguishes prerequisite skips, compile/link failures, unsupported kernels, numerical failures, crashes, timeouts, and other runtime errors. Its manifest check fails if a new standalone .cu fixture is not enrolled.
  • conformance_llmc_gpt2fp32cu is registered only when llm.c is configured (set CUMETAL_LLMC_DIR before CMake configure, or place checkout at ../llm.c relative to this repo root).
  • conformance_llmc_gpt2fp32cu skips when gpt2_124M.bin is missing from the llm.c checkout (place under checkout root or dev/data/ per upstream layout).
  • When registered, conformance_llmc_gpt2fp32cu auto-wires CuMetal's LLVM+fatbin shim flow: scripts/build_llmc_test_gpt2fp32cu.sh + scripts/run_llmc_test_gpt2fp32cu.sh.
  • conformance_llmc_gpt2fp32cu fails on any TENSOR NOT OK marker and requires overall okay: 1 in output.
  • By default, the gate disables llm.c CPU emulation, requires numerical parity, and requires successful Apple-GPU provenance. CPU fallback and stub provenance are failures.
  • To force pure PTX-lowered execution (no llm.c emulation fallback), run:
CUMETAL_LLMC_REQUIRE_NO_EMULATION=1 \
ctest --test-dir build -R conformance_llmc_gpt2fp32cu --output-on-failure

Direct invocation with custom threshold/regex:

./tests/conformance/run_conformance_suite.sh build 90 '^functional_'

Optional llm.c stress harness setup:

export CUMETAL_LLMC_DIR="/path/to/llm.c"
bash scripts/fetch_llmc_assets.sh   # gpt2_124M.bin + debug state (once)
# optional overrides:
export CUMETAL_LLMC_BUILD_CMD="scripts/build_llmc_test_gpt2fp32cu.sh"
export CUMETAL_LLMC_TEST_CMD="scripts/run_llmc_test_gpt2fp32cu.sh"
# optional: gradient checker tolerance applied by build shim patching
export CUMETAL_LLMC_GRAD_TOL="1.2e-2"
# optional: hard-disable llm.c runtime emulation fallback
export CUMETAL_DISABLE_LLMC_EMULATION="1"
export CUMETAL_ENABLE_LLMC_CPU_EMULATION="0"

Kernel argument notes:

  • Scalar kernel params should be passed as CUMETAL_ARG_BYTES (with size_bytes) or via a device buffer.
  • Passing a host scalar pointer as CUMETAL_ARG_BUFFER is invalid and will fail launch with cudaErrorInvalidDevicePointer.

Benchmark runner

./scripts/generate_reference_metallib.sh
./build/cumetal_bench \
  --metallib tests/air_abi/reference/reference.metallib \
  --kernel vector_add \
  --elements 262144 \
  --warmup 5 \
  --iterations 50 \
  --max-ratio 2.0

If xcrun metal/xcrun metallib are unavailable

./build/cumetal-air-emitter \
  --input tests/air_abi/reference/vector_add_air.ll \
  --output /tmp/vector_add.experimental.metallib \
  --mode experimental \
  --overwrite

./build/air_validate /tmp/vector_add.experimental.metallib \
  --require-function-list --require-metadata