This issue was opened automatically by the Test Playbooks workflow after the test run-llm-simple failed on the main branch.
Failure scope
Hardware / OS to use to reproduce
Run the failing test on a machine that matches the runner labels above (OS = windows, device = halo). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.
How to dispatch the same test from CI
Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:
gh workflow run test-playbooks.yml --repo amd/playbooks -f playbook_id=pytorch-rocm-llms
The workflow's matrix narrows down to this (device, platform) combination automatically based on the playbook's tested_platforms.
How to run just this test locally
python .github/scripts/run_playbook_tests.py --playbook pytorch-rocm-llms --platform windows --device halo
The runner extracts test blocks from playbooks/*/pytorch-rocm-llms/README.md (the failing block starts around line 448).
Failing test (verbatim from the README)
- Setup:
pytorch-env\Scripts\activate
- Timeout:
600s
python run_llm.py --model openai/gpt-oss-20b
Result
stderr (last lines)
`torch_dtype` is deprecated! Use `dtype` instead!
MXFP4 quantization requires Triton and kernels installed: CUDA requires Triton >= 3.4.0, XPU requires Triton >= 3.5.0, we will default to dequantizing the model to bf16
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:16<00:33, 16.91s/it]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:32<00:16, 16.22s/it]
stdout (last lines)
========== ROCm Configuration==========
ROCm available: True
GPU: AMD Radeon(TM) 8060S Graphics
GPU Memory: 115.87 GB
Unhandled exception caught in c10/util/AbortHandler.h
00007FF97D60A31C00007FF97D5F9A50 torch_python.dll!initModule [<unknown file> @ <unknown line number>]
00007FFAB454218700007FFAB4542170 ucrtbase.dll!terminate [<unknown file> @ <unknown line number>]
00007FFA8FDA113400007FFA8FDA10F0 VCRUNTIME140.dll!_FrameUnwindFilter [<unknown file> @ <unknown line number>]
00007FFA8FDB204C00007FFA8FDB1A90 VCRUNTIME140.dll!memset [<unknown file> @ <unknown line number>]
00007FFA8FDAE7BF00007FFA8FDAE720 VCRUNTIME140.dll!_C_specific_handler [<unknown file> @ <unknown line number>]
00007FFAB730479F00007FFAB7304700 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FFAB71E5E9700007FFAB71E5900 ntdll.dll!RtlLocateExtendedFeature [<unknown file> @ <unknown line number>]
00007FFAB722D01100007FFAB722CDF0 ntdll.dll!RtlRaiseException [<unknown file> @ <unknown line number>]
00007FFAB4B81ADA00007FFAB4B81A50 KERNELBASE.dll!RaiseException [<unknown file> @ <unknown line number>]
00007FFA8FDA526700007FFA8FDA51D0 VCRUNTIME140.dll!CxxThrowException [<unknown file> @ <unknown line number>]
00007FFA7BFB5F1200007FFA7BFB5DB0 c10_hip.dll!c10::cuda::c10_cuda_check_implementation [<unknown file> @ <unknown line number>]
00007FFA7BFB683C00007FFA7BFB67E0 c10_hip.dll!c10::cuda::SetDevice [<unknown file> @ <unknown line number>]
00007FF97EE7745400007FF97EE77360 torch_hip.dll!torch::cuda::CUDAPluggableAllocator::CUDAPluggableAllocator::isEnabled [<unknown file> @ <unknown line number>]
00007FF97F2DFA8F00007FF97F2D69D0 torch_hip.dll!at::native::slow_conv2d_backward_cuda [<unknown file> @ <unknown line number>]
00007FFA8FDAF73000007FFA8FDAF4D0 VCRUNTIME140.dll!_CxxFrameHandler3 [<unknown file> @ <unknown line number>]
00007FFA8FDA3A8600007FFA8FDA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FFA8FDAEC0400007FFA8FDAE720 VCRUNTIME140.dll!_C_specific_handler [<unknown file> @ <unknown line number>]
00007FFA8FDA2C9400007FFA8FDA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FFA8FDAF54100007FFA8FDAF4D0 VCRUNTIME140.dll!_CxxFrameHandler3 [<unknown file> @ <unknown line number>]
00007FFAB730481F00007FFAB7304700 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FFAB71E210300007FFAB71E1E80 ntdll.dll!RtlUnwindEx [<unknown file> @ <unknown line number>]
00007FFA8FDAF0CA00007FFA8FDAE720 VCRUNTIME140.dll!_C_specific_handler [<unknown file> @ <unknown line number>]
00007FFA8FDA18E500007FFA8FDA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FFA8FDA1D0000007FFA8FDA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FFA8FDA2D9000007FFA8FDA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FFA8FDAF54100007FFA8FDAF4D0 VCRUNTIME140.dll!_CxxFrameHandler3 [<unknown file> @ <unknown line number>]
00007FFAB730479F00007FFAB7304700 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FFAB71E5E9700007FFAB71E5900 ntdll.dll!RtlLocateExtendedFeature [<unknown file> @ <unknown line number>]
00007FFAB722D01100007FFAB722CDF0 ntdll.dll!RtlRaiseException [<unknown file> @ <unknown line number>]
00007FFAB4B81ADA00007FFAB4B81A50 KERNELBASE.dll!RaiseException [<unknown file> @ <unknown line number>]
00007FFA8FDA526700007FFA8FDA51D0 VCRUNTIME140.dll!CxxThrowException [<unknown file> @ <unknown line number>]
00007FFA7BFB5F1200007FFA7BFB5DB0 c10_hip.dll!c10::cuda::c10_cuda_check_implementation [<unknown file> @ <unknown line number>]
00007FFA7BF7230900007FFA7BF722A0 c10_hip.dll!c10::cuda::memcpy_and_sync [<unknown file> @ <unknown line number>]
00007FF97F2DF3DA00007FF97F2D69D0 torch_hip.dll!at::native::slow_conv2d_backward_cuda [<unknown file> @ <unknown line number>]
00007FF991E415CA00007FF991E40C60 torch_cpu.dll!at::native::copy_ [<unknown file> @ <unknown line number>]
00007FF992EDFC2D00007FF992EDFB40 torch_cpu.dll!at::_ops::copy_::call [<unknown file> @ <unknown line number>]
00007FF9921C8CE500007FF9921C8020 torch_cpu.dll!at::native::_to_copy [<unknown file> @ <unknown line number>]
00007FF9933576D300007FF993357390 torch_cpu.dll!at::compositeexplicitautograd::_fused_moving_avg_obs_fq_helper_outf [<unknown file> @ <unknown line number>]
00007FF99292C25E00007FF99292C160 torch_cpu.dll!at::_ops::_to_copy::redispatch [<unknown file> @ <unknown line number>]
00007FF9932323A200007FF9932056D0 torch_cpu.dll!at::_ops::_foobar_out::redispatch [<unknown file> @ <unknown line number>]
00007FF99292C25E00007FF99292C160 torch_cpu.dll!at::_ops::_to_copy::redispatch [<unknown file> @ <unknown line number>]
00007FF99539D44D00007FF9952D9A30 torch_cpu.dll!torch::autograd::impl::JitDecompRegisterer::JitDecompRegisterer [<unknown file> @ <unknown line number>]
00007FF99292B04300007FF99292AE80 torch_cpu.dll!at::_ops::_to_copy::call [<unknown file> @ <unknown line number>]
00007FF9921C97DE00007FF9921C96A0 torch_cpu.dll!at::native::to [<unknown file> @ <unknown line number>]
00007FF9935B853E00007FF9935B82E0 torch_cpu.dll!at::compositeimplicitautograd::_autocast_to_full_precision [<unknown file> @ <unknown line number>]
00007FF992A9517C00007FF992A94FA0 torch_cpu.dll!at::_ops::to_dtype_layout::call [<unknown file> @ <unknown line number>]
00007FF99187809800007FF991877F70 torch_cpu.dll!at::Tensor::to [<unknown file> @ <unknown line number>]
00007FF97D2321C600007FF97D230AB0 torch_python.dll!THPPointer<_frame>::~THPPointer<_frame> [<unknown file> @ <unknown line number>]
00007FF97D1396AD00007FF97D128760 torch_python.dll!torch::PythonArgs::pyobject [<unknown file> @ <unknown line number>]
00007FFA2559AD3400007FFA2559ACD0 python313.dll!PyThread_get_thread_ident [<unknown file> @ <unknown line number>]
00007FFA2564D93300007FFA2564D870 python313.dll!PyObject_Call [<unknown file> @ <unknown line number>]
00007FFA766EA701 <unknown symbol address> _safetensors_rust.pyd!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFA766D10A6 <unknown symbol address> _safetensors_rust.pyd!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFA766E4A07 <unknown symbol address> _safetensors_rust.pyd!<unknown symbol> [<unknown file> @ <unknown line number>]
00007FFA2561D58F00007FFA2561D554 python313.dll!PyObject_GetItem [<unknown file> @ <unknown line number>]
00007FFA2563490D00007FFA256320D0 python313.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFA255BEA0400007FFA255BE730 python313.dll!PyObject_CallNoArgs [<unknown file> @ <unknown line number>]
00007FFA2564D98B00007FFA2564D870 python313.dll!PyObject_Call [<unknown file> @ <unknown line number>]
00007FFA2563653000007FFA256320D0 python313.dll!PyEval_EvalFrameDefault [<unknown file> @ <unknown line number>]
00007FFA256780C400007FFA25677EC4 python313.dll!PyEval_EvalCode [<unknown file> @ <unknown line number>]
00007FFA25677F7A00007FFA25677EC4 python313.dll!PyEval_EvalCode [<unknown file> @ <unknown line number>]
00007FFA2567834100007FFA25677EC4 python313.dll!PyEval_EvalCode [<unknown file> @ <unknown line number>]
00007FFA2567844C00007FFA25677EC4 python313.dll!PyEval_EvalCode [<unknown file> @ <unknown line number>]
00007FFA256B4CB700007FFA256B4AD4 python313.dll!PyDict_PopString [<unknown file> @ <unknown line number>]
This issue is opened and deduplicated by .github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.
This issue was opened automatically by the Test Playbooks workflow after the test
run-llm-simplefailed on themainbranch.Failure scope
pytorch-rocm-llmsrun-llm-simplehalowindowsself-hosted,Windows,haloxsj-aimlab-halo-00c9674c64fe8be56732206adea82b8e6fbfd315fHardware / OS to use to reproduce
Run the failing test on a machine that matches the runner labels above (OS =
windows, device =halo). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.How to dispatch the same test from CI
Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:
The workflow's matrix narrows down to this
(device, platform)combination automatically based on the playbook'stested_platforms.How to run just this test locally
The runner extracts test blocks from
playbooks/*/pytorch-rocm-llms/README.md(the failing block starts around line 448).Failing test (verbatim from the README)
pytorch-env\Scripts\activate600sResult
3221226505stderr (last lines)
stdout (last lines)
This issue is opened and deduplicated by
.github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.