Skip to content

[CI] pytorch-finetuning / quick-train-full-finetuning failed on halo (windows) #672

Open
Open
[CI] pytorch-finetuning / quick-train-full-finetuning failed on halo (windows)#672
Assignees
lucbruni-amd
Labels
bugci-test-failure

Description

@github-actions
github-actions
bot
opened on Aug 6, 2026

This issue was opened automatically by the Test Playbooks workflow after the test quick-train-full-finetuning failed on the main branch.

Failure scope

  • Playbook: pytorch-finetuning
  • Test id: quick-train-full-finetuning
  • Device: halo
  • Operating system: windows
  • Runner labels: self-hosted, Windows, halo
  • Runner name: xsj-aimlab-halo-0
  • Commit: cc719520734030394f02d7217832c970c7cea32a
  • Workflow run: https://github.com/amd/playbooks/actions/runs/31086878506

Hardware / OS to use to reproduce

Run the failing test on a machine that matches the runner labels above (OS = windows, device = halo). The repo's self-hosted runners already advertise these labels; if you reproduce locally, use the same OS family and the same AMD device class.

How to dispatch the same test from CI

Re-run only the failing playbook on the same matrix entry by triggering the workflow with the playbook id:

gh workflow run test-playbooks.yml --repo amd/playbooks -f playbook_id=pytorch-finetuning

The workflow's matrix narrows down to this (device, platform) combination automatically based on the playbook's tested_platforms.

How to run just this test locally

python .github/scripts/run_playbook_tests.py --playbook pytorch-finetuning --platform windows --device halo

The runner extracts test blocks from playbooks/*/pytorch-finetuning/README.md (the failing block starts around line 404).

Failing test (verbatim from the README)

  • Setup: finetune-venv\Scripts\activate
  • Timeout: 1200s
import os
import subprocess
import sys

os.environ["QUICK_TRAIN"] = "1"
os.environ["QUICK_TRAIN_MODEL"] = "unsloth/gemma-3-4b-it"
r = subprocess.run([sys.executable, "train_full_finetuning.py"], timeout=600)
sys.exit(r.returncode)

Result

  • Exit code: 3221226505

stderr (last lines)


Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:10<00:10, 10.74s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:17<00:00,  8.57s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:17<00:00,  8.89s/it]

Tokenizing train dataset:   0%|          | 0/6 [00:00<?, ? examples/s]
Tokenizing train dataset: 100%|██████████| 6/6 [00:00<00:00, 214.05 examples/s]

Truncating train dataset:   0%|          | 0/6 [00:00<?, ? examples/s]
Truncating train dataset: 100%|██████████| 6/6 [00:00<00:00, 749.01 examples/s]

Tokenizing eval dataset:   0%|          | 0/2 [00:00<?, ? examples/s]
Tokenizing eval dataset: 100%|██████████| 2/2 [00:00<00:00, 180.54 examples/s]

Truncating eval dataset:   0%|          | 0/2 [00:00<?, ? examples/s]
Truncating eval dataset: 100%|██████████| 2/2 [00:00<00:00, 291.77 examples/s]
The model is already on multiple devices. Skipping the move to device specified in `args`.

  0%|          | 0/1 [00:00<?, ?it/s]C:\actions-runner\_work\playbooks\playbooks\playbooks\supplemental\pytorch-finetuning\assets\finetune-venv\Lib\site-packages\transformers\integrations\sdpa_attention.py:96: UserWarning: Mem Efficient attention on Current AMD GPU is still experimental. Enable it with TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1. (Triggered internally at B:\src\pytorch\aten\src\ATen\native\transformers\hip\sdp_utils.cpp:384.)
  attn_output = torch.nn.functional.scaled_dot_product_attention(

stdout (last lines)

Loading dataset...
QUICK_TRAIN=1: using non-gated model for smoke test: unsloth/gemma-3-4b-it
QUICK_TRAIN=1: using 1 step and a tiny dataset (smoke test).
Train samples: 6, Test samples: 2
Total selected samples: 8

Loading unsloth/gemma-3-4b-it...
Note: Model is stored as MXFP4 on Hugging Face but will be loaded as BF16 for training
(This is expected - the warning about MXFP4 is informational)

Model loaded. Weights footprint: 8.60 GB
Gradient checkpointing enabled (saves memory during backprop)
Using bf16 mixed precision.
Starting Full Fine-tuning
Model: unsloth/gemma-3-4b-it
Trainable parameters: 4,300,079,472
Effective batch size: 16
Learning rate: 2e-05
Quick smoke mode enabled: tiny dataset + max_steps=1

Unhandled exception caught in c10/util/AbortHandler.h
00007FFFBA9DA31C00007FFFBA9C9A50 torch_python.dll!initModule [<unknown file> @ <unknown line number>]
00007FF8A9D5218700007FF8A9D52170 ucrtbase.dll!terminate [<unknown file> @ <unknown line number>]
00007FF88CAA113400007FF88CAA10F0 VCRUNTIME140.dll!_FrameUnwindFilter [<unknown file> @ <unknown line number>]
00007FF88CAB204C00007FF88CAB1A90 VCRUNTIME140.dll!memset [<unknown file> @ <unknown line number>]
00007FF88CAAE7BF00007FF88CAAE720 VCRUNTIME140.dll!_C_specific_handler [<unknown file> @ <unknown line number>]
00007FF8AC22479F00007FF8AC224700 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FF8AC105E9700007FF8AC105900 ntdll.dll!RtlLocateExtendedFeature [<unknown file> @ <unknown line number>]
00007FF8AC14D01100007FF8AC14CDF0 ntdll.dll!RtlRaiseException [<unknown file> @ <unknown line number>]
00007FF8A9681ADA00007FF8A9681A50 KERNELBASE.dll!RaiseException [<unknown file> @ <unknown line number>]
00007FF88CAA526700007FF88CAA51D0 VCRUNTIME140.dll!CxxThrowException [<unknown file> @ <unknown line number>]
00007FF86C205F1200007FF86C205DB0 c10_hip.dll!c10::cuda::c10_cuda_check_implementation [<unknown file> @ <unknown line number>]
00007FF86C20683C00007FF86C2067E0 c10_hip.dll!c10::cuda::SetDevice [<unknown file> @ <unknown line number>]
00007FF86C1EE79300007FF86C1C8F60 c10_hip.dll!c10::FreeCudaMemoryCallbacksRegistry [<unknown file> @ <unknown line number>]
00007FFF60A8C91500007FFF60A8C800 torch_hip.dll!at::detail::empty_strided_cuda [<unknown file> @ <unknown line number>]
00007FF88CAAF73000007FF88CAAF4D0 VCRUNTIME140.dll!_CxxFrameHandler3 [<unknown file> @ <unknown line number>]
00007FF88CAA3A8600007FF88CAA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FF88CAAEC0400007FF88CAAE720 VCRUNTIME140.dll!_C_specific_handler [<unknown file> @ <unknown line number>]
00007FF88CAA2C9400007FF88CAA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FF88CAAF54100007FF88CAAF4D0 VCRUNTIME140.dll!_CxxFrameHandler3 [<unknown file> @ <unknown line number>]
00007FF8AC22481F00007FF8AC224700 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FF8AC10210300007FF8AC101E80 ntdll.dll!RtlUnwindEx [<unknown file> @ <unknown line number>]
00007FF88CAAF0CA00007FF88CAAE720 VCRUNTIME140.dll!_C_specific_handler [<unknown file> @ <unknown line number>]
00007FF88CAA18E500007FF88CAA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FF88CAA1D0000007FF88CAA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FF88CAA2D9000007FF88CAA1230 VCRUNTIME140.dll!is_exception_typeof [<unknown file> @ <unknown line number>]
00007FF88CAAF54100007FF88CAAF4D0 VCRUNTIME140.dll!_CxxFrameHandler3 [<unknown file> @ <unknown line number>]
00007FF8AC22479F00007FF8AC224700 ntdll.dll!_chkstk [<unknown file> @ <unknown line number>]
00007FF8AC105E9700007FF8AC105900 ntdll.dll!RtlLocateExtendedFeature [<unknown file> @ <unknown line number>]
00007FF8AC14D01100007FF8AC14CDF0 ntdll.dll!RtlRaiseException [<unknown file> @ <unknown line number>]
00007FF8A9681ADA00007FF8A9681A50 KERNELBASE.dll!RaiseException [<unknown file> @ <unknown line number>]
00007FF88CAA526700007FF88CAA51D0 VCRUNTIME140.dll!CxxThrowException [<unknown file> @ <unknown line number>]
00007FF86C205F1200007FF86C205DB0 c10_hip.dll!c10::cuda::c10_cuda_check_implementation [<unknown file> @ <unknown line number>]
00007FF86C1DFA9800007FF86C1C8F60 c10_hip.dll!c10::FreeCudaMemoryCallbacksRegistry [<unknown file> @ <unknown line number>]
00007FF86C1D461B00007FF86C1C8F60 c10_hip.dll!c10::FreeCudaMemoryCallbacksRegistry [<unknown file> @ <unknown line number>]
00007FF86C1C9D5800007FF86C1C8F60 c10_hip.dll!c10::FreeCudaMemoryCallbacksRegistry [<unknown file> @ <unknown line number>]
00007FF879E3A0BF00007FF879E3A080 c10.dll!c10::StorageImpl::StorageImpl [<unknown file> @ <unknown line number>]
00007FFF7179406300007FFF71793710 torch_cpu.dll!at::detail::empty_generic [<unknown file> @ <unknown line number>]
00007FFF71794A8100007FFF71794910 torch_cpu.dll!at::detail::empty_strided_generic [<unknown file> @ <unknown line number>]
00007FFF60A8C8AB00007FFF60A8C800 torch_hip.dll!at::detail::empty_strided_cuda [<unknown file> @ <unknown line number>]
00007FFF60A8C9C300007FFF60A8C930 torch_hip.dll!at::detail::empty_strided_cuda [<unknown file> @ <unknown line number>]
00007FFF6013C40100007FFF6013C390 torch_hip.dll!at::native::empty_strided_cuda [<unknown file> @ <unknown line number>]
00007FFF60E74F6700007FFF60E74BF0 torch_hip.dll!at::cuda::empty_strided [<unknown file> @ <unknown line number>]
00007FFF6100327E00007FFF60FC9470 torch_hip.dll!at::cuda::_fused_adagrad_ [<unknown file> @ <unknown line number>]
00007FFF72BB580A00007FFF72BB5530 torch_cpu.dll!at::_ops::empty_strided::redispatch [<unknown file> @ <unknown line number>]
00007FFF72BB564400007FFF72BB5530 torch_cpu.dll!at::_ops::empty_strided::redispatch [<unknown file> @ <unknown line number>]
00007FFF730B39E100007FFF730A56D0 torch_cpu.dll!at::_ops::_foobar_out::redispatch [<unknown file> @ <unknown line number>]
00007FFF72BB3A3400007FFF72BB3860 torch_cpu.dll!at::_ops::empty_strided::call [<unknown file> @ <unknown line number>]
00007FFF72069ABE00007FFF720696A0 torch_cpu.dll!at::native::to [<unknown file> @ <unknown line number>]
00007FFF72068CB400007FFF72068020 torch_cpu.dll!at::native::_to_copy [<unknown file> @ <unknown line number>]
00007FFF731F76D300007FFF731F7390 torch_cpu.dll!at::compositeexplicitautograd::_fused_moving_avg_obs_fq_helper_outf [<unknown file> @ <unknown line number>]
00007FFF727CC25E00007FFF727CC160 torch_cpu.dll!at::_ops::_to_copy::redispatch [<unknown file> @ <unknown line number>]
00007FFF730D23A200007FFF730A56D0 torch_cpu.dll!at::_ops::_foobar_out::redispatch [<unknown file> @ <unknown line number>]
00007FFF727CC25E00007FFF727CC160 torch_cpu.dll!at::_ops::_to_copy::redispatch [<unknown file> @ <unknown line number>]
00007FFF7523D44D00007FFF75179A30 torch_cpu.dll!torch::autograd::impl::JitDecompRegisterer::JitDecompRegisterer [<unknown file> @ <unknown line number>]
00007FFF727CB04300007FFF727CAE80 torch_cpu.dll!at::_ops::_to_copy::call [<unknown file> @ <unknown line number>]
00007FFF720697DE00007FFF720696A0 torch_cpu.dll!at::native::to [<unknown file> @ <unknown line number>]
00007FFF7345853E00007FFF734582E0 torch_cpu.dll!at::compositeimplicitautograd::_autocast_to_full_precision [<unknown file> @ <unknown line number>]
00007FFF7293517C00007FFF72934FA0 torch_cpu.dll!at::_ops::to_dtype_layout::call [<unknown file> @ <unknown line number>]
00007FFF7171809800007FFF71717F70 torch_cpu.dll!at::Tensor::to [<unknown file> @ <unknown line number>]
00007FFF7211B48100007FFF7211B3B0 torch_cpu.dll!at::native::type_as [<unknown file> @ <unknown line number>]
00007FFF72D8DEFF00007FFF72D8DE20 torch_cpu.dll!at::_ops::type_as::call [<unknown file> @ <unknown line number>]
00007FFFBA5DC7C700007FFFBA4F8760 torch_python.dll!torch::PythonArgs::pyobject [<unknown file> @ <unknown line number>]
00007FF809FA569500007FF809FA4A80 python313.dll!PyErr_ExceptionMatches [<unknown file> @ <unknown line number>]
00007FF809F8704C00007FF809F86F70 python313.dll!PyObject_Vectorcall [<unknown file> @ <unknown line number>]

This issue is opened and deduplicated by .github/scripts/create_failure_issues.py. Close it once the failure is fixed; subsequent failures with the same scope will reopen a fresh issue.

Metadata

Metadata

Assignees

Labels

bugci-test-failure

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions