Skip to content

[OpenVINO][Windows] onnxruntime_provider_test silently exits during MatMulNBits.Float32_4b_Accuracy0 #31760

Description

@tianleiwu

Describe the issue

The Windows OpenVINO CI job 93078339501 failed because onnxruntime_provider_test.exe terminated silently while running MatMulNBits.Float32_4b_Accuracy0.

This was not a logged numerical accuracy failure:

  • MatMulNBits.Float32_4b_Accuracy0 started at 2026-08-08T08:21:30.4228366Z.
  • About 4.1 seconds later, CTest reported onnxruntime_provider_test as failed.
  • There was no [ OK ] or [ FAILED ] result for the test, no gtest assertion, no C++ exception, and no Windows exception/status code in the log.
  • CTest reported 91% tests passed, 1 tests failed out of 11, returned exit status 8, and build.py consequently returned 1.
  • The NPU availability skips and expected negative-test error logs earlier in the job were unrelated.

The retry, job 93163934771, passed the same test in 5.117 seconds and completed onnxruntime_provider_test successfully.

Environment comparison

Failed attempt Successful retry
ORT commit 6c227c76265711189e15e08e5333430a29a897e7 Same
OpenVINO 2026.2.1 Same
OpenVINO package commit ede283a88e3 Same
CPU AMD EPYC 9V74 80-Core Processor Same
Runner 4f589377c000005 862958acc000001

The observed failure is therefore intermittent and runner-instance-specific so far.

Source analysis

  • The workflow downloads openvino_toolkit_windows_vc_mt_2026.2.1.21919.ede283a88e3_x86_64.zip; commit ede283a88e3 matches the OpenVINO 2026.2.1 tag.
  • OpenVINO's com.microsoft::MatMulNBits frontend implementation is identical between tags 2026.1.0 and 2026.2.1. It lowers the op to u4 Convert -> Subtract -> Multiply -> Reshape/Slice -> MatMul.
  • The relevant OpenVINO CPU conversion/FullyConnected source is also unchanged between those versions. The downstream oneDNN revision does differ (6b6492b1 to 87f65fdd), including FP32 matmul dispatch changes, but there is not enough evidence to identify that as the cause.
  • OpTester runs registered providers separately, CPU first and OpenVINO later. Because the process exited without an exception or provider message, the log does not identify which EP was active at termination.
  • No crash dump or test-result artifact was uploaded by the failed run.

PR #31759 proposes pinning Windows OpenVINO CI to 2026.1.0. That may be a temporary mitigation, but the current evidence does not establish a deterministic OpenVINO 2026.2.1 regression, and the MatMulNBits frontend itself is unchanged between the two versions.

To reproduce

The failure is not yet reliably reproducible:

  1. Build commit 6c227c76265711189e15e08e5333430a29a897e7 on Windows x64 with OpenVINO 2026.2.1 using .github/workflows/windows_openvino.yml.
  2. Run onnxruntime_provider_test.exe --gtest_filter=MatMulNBits.Float32_4b_Accuracy0 repeatedly.
  3. The first CI attempt terminated during the test; the retry passed on another runner with the same CPU model.

Suggested diagnostics for the next recurrence:

  1. Add temporary explicit CPU-only and OpenVINO-only variants to identify the active EP.
  2. Run the isolated test with --gtest_repeat=100.
  3. Enable Windows Error Reporting LocalDumps or ProcDump for onnxruntime_provider_test.exe, and upload full dumps as workflow artifacts.
  4. Set ONEDNN_VERBOSE=1 to capture the selected primitive and effective ISA before termination.
  5. Record the child process NT exit status rather than only CTest/build.py's wrapper exit codes.

Expected behavior

onnxruntime_provider_test should complete normally. If a native failure occurs, CI should retain enough information (provider, NT status, and dump) to identify the crashing module and stack.

System information

  • Platform: Windows, self-hosted GitHub Actions runner with Visual Studio 2022
  • Architecture: x64
  • ONNX Runtime installation: built from source
  • ONNX Runtime commit: 6c227c76265711189e15e08e5333430a29a897e7
  • Execution providers: CPU and OpenVINO
  • OpenVINO version: 2026.2.1 (ede283a88e3)
  • CPU: AMD EPYC 9V74 80-Core Processor
  • Related run: https://github.com/microsoft/onnxruntime/actions/runs/31247477067

Metadata

Metadata

Assignees

No one assigned

    Labels

    ep:OpenVINOissues related to OpenVINO execution providerep:oneDNNquestions/issues related to DNNL EPplatform:windowsissues related to the Windows platform

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions