ci: self-hosted Windows stx runner broken — uv.exe blocked by Application Control, CMake missing CMAKE_ROOT

[itomek]

The self-hosted Windows stx runner pool is in a broken state and is failing jobs on main and on every PR that touches the relevant paths. Two distinct symptoms, same runner pool, both starting today (2026-08-04) — neither is caused by any PR's code.

1. uv.exe is blocked by an Application Control policy — breaks Test GAIA CLI on Windows (Full Integration)

Program 'uv.exe' failed to run: An Application Control policy has blocked this file

.github/actions/setup-venv/action.yml:46 installs uv via irm https://astral.sh/uv/install.ps1 | iex, which lands uv.exe in %USERPROFILE%\.local\bin. A WDAC/AppLocker policy on the box now blocks execution from that user-writable path, so the job dies in "Setup Python Environment" after ~8s, before any test runs.

2. CMake cannot find its own root — breaks C++ Integration Tests (STX)

CMake Error: Could not find CMAKE_ROOT !!!
CMake Error: Error executing cmake::LoadCache(). Aborting.

This one has been failing longer, across unrelated branches (cpp/tui-event-loop, cpp/coding-toolbelt, cpp/chunking), so it is not specific to any change either.

Both jobs declare runs-on: [self-hosted, Windows, <stx label>], so they land on the same machine(s).

Evidence it is the runner, not the code — test_gaia_cli_windows.yml history:

result	branch	time (UTC)
failure	issue-2765	14:04
failure	main	13:55
success	cpp/chunking	11:21
success	issue-2765	05:01

The same branch passed at 05:01 and failed at 14:04 with no code change to the setup path. main fails with the identical error.

What's needed (requires access to the box, which I don't have):

• Allow uv.exe under the Application Control policy, or pre-install uv to an allowed location and have setup-venv prefer it when present. Pinning to a machine-wide install is the more durable fix — a per-user download into %USERPROFILE%\.local\bin will keep tripping any policy of this kind.
• Repair the CMake install so CMAKE_ROOT resolves.

Until then, treat Test GAIA CLI on Windows (Full Integration) and C++ Integration Tests (STX) red as environmental on any PR, and don't chase them in PR diffs.

[itomek]

Blast radius is wider than the two jobs above — a third has now failed with the identical uv.exe block:

• Test GAIA CLI on Windows (Full Integration) — dies in "Setup Python Environment", ~23s
• Example Agents Integration Tests (stx) — dies in "Setup Python environment", ~24s

Both fail before executing a single test, so any PR they run against will show red for reasons unrelated to its diff.

Every workflow that pairs .github/actions/setup-venv with a self-hosted runner is a candidate to fail the same way as jobs get scheduled onto the affected box:

build_cpp.yml                        test_email_agent_eval.yml
claude-weekly-doc-walkthrough.yml    test_embeddings.yml
email_scorecard_refresh.yml          test_examples.yml
test_agent_behavior_e2e.yml          test_gaia_cli_windows.yml
test_agent_sdk.yml                   test_lemonade_server.yml
test_api.yml                         test_npu_embedder.yml
test_rag.yml                         test_sd.yml

(That list is the upper bound — it's every workflow referencing both, not a confirmed-failing set. Only the two named above are confirmed so far.)

Worth noting for whoever fixes it: the failure is in a composite action shared by all of them, so a single change to how uv is obtained in setup-venv fixes every one at once. Pre-installing uv machine-wide to a policy-allowed location and having setup-venv prefer it when present is the one-line-per-job fix; the current irm https://astral.sh/uv/install.ps1 | iex into %USERPROFILE%\.local\bin will keep tripping Application Control no matter how many times it's re-run.

[itomek]

The same Application Control block now appears to be hitting llama-server.exe, not just uv.exe — and it is currently blocking a merge-ready PR.

On #2981 the rag_quality + context_retention + tool_selection vs Gemma baselines check fails in preflight, twice, with:

LemonadeClientError: Request failed with status 500:
  {"code":"model_load_error",
   "message":"Failed to load model 'user.embeddinggemma-300m-GGUF': llama-server failed to start"}

The workflow ruled out the usual suspects on its own before giving up — it re-registered and re-pulled the model, then cleared the llama.cpp backend cache and retried. Its own conclusion:

Registration, download and the backend cache have all been ruled out, so this is the RUNNER … Check whether an Application Control policy is blocking llama-server.exe (#2826).

No Lemonade log is written on the runner either, so the actual llama-server failure is only visible on the Lemonade console there.

Why this is worth prioritising: the check is path-filtered to email-agent changes, so it skips on most PRs and looks quiet — but it fails on every PR that touches the email agent. #2981 is otherwise green (44 pass / 8 skip / 1 fail) and mergeable, with that one red being this. Any email-agent PR behind it will hit the same wall.

Re-running does not help and is expensive — the workflow says so explicitly: "Do NOT re-run the eval until it passes - it costs 3.5h and measures nothing."