Repository navigation
Add UI automation benchmarks (agent-driven and scripted perf) under benchmarks/ui - #1053
Open
Nikola Metulev (nmetulev) wants to merge 8 commits into
Open
Nikola Metulev (nmetulev) wants to merge 8 commits into
Nikola Metulev (nmetulev) wants to merge 8 commits into
Conversation
Adds a rerunnable benchmark where Copilot CLI drives Calculator and the WinUI 3 Gallery with a method (the winapp plugin, or none), and the harness scores the final app state itself through UIA, process counts and window state. Also adds a scripted, agent-free perf benchmark that times each winapp ui operation and fixed command sequences. Reuses benchmarks/agents/lib/Benchmark.psm1, extended with timing fields and Format-Count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Record the mode the first launched Calculator shows, switch every owned Calculator back to it at teardown instead of forcing Standard, and keep the record across interrupted runs. Launch readiness now waits for the nav button, which exists in every Calculator mode. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Invoke-InstanceSetup: don't enumerate .Keys on an empty expect hashtable under StrictMode - Start-ScenarioInstances: return an array so callers don't get a nested list - perf.json: set-value targets the Gallery search Edit (TextBox), not its Group wrapper - perf.ps1: exit 0/1 from the run outcome instead of a stray native exit code Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…timates - run.ps1: exit 0 only when every run passed and none stopped early, instead of a stray native exit code - run.ps1: rerunning a finished -OutDir keeps the existing summary.md instead of rewriting it without its header - config.json: per-session estimates from the first full run (7.1 credits, 0.46 min measured) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…n-agent-benchmark
- Read the ownership ledger with start times kept as their saved strings. ConvertFrom-Json turned them into DateTime, so recovery after an interrupted run never matched and left the processes the benchmark had launched running (and then treated them as foreign). - Record a run's abort reason inside finally, so the early returns for an app opened mid-run or a stuck setup process actually stop the loop. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Nikola Metulev (nmetulev)
added
the
agent-preparing
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Resume integrity, perf error handling, exit statuses, and sequence statistics currently produce skipped or misleading benchmark results.
6 open findings
What changed in this PR
Adds repeatable agent-driven and scripted performance benchmarks for winapp ui, including desktop setup, safety tracking, scoring, reporting, and comparison tooling.
Changes:
- Adds seven Calculator and WinUI Gallery benchmark scenarios.
- Adds extensible agent methods and scripted perf adapters.
- Adds offline Pester coverage and shared agent timing metrics.
| File | Description |
|---|---|
.gitignore |
Ignores UI benchmark output and cache. |
benchmarks/ui/README.md |
Documents setup, usage, results, and safety. |
benchmarks/ui/config.json |
Defines defaults and pinned Gallery build. |
benchmarks/ui/run.ps1 |
Runs and scores agent-driven scenarios. |
benchmarks/ui/perf.ps1 |
Runs scripted performance measurements. |
benchmarks/ui/perf.json |
Defines operations and command sequences. |
benchmarks/ui/setup-gallery.ps1 |
Builds and installs the benchmark Gallery. |
benchmarks/ui/lib/UiBenchmark.psm1 |
Implements schemas, scoring, resume, and reports. |
benchmarks/ui/lib/AppState.psm1 |
Manages application state and ownership. |
benchmarks/ui/lib/Perf.psm1 |
Implements adapters, statistics, and perf reports. |
benchmarks/ui/adapters/winapp.json |
Maps perf operations to winapp ui. |
benchmarks/ui/methods/winapp.json |
Configures the winapp agent method. |
benchmarks/ui/methods/none.json |
Configures the no-plugin control. |
benchmarks/ui/scenarios/calc-running/scenario.json |
Adds the Calculator baseline scenario. |
benchmarks/ui/scenarios/calc-not-running/scenario.json |
Adds the Calculator launch scenario. |
benchmarks/ui/scenarios/calc-two-instances/scenario.json |
Adds multi-window targeting coverage. |
benchmarks/ui/scenarios/calc-minimized-scientific/scenario.json |
Adds minimized-window coverage. |
benchmarks/ui/scenarios/gallery-combobox/scenario.json |
Adds Gallery navigation coverage. |
benchmarks/ui/scenarios/gallery-dialog-left-open/scenario.json |
Adds modal-dialog coverage. |
benchmarks/ui/scenarios/gallery-cold-start/scenario.json |
Adds Gallery cold-start coverage. |
benchmarks/ui/tests/UiBenchmark.Tests.ps1 |
Tests schemas, scoring, resume, and reports. |
benchmarks/ui/tests/Perf.Tests.ps1 |
Tests perf configuration and statistics. |
benchmarks/ui/tests/AppState.Tests.ps1 |
Tests mode restoration and ownership records. |
benchmarks/ui/tests/fixtures/scoring-cases.json |
Supplies recorded scoring cases. |
benchmarks/agents/lib/Benchmark.psm1 |
Adds tool/API/session timing metrics. |
benchmarks/agents/tests/Benchmark.Tests.ps1 |
Tests the new timing calculations. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+303
to
+309
| $totals = @($Runs | ForEach-Object { Get-Field $_ 'commandMs' } | Where-Object { $null -ne $_ } | ForEach-Object { [double]$_ }) | ||
| [pscustomobject]@{ | ||
| Runs = $Runs.Count | ||
| Passed = @($Runs | Where-Object { $_.status -eq 'pass' }).Count | ||
| Total = Get-PerfStats $totals | ||
| Calls = Get-Median @($Runs | ForEach-Object { @(Get-Field $_ 'steps').Count } | ForEach-Object { [double]$_ }) | ||
| } |
Comment on lines
+288
to
+289
| # Resume reruns cells when the method's definition or the skills it installs change. | ||
| Hash = Get-ShortHash (($raw -replace '\s', '') + "`n" + (@($expected | Sort-Object) -join ',')) |
Comment on lines
+95
to
+99
| Write-Host "Adapters: $(@($adapters | ForEach-Object { if ($_.CheckedIn) { $_.Name } else { "$($_.Name) ($($_.SourcePath))" } }) -join ', ')" | ||
| Write-Host "Operations: $(@($operations.Id) -join ', ')" | ||
| Write-Host " $($perf.Warmup) warm-up + $($perf.Repeat) timed calls each = $($runPlan.OperationCalls) commands" | ||
| Write-Host "Sequences: $(@($sequences.Id) -join ', ')" | ||
| Write-Host " $($perf.SequenceRepeat) run(s) each = $($runPlan.SequenceRuns) runs, $($runPlan.SequenceCalls) commands" |
Comment on lines
+168
to
+169
| $tail = (Get-FileTail $err 300).Trim() | ||
| if (-not $tail) { $tail = (Get-FileTail $out 300).Trim() } |
Comment on lines
+348
to
+349
| $failedSequences = @($results.sequences | Where-Object { $_.status -ne 'pass' }) | ||
| exit ([int]([bool]$abort -or $failedSequences.Count -gt 0)) |
Comment on lines
+535
to
+537
| # Without an explicit exit, the script would return whatever native command ran last. | ||
| $notPassed = @($finished | Where-Object { $_.status -ne 'pass' }) | ||
| exit ([int]($header.Contains('Stopped early') -or $notPassed.Count -gt 0)) |
…n-agent-benchmark # Conflicts: # benchmarks/agents/lib/Benchmark.psm1
Contributor
Build Metrics ReportValidation passed. All required build and validation jobs succeeded. Binary Sizes
.NET Test Results (TRX reports)Other suites are reflected in the overall validation status above. ✅ 8331 passed, 37 skipped out of 8368 tests in 1106.2s (-65.8s vs. baseline) Test Coverage✅ 87.5% line coverage, 82.2% branch coverage · ✅ no change vs. baseline CLI Startup Time62ms median (x64, Try This BuildInstalls the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing. & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 1053Switching between builds often?Put the tool on your PATH once: & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPathThen this build is just: winapp-pr 1053Run Updated 2026-10-10 19:48:05 UTC · commit |
Nikola Metulev (nmetulev)
added
ready-for-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
benchmarks/ui/, two repeatable benchmarks forwinapp ui. Rerun them after anywinapp uichange to measure its effect.run.ps1). A real Copilot CLI agent does natural-language UI tasks in inbox Calculator and WinUI 3 Gallery. The Gallery is pinned tov2.9.3, built from source once and cached. The benchmark measures the agent experience end to end, which is the number that matters: in an earlier comparison, model turns took 75–80% of wall time.perf.ps1). A scripted benchmark with no agent and no credits that trackswinapp ui's own speed so regressions show up.winapp --version),list-windows,search,inspect,get-property,get-value,invoke,set-value,wait-for,focusandscreenshot. Each runs 20 times after one discarded warm-up, and the report gives the median, p95 and max.Methods and adapters:
winapp(this repo's plugin plus the CLI) andnone(no plugin, andwinappis not available).winapp.-Method <method.json>or-Adapter <adapter.json>. The README documents both formats.Safety (both benchmarks):
-Force.run.ps1is resumable: rerunning into the same-OutDirskips finished cells.benchmarks/agents/lib/Benchmark.psm1is reused rather than copied. A few helpers were generalized so both benchmarks share Copilot launch, events.jsonl parsing and compare.benchmarks/agentsbehavior is unchanged; its Pester suite passes.Usage Example
Results go to
benchmarks\ui\results\<timestamp>\, which is gitignored and holds JSON plussummary.md.Related Issue
None.
Type of Change
Checklist
Screenshots / Demo
First full run, measured on 2026-10-09:
claude-sonnet-5.5, 5 iterations.Agent benchmark: 70/70 passed, with no collateral damage. The run took 32 minutes and used 497.5 credits.
winappnoneHow
winappcompared withnone:winappwas 20–30% faster. Combobox took 19.4 s vs 27.2 s, the dialog left open 38.6 s vs 48.3 s, and cold start 30.3 s vs 39.6 s.winappwas slower, 36.9 s and 10 turns vs 15.0 s and 3 turns. See Additional Notes.winappused about 1.7× the credits, mostly from the extra input tokens of loading the UI-automation skill.Perf benchmark (median / p95, ms, measured from process start to exit):
Sequence medians:
Additional Notes
Validation: the UI suite ran with
Invoke-Pester -Path benchmarks\ui\testsand the agents suite withInvoke-Pester -Path benchmarks\agents\tests. Both use no UI and cover:Results: UI 130/130 passed;
benchmarks/agents120/120 passed, so it is unchanged. Before the full run, the smoke run passed and fixed several selector and teardown bugs.Known winapp gap that the perf suite surfaces on purpose: while Calculator is minimized,
winapp uisees only the root window, sofocus CalculatorResultsfails with "No element found".focuson the root window's slug does restore it, but that slug changes on every launch and the command exits 1. Thecalc-minimized-scientificsequence is kept as a regression signal; it will pass oncewinapp uican restore a minimized window. The same gap explains the slower agent result on that scenario.Not run in CI: both benchmarks drive the desktop and the agent benchmark costs credits, so they are run manually. Only the Pester suites are suitable for automation.