fix: recover cross-process UI when the frontmost tree is an empty shell

[onevcat]

Fixes #64.

Problem

A system document picker (or any remote-process presentation) leaves the frontmost app's accessibility tree an empty shell while the host app stays frontmost. describe-ui produced zero entries in that state — the agent goes blind the moment an app opens a file picker — even though --point could hit the visible picker elements from another process. Reproduced live on current main (Safari + <input type="file"> → cross-process picker → App: header only, 0 probe(s) calibration advisory, zero entries).

Root cause & fix

The tree fetch rides accessibilityElementForFrontmostApplication() and serialized with bare options. Upstream idb already ships the missing capability: FBAccessibilityRequestOptions.remoteContentOptions grid-hit-tests the screen regions the frontmost tree doesn't cover and returns other processes' elements (tagged, deduped by pid/frame) — sim-use simply never enabled it.

The fetch now detects the shell and retries once with discovery enabled:

• isEmptyShellTree (pure, unit-tested): a payload is a shell iff no non-application node carries a positive-area frame. The application container's own frame is just the screen rectangle and proves nothing — observed live both as a bare {pid, role} root (the issue's 0.10.0-era shape) and as a full-screen-framed AXApplication with zero children (current runtimes). Unrecognized shapes and oversized trees fail closed to the no-retry path.
• Recovered results carry a remote_content_recovery advisory (envelope advisory key, merged with any calibration advisory) warning that the hierarchy is flat and may not cover every element.
• Two-phase by design — the hot path is untouched. Spike measurements that shaped this: healthy native screens report coverage=1.0 and the discovery grid probes nothing; sparse-but-valid WebView pages report coverage as low as 0.03 while their tree is fine, which is why the trigger is the shell predicate and not a coverage threshold; the empty scene reports coverage=0.0 and discovery restores ~20% coverage of real elements.

Testing

• TDD: EmptyShellTreeTests written first against a never-retry stub (4 red / 3 green). Honest note: the first predicate draft keyed on the issue's 0.10.0 output shape (frameless root) and missed the current runtime's shell (full-screen-framed, childless) — the live failing scene supplied the second fixture, red again, then green. The shipped suite pins both shapes.
• E2E: new RemoteContentRecoveryTests (SIM_USE_E2E-gated, registered in test-runner.sh) drives the full scene — local HTTP page with a file input, Safari, action sheet, cross-process picker — and asserts recovered entries and the remote_content_recovery advisory kind. It restarts the per-UDID daemon first: a daemon spawned by an older binary survives across builds and would otherwise serve the pre-fix fetch path (this bit us during development; the runner has no daemon hygiene of its own).
• Full suite: 1201 tests green; E2E verified live on a booted simulator (10.6 s).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Rir4WwrPShf5DYzziJ8RyC

[onevcat]

Review follow-up: P1 (landscape sampling space) — confirmed and fixed in ba797d7

The finding is correct, and it's the #34 trap again: runFrontmostApplication derives screenBounds from element.axFrame() (UI space) while the discovery grid feeds translator.object(at:) (framebuffer space). Under rotation the sampled band is misaligned — points past the native width hit nothing, and a whole native band is never sampled. Portrait is the identity mapping, which is exactly why the original verification and the E2E passed.

Fix — no upstream patch needed: FBAccessibilityRemoteContentOptions.region is an existing override hook, so the retry now passes an explicit sampling region in native-portrait bounds (remoteContentSamplingRegion(native:), unit-tested). A full native grid covers every visible pixel regardless of orientation. Unknown screen size falls back to upstream's default region (correct in portrait, best-effort elsewhere). The coverage grid's UI-space bookkeeping is unaffected in practice: the retry only fires on empty shells, where the grid has nothing filled and isFilled never gates a point.

Landscape verification (live, landscape-left picker):

Frontmost accessibility tree is an empty shell; retrying with remote-content discovery
[i] The frontmost app exposed an empty accessibility tree; visible elements were recovered from other processes …
App: Safari浏览器  874x402  (landscape-left)
  @2  Heading  "最近项目"  (403,36 68x21)
  @3  Button  "取消"  (741,28 37x36)
  @4  Button  "更多"  #OverflowBarButtonItem  (794,28 38x36)
  @5  TextField  "搜索"  (78,79 718x44)

Recovered elements span the full landscape width (x up to ~794 on a 874-pt-wide UI) with correct UI-space frames, and the recovered elements in turn give calibration its discriminators — the header reports the orientation correctly.

On the landscape E2E suggestion: simulator rotation is not scriptable from the harness (the Simulator menu requires accessibility permission; simctl has no rotate), so the landscape scene remains a documented live check while the portrait E2E pins the end-to-end flow. The portrait E2E re-ran green after the change (11.3 s), full suite 1203 green.

[onevcat]

Review follow-up: P1 round 2 (coverage-grid cross-space shadowing) — mechanism confirmed, empirically bounded, fixed in 53434da

Code-level: confirmed. discoverRemoteElements marks each discovered element's UI-space axFrame into the grid and gates later framebuffer-space sample points on isFilled — the shadowing mechanism is exactly as described.

Empirical check before changing anything (per request): live picker scene, grid on vs off, element-set diff.

• Portrait control: identical (expected — identity mapping).
• Landscape, two runs: identical. No element was actually lost in this scene — the picker's controls are larger than the 50-pt grid step (multiple sample points hit each) and upstream's frame-key dedup absorbs the overlap, so a shadowed point's element had already been discovered from a neighboring point.

So the shadow is real but needs a small element (sub-grid-step) sitting inside a discovered element's numerically-overlapping band to bite. The decisive argument for fixing it anyway is cost-benefit, not probability: on the only path that runs discovery (an empty shell) the grid starts empty, so the isFilled gate can never save a probe — it can only mis-skip one. Disabling collectFrameCoverage is zero-cost correctness (dedup is UI-space-vs-UI-space and unaffected).

Options assembly is now factored into LegacyAccessibilityRequestBuilder with unit tests pinning the request contract: no coverage grid + explicit native-portrait sampling region.

Bonus finding while verifying on a rotated simulator: the E2E failed there for an unrelated reason — the action sheet's symmetric, sparse layout gives orientation calibration zero discriminators, and the ambiguous guess (landscape-right vs actual landscape-left) mapped the second tap one row off, into the camera option. That is the known calibration wobble showing up as a real mis-tap for the first time; the E2E now requires portrait up front with a clear message, and the wobble will be tracked as its own issue.

Full suite 1206 green; portrait E2E green post-change (13.5 s).