fix: recover cross-process UI when the frontmost tree is an empty shell

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- iOS `describe-ui` no longer goes blind when a system document picker (or another remote-process presentation) is on screen while the host app stays frontmost (#64). The frontmost tree comes back as an empty shell in that state — a bare or full-screen-framed `AXApplication` with no content — and previously produced zero entries even though `--point` could hit the visible picker. The fetch now detects the shell and retries once with upstream's cross-process content discovery (coverage-grid hit-testing), surfacing the picker's elements with a `remote_content_recovery` advisory that notes the recovered hierarchy is flat. The hot path is untouched: healthy and sparse-but-valid trees never pay for the retry, and on full-coverage native screens the discovery grid probes nothing.
 - iOS directional gesture presets (`gesture scroll-up/down/left/right`, `swipe-from-*-edge`) are now orientation-aware (#66). They previously emitted device-native portrait axes, so on a rotated device `scroll-up` pointed 90° (landscape) or 180° (upside-down) away from its name — no scroll, or an unintended row navigation. Preset math now runs in the current visual space (auto-calibrated per command, the same #34 mapping `tap` selectors use) and the endpoints are transformed into HID coordinates; this covers both the standalone command and `ios batch` gesture steps (which share the batch-wide calibration). When the orientation cannot be determined the gesture falls back to the legacy portrait dispatch and says so via the `advisory` envelope key. Explicit `swipe`/`touch` coordinates remain device-native portrait by contract; pinch/rotate presets are likewise unchanged.
 
 ### Changed

README.md

@@ -74,7 +74,7 @@ interpreted in the device-native portrait space.
 ## Why sim-use
 
 - **Token-efficient.** The outline representation is ~16x more compact than a raw JSON accessibility tree. An LLM can read and reason about an entire screen in a few hundred tokens.
-- **Nothing hidden.** sim-use walks the full accessibility tree including WebViews, system overlays, and embedded content — no elements are silently skipped.
+- **Nothing hidden.** sim-use walks the full accessibility tree including WebViews, system overlays, and embedded content — no elements are silently skipped. When the frontmost app exposes an empty tree because a remote process owns the visible UI (a system document picker, for example), `ui` automatically retries with cross-process discovery and flags the recovered, flat hierarchy via the `advisory` envelope key.
 - **AI-native.** Designed from day one for agent loops, not human testers. Alias-cached taps (`@N`), structured `--json` envelopes with actionable `hint` fields on errors, and a bundled agent skill (`sim-use init --client claude`) that teaches your AI client the full command surface.
 - **Fast.** A per-device background daemon amortises init cost across calls. After the first command, each observe-act round trip completes in ~300 ms.
 - **Cross-platform.** One command surface drives both iOS Simulator and Android emulator/device. Same verbs, same flags, same `--json` shape — write one agent loop that works on both.

Sources/SimUseCore/CommandAdvisory.swift

@@ -9,6 +9,7 @@ public struct CommandAdvisory: Codable, Equatable, Sendable {
     public enum Kind: String, Codable, Equatable, Sendable {
         case fullScreenTapTarget = "full_screen_tap_target"
         case orientationCalibrationFallback = "orientation_calibration_fallback"
+        case remoteContentRecovery = "remote_content_recovery"
     }
 
     public let kind: Kind

Sources/iOSSimBackend/A11y/AccessibilityFetcher.swift

@@ -26,10 +26,19 @@ public struct AccessibilityFetcher {
     /// Tree (or point-query) payload plus the orientation calibration the
     /// fetch ran under. `calibration` is nil only for surfaces that never
     /// calibrated (legacy shims); a degraded calibration is still present
-    /// with its advisory attached.
+    /// with its advisory attached. `advisory` is non-nil when the tree came
+    /// back as an empty shell and the visible elements were recovered from
+    /// other processes via the remote-content retry (issue #64).
     public struct FetchResult {
         public let data: Data
         public let calibration: OrientationCalibration?
+        public let advisory: CommandAdvisory?
+
+        public init(data: Data, calibration: OrientationCalibration?, advisory: CommandAdvisory? = nil) {
+            self.data = data
+            self.calibration = calibration
+            self.advisory = advisory
+        }
     }
 
     public static func fetchAccessibilityInfoJSONData(
@@ -100,9 +109,34 @@ public struct AccessibilityFetcher {
             )
         }
 
-        let info: AnyObject = try await target.legacyAccessibilityElements(nestedFormat: true)
+        var info: AnyObject = try await target.legacyAccessibilityElements(nestedFormat: true)
         perf.stage("tree fetch XPC")
 
+        // Empty-shell retry (issue #64): a remote-process presentation
+        // (system document picker) leaves the frontmost tree a bare,
+        // frameless AXApplication. Refetch once with upstream's
+        // remote-content discovery so the visible cross-process elements
+        // materialize. The plain first fetch keeps the hot path untouched —
+        // healthy and sparse-but-valid trees never pay for the grid probes.
+        var remoteAdvisory: CommandAdvisory? = nil
+        if isEmptyShellTree(info) {
+            logger.info().log("Frontmost accessibility tree is an empty shell; retrying with remote-content discovery")
+            if let retried = try? await target.legacyAccessibilityElements(
+                    nestedFormat: true,
+                    includeRemoteContent: true,
+                    remoteSamplingRegion: remoteContentSamplingRegion(native: native)),
+               !isEmptyShellTree(retried) {
+                info = retried
+                remoteAdvisory = CommandAdvisory(
+                    kind: .remoteContentRecovery,
+                    message: "The frontmost app exposed an empty accessibility tree; visible elements were recovered from other processes (e.g. a system picker). The hierarchy is flat and may not cover every element — prefer visible labels over structure."
+                )
+            } else {
+                logger.info().log("Remote-content retry did not surface any elements; keeping the original tree")
+            }
+            perf.stage("remote-content retry")
+        }
+
         let calibration = await calibrate(info: info, native: native, probe: probe, logger: logger)
         perf.stage("calibrate")
 
@@ -127,7 +161,61 @@ public struct AccessibilityFetcher {
         let data = try serializeAccessibilityInfo(recovered)
         perf.stage("serialize")
         perf.finish()
-        return FetchResult(data: data, calibration: calibration)
+        return FetchResult(data: data, calibration: calibration, advisory: remoteAdvisory)
+    }
+
+    /// Whether a fetched tree payload is an "empty shell" warranting the
+    /// remote-content retry (issue #64): no non-application node anywhere
+    /// in it carries a positive-area frame. The frontmost app reports
+    /// exactly this while a remote process (system document picker) owns
+    /// the visible UI — observed live both as a bare `{pid, role}` root
+    /// (0.10.0-era reports) and as a full-screen-framed AXApplication
+    /// with zero children (current runtimes). The application container's
+    /// own frame is just the screen rectangle and proves nothing about
+    /// visible content, so it never vetoes the retry. Unrecognized shapes
+    /// and oversized trees are NOT shells: failing closed keeps the retry
+    /// (and its full-screen probe cost) off every path this predicate
+    /// doesn't positively understand.
+    /// The grid-sampling region for the remote-content retry: the native
+    /// portrait framebuffer bounds. Upstream's default region is the root
+    /// element's UI-space frame, but the grid points feed the point
+    /// hit-test, which consumes FRAMEBUFFER points (issue #34) — under
+    /// rotation a UI-space region samples the wrong band (points past the
+    /// native width hit nothing; a whole native band is never sampled).
+    /// A full native-portrait grid covers every visible pixel regardless
+    /// of orientation. Nil (unknown screen size) falls back to upstream's
+    /// default region: correct in portrait, best-effort elsewhere.
+    nonisolated static func remoteContentSamplingRegion(native: NativePortraitSize?) -> CGRect? {
+        native.map { CGRect(x: 0, y: 0, width: $0.width, height: $0.height) }
+    }
+
+    nonisolated static func isEmptyShellTree(_ info: AnyObject) -> Bool {
+        let roots: [[String: Any]]
+        if let array = info as? [[String: Any]] {
+            roots = array
+        } else if let dict = info as? [String: Any] {
+            roots = [dict]
+        } else {
+            return false
+        }
+        var stack = roots
+        var visited = 0
+        while let node = stack.popLast() {
+            visited += 1
+            if visited > 500 {
+                // A tree this large is definitionally not a shell.
+                return false
+            }
+            let isApplication = (node["role"] as? String) == "AXApplication"
+                || (node["type"] as? String) == "Application"
+            if !isApplication, OrientationCalibrator.frameRect(of: node) != nil {
+                return false
+            }
+            if let children = node["children"] as? [[String: Any]] {
+                stack.append(contentsOf: children)
+            }
+        }
+        return true
     }
 
     public static func fetchAccessibilityElements(

Sources/iOSSimBackend/A11y/LegacyAccessibilityBridge.swift

@@ -17,10 +17,32 @@ extension FBSimulator {
     /// The frontmost application's accessibility tree, in the same shape the
     /// pre-Swiftification `accessibilityElements(withNestedFormat:)` returned:
     /// an array of dictionaries (a single root for the nested format).
-    func legacyAccessibilityElements(nestedFormat: Bool) async throws -> AnyObject {
+    ///
+    /// `includeRemoteContent` opts into upstream's coverage-grid discovery
+    /// of elements owned by other processes (grid hit-testing over screen
+    /// regions the frontmost tree does not cover). Off by default: on a
+    /// healthy full-coverage tree it probes nothing, but on sparse (yet
+    /// perfectly valid) trees it burns a grid of hit-test XPCs — callers
+    /// enable it only when the plain fetch came back as an empty shell
+    /// (issue #64).
+    ///
+    /// `remoteSamplingRegion` overrides upstream's default sampling region
+    /// (the root's UI-space frame). Pass the native-portrait bounds: the
+    /// grid points feed the framebuffer-space point hit-test (issue #34),
+    /// so a UI-space region samples the wrong band under rotation.
+    func legacyAccessibilityElements(
+        nestedFormat: Bool,
+        includeRemoteContent: Bool = false,
+        remoteSamplingRegion: CGRect? = nil
+    ) async throws -> AnyObject {
         let element = try await accessibilityElementForFrontmostApplication()
         defer { element.close() }
-        let response = try element.serialize(with: FBAccessibilityRequestOptions(nestedFormat: nestedFormat))
+        let options = LegacyAccessibilityRequestBuilder.options(
+            nestedFormat: nestedFormat,
+            includeRemoteContent: includeRemoteContent,
+            remoteSamplingRegion: remoteSamplingRegion
+        )
+        let response = try element.serialize(with: options)
         return response.elements as AnyObject
     }
 
@@ -34,3 +56,36 @@ extension FBSimulator {
         return response.elements as AnyObject
     }
 }
+
+/// Options assembly for the legacy tree fetch, factored out so the
+/// remote-retry request contract stays unit-testable without a
+/// simulator.
+enum LegacyAccessibilityRequestBuilder {
+    static func options(
+        nestedFormat: Bool,
+        includeRemoteContent: Bool,
+        remoteSamplingRegion: CGRect?
+    ) -> FBAccessibilityRequestOptions {
+        var options = FBAccessibilityRequestOptions(nestedFormat: nestedFormat)
+        if includeRemoteContent {
+            // Deliberately WITHOUT collectFrameCoverage: the coverage grid
+            // is created and filled with UI-space frames while its
+            // isFilled gate consumes the discovery grid's
+            // framebuffer-space sample points — under rotation a
+            // discovered element's UI frame would shadow a numerically
+            // overlapping but visually unrelated framebuffer band,
+            // skipping later sample points. On the only path that runs
+            // discovery (an empty shell) the gate's upside is zero anyway:
+            // the grid starts empty, so it can never save a probe — it
+            // can only mis-skip one. Duplicate hits are already collapsed
+            // by upstream's frame-key dedup, which compares UI-space
+            // frames against UI-space frames.
+            var remote = FBAccessibilityRemoteContentOptions()
+            if let remoteSamplingRegion {
+                remote.region = remoteSamplingRegion
+            }
+            options.remoteContentOptions = remote
+        }
+        return options
+    }
+}

Sources/iOSSimBackend/A11y/OrientationCalibrator.swift

@@ -328,7 +328,7 @@ public enum OrientationCalibrator {
         return contained.count == 1 ? contained.first : nil
     }
 
-    static func frameRect(of node: [String: Any]) -> CGRect? {
+    nonisolated static func frameRect(of node: [String: Any]) -> CGRect? {
         guard let f = node["frame"] as? [String: Any] else { return nil }
         func number(_ v: Any?) -> Double? {
             if let d = v as? Double { return d }

Sources/iOSSimBackend/Verbs/IOSSimDescribeUICommand.swift

@@ -286,10 +286,13 @@ public struct IOSSimDescribeUICommand: SimUseExecutableCommand {
             appLabel: outline.appLabel,
             appPackage: appPackage,
             orientation: orientation?.rawValue,
-            // Degraded calibration (guessed orientation) must reach the
-            // caller: the outline may have lost regions to mis-mapped
-            // recovery probes and `orientation` is a guess, not a fact.
-            commandAdvisory: fetchResult.calibration?.advisory
+            // Degraded calibration (guessed orientation) and remote-content
+            // recovery (issue #64) must both reach the caller: the outline
+            // may be a guess-mapped or cross-process-recovered view rather
+            // than a plain frontmost tree.
+            commandAdvisory: CommandAdvisory.merged(
+                [fetchResult.advisory, fetchResult.calibration?.advisory].compactMap { $0 }
+            )
         )
     }