ios-device: back/escape navigation — agents get stuck on detail screens

[onevcat]

Context

Verified on an iPhone 15 Pro Max (iOS 26.6) with the SwiftUI playground: after ios-device tap navigates into a detail screen, the navigation bar's back button does not appear in the accessibility hierarchy at all (the detail tree had 5 elements: header + static texts). With no coordinate taps on this channel, an agent that navigates forward has no way back — today's only escape hatch is relaunching the app from outside sim-use (devicectl device process launch --terminate-existing).

Proposal

Investigate whether the audit daemon exposes an Escape/back action (the VoiceOver two-finger-scrub equivalent) as another AXAction-<code>, and expose it as sim-use ios-device back (or similar). The existing action codes (2010 Activate, 2006/2007 scroll) were discovered by reading the Actions section of focus events — the same technique should reveal whether an escape code exists.

Notes

• If the daemon offers no such action, fall back to documenting the relaunch escape hatch in skills/sim-use/SKILL.md so agents don't spin.
• Also worth checking whether the back button's absence is specific to SwiftUI NavigationStack or general.

[onevcat]

Root cause found + viable mechanism identified

Verified with the same development-signed SwiftUI playground on both backends:

The Back button exists and is AX-exposed — the device hierarchy channel just doesn't serve it.

• Simulator (sim-use ui): the detail screen shows Button "sim-use Playground" #BackButton (20,62 44x44).
• Device (ios-device ui, raw walk dump): <SwiftUI.UIKitNavigationBar> appears as a node but expanding it via _AXHierarchyElementsAttribute returns no children. The nav bar title (Header "Tap Test") shows up flattened as a root child; the back button never appears in any of the 20 raw nodes.
• deviceInspectorShowIgnoredElements: true makes no difference — it is not ignore-filtering.

The focus channel reaches it. Sending deviceInspectorMoveWithOptions: (argument encoding per pymobiledevice3: passthrough dict {allowNonAX: 0, direction: N, includeContainers: 1}, directions Previous=3 / Next=4 / First=5 / Last=6) with First on the detail screen makes the daemon broadcast hostInspectorCurrentElementChanged: carrying:

• CaptionTextValue_v1 = "sim-use Playground, Button" — the back button,
• AccessibilityIdentifier_v1 = BackButton,
• a live PlatformElementValue_v1 element token, and
• InspectorSectionsValue_v1 (per-element attribute/action descriptors).

So a back verb is implementable as: focus First → consume the focus-change event → deviceElement:performAction:withValue: Activate on the carried token, inside one session. Implementation note: focus-change events are daemon-initiated DTX messages, which DTXConnection currently drops by design — the connection needs an opt-in event listener.

Two further observations from the same session:

• Focus events fire even on system UI (captured a lock-screen caption) where the hierarchy channel reports "unsupported" — the focus channel is strictly more capable there.
• deviceFetchElementAtNormalizedDeviceCoordinate: (in deviceCapabilities, apiVersion 26) accepts an NSValue(CGPoint) argument — a dictionary argument fails with -[__NSDictionaryI CGPointValue]: unrecognized selector, which leaks the expected type — but returned an empty reply at several coordinates in our session state. Possibly requires inspector visuals/event monitoring to be enabled; worth a follow-up probe as an alternative (and as a general hit-test primitive).

End-to-end Activate-on-back verification is pending (device auto-locked mid-session); will confirm and attach the result.

[onevcat]

End-to-end confirmed on the iPhone 15 Pro Max (iOS 26.6), development-signed playground, from the Tap Test detail screen:

1. deviceInspectorMoveWithOptions: direction First → hostInspectorCurrentElementChanged: event captioned "sim-use Playground, Button" (the back button) with a live element token.
2. deviceElement:performAction:withValue: Activate on that token, same session.
3. A hierarchy read afterwards shows the full main menu — navigation went back.

The mechanism for a back verb is proven. Remaining implementation work: an opt-in listener for daemon-initiated DTX events in DTXConnection (they are currently dropped by design), then focus First → activate captured token inside one session.

[onevcat]

Fixed by #114 — the premise turned out wrong in a good way: the back button was in the hierarchy all along, deterministically dropped by a tree-walk bug (deviceFetchSpecialElement: 0 returns the back button itself as the root on pushed screens, so seeding the visited set with the raw root token discarded it). After the (token, summary, role) dedup fix the back button renders as an ordinary row (Button "<previous screen title>" #BackButton) and going back is a plain ios-device tap '#BackButton' — no special verb needed, verified live.

The remaining side-quest here — whether the daemon exposes an Escape/scrub AXAction-<code> (useful for modal sheets without a back button) — folds into #107's custom-action enumeration.