Repository navigation
fix: surface the navigation-bar back button (and #id) in ios-device ui - #114
Merged
Merged
Conversation
The physical-device `ui` walk dropped the navigation-bar back button: on a pushed screen `deviceFetchSpecialElement: 0` returns the back button as the root, so its token aliases a real element, and seeding the walk's visited set with the raw root token discarded it as "already seen". Dedup on a composite (token, summary, role) key so a token the daemon reuses across distinct elements no longer collapses them, while genuine repeats from the deep child walk still fold. Also surface each element's accessibility identifier as `#id` in the outline — a stable reference when a label is dynamic (the back button is labelled with the previous screen's title but keeps `#BackButton`). The back button is now an ordinary outline row, tappable with the existing `tap` by the label shown in the same read; no back verb, no focus-channel exception. Verified live on iPhone 15 Pro Max (iOS 26.6): the back button appears in `ui` on a pushed screen and `tap --label` navigates back; the menu screen is unchanged. Offline coverage reproduces the root-token alias drop and the genuine-repeat collapse through a fake transport. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
Now that the outline renders each element's #id, let `ios-device tap` target it: a positional `#<id>` or `--id`, the same identifier forms the simulator tap accepts. Prefer it when a label is dynamic — the navigation-bar back button keeps `#BackButton` though its label is the previous screen's title. `--label` / `--label-contains` / `--element-type` are unchanged. A positional `@N` is rejected with a pointer to `#id`, since expiring handles cannot back a cross-invocation alias (the same reason, like the absent geometry, that coordinate taps are unavailable). Verified live (iPhone 15 Pro Max, iOS 26.6): `tap '#BackButton'` and `tap --id BackButton` both navigate back from a detail screen. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
Review follow-up on the #id / --id tap selector: - Do not apply the button-preference tie-break to identifier matches. It exists for label selectors, where a button and a nested static text legitimately share a label. An accessibility identifier is a unique literal handle (as on the simulator / Android surfaces), so a duplicate id is a real ambiguity to report — not something to silently resolve to the button. --element-type can still narrow a duplicate to a single match. - Compare identifiers exactly and case-sensitively (trimming only incidental whitespace on the selector), instead of a locale-aware case-insensitive compare, so `BackButton` and `backButton` are distinct identities and never merge. Adds regression tests for both: a duplicate id with one Button match is ambiguous (and narrows via --element-type), and identifier matching is exact/case-sensitive. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
…roid policy Review follow-up, closing the last drift between the physical-device tap resolver and the rest of sim-use: - Identifier match trims both the selector and the daemon-reported id (the daemon can pad an identifier; `ui` now renders it trimmed too), so a copied `#id` resolves. Comparison stays exact and case-sensitive, matching the simulator's `normalizedUniqueId == trimmed query`. - Label / label-contains now route through SimUseCore's SelectorTextMatcher — the one case-sensitive, exact-first, whitespace-collapse policy already shared by the simulator and Android resolvers and both outline renderers — instead of a bespoke locale-aware case-insensitive compare. Physical-device label matching no longer drifts from the other surfaces. - --element-type narrows the candidate pool before matching, as the simulator resolver does. The outline renders identifiers trimmed so the shown `#id` is exactly what `tap #<id>` matches. Tests: whitespace-padded id still matches, outline trims the id for display, and label matching is case-sensitive. Live-verified on iPhone 15 Pro Max / iOS 26.6: exact-case label and `#BackButton` navigate; a wrong-case label now misses, as on the simulator. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
Follow-up to routing physical-device label matching through the shared SelectorTextMatcher: the `--label-contains` help still said "Case-insensitive". The simulator and Android surfaces have always matched case-sensitively, so aligning ios-device to them made it case-sensitive too; the help now says so. No CHANGELOG "Changed" entry: the ios-device surface is unreleased, so there is no shipped case-insensitive behaviour to have changed. The shared, case-sensitive matching is noted where the feature is described under Added instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The experimental
sim-use ios-device uisilently dropped the navigation-bar back button, so an agent that navigated into a detail screen had no way back. This turned out to be a walk bug, not a device limitation — the back button is in the hierarchy data all along. This PR fixes the walk, surfaces accessibility identifiers, and letstaptarget them, so going back is just an ordinarytapwith no special verb.Verified live on an iPhone 15 Pro Max (iOS 26.6) with a development-signed Playground.
Root cause
On a pushed screen
deviceFetchSpecialElement: 0returns the back button itself as the root, so the root's element token aliases a real, distinct element.DeviceTreeFetcherseeded itsvisitedset with the raw root token and then discarded the back button as "already seen" — deterministically (children(of: root)returned it every time;fetchTreedropped it every time).Changes
(token, summary, role)key instead of the bare token, so a token the daemon reuses across distinct elements no longer collapses them, while the genuine repeats the deep child walk emits still fold. The back button now appears inuiand is tappable with the ordinarytap.ios-device uirenders each element's accessibility identifier as#id. It is stable across reads, unlike a dynamic label (the back button is labelled with the previous screen's title but keeps#BackButton).ios-device tapaccepts a positional#<id>or--id, mirroring the simulator tap, alongside--label/--label-contains/--element-type. A positional@Nis rejected with a pointer to#id— expiring handles cannot back a cross-invocation alias, the same reason (like the absent geometry) coordinate taps are unavailable here. Preserving@Non this surface via fingerprint re-resolution is tracked in ios-device: preserve the @N ergonomic via fingerprint re-resolution (and a feedback path to harden simulator @N) #113.Verification
make build— passed, 0 warnings.make test— 1331 tests passed, 0 warnings. New offline coverage reproduces the root-token-alias drop, the genuine-repeat collapse,#idrendering,#idresolution, and the@Nrejection through a fake transport.uinow listsButton "sim-use Playground" #BackButton; the menu screen is unchanged (no duplicates, no phantom rows).tap --label "sim-use Playground",tap '#BackButton', andtap --id BackButtonall navigate back to the menu.Scope
Physical-device (
sim-use ios-device) only; the simulator and Android surfaces are untouched. README, the bundled skill, and CHANGELOG are updated.