Touchscreen behavior for the interesttarget attribute

[mfreed7]

What is the issue with the HTML Standard?

Some references first:

• The interest invokers explainer's touchscreen section
• The OpenUI discussion thread about touchscreen behavior
• Some quick mockups of what the proposed behavior could look like. (Better ones are coming soon.)

This issue is a narrowed down version of the OpenUI discussion above, to hone in on the right final behavior. Essentially, the explainer lays this out:

For touchscreen users, the widely-adopted standard for “showing interest” in an element is the long-press gesture. For native apps, this is the user-expected shortcut to show context menus and access other actions without explicitly activating the element. (See this comment for more context on this point, from developers.)

On the web... most browsers already overload the long-press gesture to provide additional functionality [such as context menus]. Users often appreciate these additional capabilities, and do not want to lose access to them. Therefore, this proposal provides a way to keep both capabilities: the interesttarget behavior and the existing context menus and behaviors.

To show interest in an element via touchscreen, the user simply long-presses that element. This does not activate (e.g. click) the element, it merely triggers interest in the element. This long press immediately fires the "interest" event, but also shows any context menus that would have shown if the interesttarget attribute were not present. So both things happen together. In the case of a popover target element, the idea is that the hovercard popover shows up in addition to the context menu, so that the user can tap on either of those things.

That's the current direction, coming out of OpenUI discussions. This plan seems to have fairly broad consensus from the developers in OpenUI, who seem happy that it will meet their use case. In particular, that use case has been fairly definitively established as: Showing interest on a touchscreen must be done via long-press, and that long-press must simultaneously provide access to the popover hovercard and the UA-provided context menu. Exactly how that's done is up for debate. The explainer goes into detail about the proposed approach, which shifts around the context menu and exposes a "safe area" in which to render the hovercard.

This issue is to check that with the folks in WHATWG, to see if the proposed approach is workable, or if there are better suggestions for how to meet the use case.

[annevk]

I don't understand how the suggestion works because on iOS when you long press a link the entire screen is taken over.

[mfreed7]

I don't understand how the suggestion works because on iOS when you long press a link the entire screen is taken over.

Right - that's similar to what happens on Android as well. That behavior would need to change for elements that have interesttarget, as described in the explainer.

[annevk]

I don't really see how that would work given what we display today. You'd have to not display the preview or something, which I'm not sure is in the end user's best interest. It also seems rather risky to show both platform and web developer UI side-by-side.

[mfreed7]

I don't really see how that would work given what we display today. You'd have to not display the preview or something, which I'm not sure is in the end user's best interest.

I agree you'd have to not display the page preview, and in its place leave room for the page's (popover) version of that preview. This is why it seems actually better for the user. Take the Github example right on this page. Long-pressing a username on mobile Safari gives a rather large full preview of the page you get if you click that link. It includes login links, lots of extra stuff, and critically is not interactive at all. While I can see a button marked "Follow" in that preview, I can't tap it. If I do tap it, that navigates me to the new page (potentially losing state from the previous page), and then from there I can tap "Follow". But now I need to know to hit Back to get back to where I was before, and hope that bfcache or form preservation keeps whatever I had been doing there. Total interaction: Long press, tap once, tap twice, Back button.

Contrast that with the new experience, with interesttarget: the preview is smaller and more concise, exactly like the one you see on desktop. There is a "follow" button, which is directly interactive, and can be tapped. That closes the context menu (which I didn't need for this interaction) and likely closes the hovercard also. I'm now back to the original page, having never left it. And I accomplished my goal with a single long-press and a single tap. This feels much closer to the desktop experience to me, and much more user friendly. This is also what a majority of developers would prefer, at least from the OpenUI discussions.

It also seems rather risky to show both platform and web developer UI side-by-side.

I initially shared your concern, but we discussed this at some length in OpenUI, with the conclusion being there's no actual risk. Let me know whether that discussion alleviates your concerns also.

[smaug----]

It would be amazingly confusing if one user action triggers two unrelated UIs.

[hober]

I don't really see how that would work given what we display today. You'd have to not display the preview or something, which I'm not sure is in the end user's best interest.

I agree you'd have to not display the page preview, and in its place leave room for the page's (popover) version of that preview.

It would be jarring for a previously system-wide behavior to suddenly start doing something unpredictably different, but only in some web views on some web content. That's not a user experience regression we're interested in.

[martinthomson]

This seems to be entirely an encroachment into the rights of different browsers to present different UX. Safari have chosen to use a long press to present a preview (a useful feature). Whether that is subject to changes that are negotiated in a standards forum is a matter of whether Safari consents to that or not. It seems like the answer is "not".

Is there some other way that someone might signal interest in a particular element that isn't a long press? Could I touch the element and wiggle my finger around a little (which is scrolling, but not)?

[mfreed7]

It would be amazingly confusing if one user action triggers two unrelated UIs.

FYI this is what Safari currently does. For example, see the top and bottom half of this screenshot after long-pressing a URL:

With the interesttarget proposal, this would look nearly identical, except that the card at the top wouldn't include the (not useful) Login, hamburger menu, and other extraneous UI elements, which are only there because this is an anonymous preview of the URL, not a customized version coming from the page itself.

I agree you'd have to not display the page preview, and in its place leave room for the page's (popover) version of that preview.

It would be jarring for a previously system-wide behavior to suddenly start doing something unpredictably different, but only in some web views on some web content. That's not a user experience regression we're interested in.

I don’t think this would be jarring. From most users' point of view, I don't actually think they'd see this as materially different, even. They'd likely see it as an evolution of the current UX, wherein a) the hovercard can be customized by the developer, and b) the buttons they see on the screen can actually be tapped. If done well, I bet many users would think the browser made things better.

This seems to be entirely an encroachment into the rights of different browsers to present different UX. Safari have chosen to use a long press to present a preview (a useful feature). Whether that is subject to changes that are negotiated in a standards forum is a matter of whether Safari consents to that or not. It seems like the answer is "not".

So my original proposal (which was “option 1” in this comment) was to leave this behavior (exactly how to show interest via touchscreen) to the user agents to define. However, Webkit (rightly, in my view) pushed for us to define the exact mechanism for showing interest on touchscreen also. As a result of that push, at OpenUI we hashed through all of the options, and came up with something that feels like a great balance between what the UA can decide and what the site has the power to show.

With the current proposal (see e.g. here), the UA is still in charge of several things:

1. The position and size of the UA-provided context menu.
2. The available space and location for the developer/site-provided hovercard.
3. The decision whether to show interest at all. For example, on very small screens, the UA might decide to only show the context menu and not trigger interest, because there's no space.

With these abilities retained, this feels like the UA hasn’t given up many “rights”, and in the balance, users and developers have gained a powerful (and popular) UX pattern that works across input modalities. Feels like a win to me.

Is there some other way that someone might signal interest in a particular element that isn't a long press? Could I touch the element and wiggle my finger around a little (which is scrolling, but not)?

We explored this at some length in OpenUI, and the result seems to be a fairly strong conclusion: on mobile platforms, there is a long-standing precedent for how to “show interest” in something. You long press it. Users have learned this behavior, and would be confused or unsure about how to use any other “magic” gesture such as “touch and wiggle”. (Sounds funny, but we did explore several such options, e.g. long-press for a specific time, long-press-and-drag and a few others that didn’t get minuted.) Design system authors involved with the conversation were adamant that the only viable UX pattern that users will understand and use is long-press.

Mobile apps other than web browsers use long-press for this interaction, on both iOS and Android. It’s only because mobile browsers already have a context menu that we now have to figure out how to include both things.

[css-meeting-bot]

The CSS Working Group just discussed Touchscreen behavior for the `interesttarget` attribute.

The full IRC log of that discussion

<jarhar> masonf: on touch screens, the situation is worse than keyboard
<jarhar> masonf: if we were starting browsers today, it would be pretty obvious what to do. every native app has an interaction pattern to show interest, and thats long press
<jarhar> masonf: the issue that we have is that browsers already did that a long time ago, and it already does stuff, and users like it
<jarhar> masonf: long pressing on a link gives you a preview of the link, and you get a menu like share or copy the link or open in a new tab
<jarhar> masonf: users do use those context menu items and expect them to be there
<jarhar> masonf: when talking to developers about this, what should the mechanism be? ok touch screen is taken, lets make up a new gesture, like long press and slide or double tab
<jarhar> masonf: lots of creative things, any of which could be valid, but none of which are engrained in what users know how to do
<jarhar> masonf: if you invent a new gesture just for this, nobody will find it
<tantek> +1 masonf concerns, weird touch gestures are very undiscoverable
<jarhar> masonf: the very strong pushback from devs seems to be that long press needs to be the thing
<jarhar> masonf: now you have two things to do with this one gesture
<jarhar> masonf: we also considered where you long press, and it gives you the existing menu with another item that says i want to show interest in this thing
<jarhar> masonf: or some magical ux way that you have the existing thing and a new thing
<jarhar> masonf: its too much. long pressing youve already invested a lot of time. if you didnt get what you wanted immediately then youll give up
<astearns> q+
<jarhar> masonf: with one long press you should get both: the context menu and the popover
<jarhar> masonf: akin to keyboard, im presenting the current direction
<jarhar> masonf: when you long press on a link, which seems to be the use case that matter the most (buttons are easier), the context menu that you would have gotten still shows up, but in a position that leaves room on the screen for the hovercard that is provided by the site
<jarhar> masonf: in portrait mode on a phone, a context menu shows at the bottom of the screen and the hovercard appears at the top. the user can tap both things
<jarhar> masonf: to do that, the browser has to make room, put the context menu somewhere else
<jarhar> masonf: the browser still maintains control and can put it wherever they want
<jarhar> masonf: on very small screens they can choose to only show the context menu
<jarhar> masonf: it should expose the remaining area with css environment variables
<jarhar> masonf: thats the basic behavior, theres a lot of details
<jarhar> masonf: in most browsers right now, those context menus come with a blurring of the site. that cant be done because then youd blur the hovercard
<jarhar> masonf: that is the concept
<jarhar> masonf: we are prototyping this in chromium. there are a lot of ux concerns. our ux teams are looking at this and deciding whether this is good for users
<jarhar> masonf: is this confusing or is it natural
<jarhar> masonf: there are a lot of open questions about how to do this. just moving the context menu - what if youre on a tablet or landscape orientation
<jarhar> masonf: id love to hear feedback or better ideas
<jarhar> astearns: this idea of adding something to the existing long press ui would have security concerns. the existing long press ui is provided by the browser, so users trust it more than something on a webpage. if any webpage can mimic this part of the browser ui, it can pretend that it is the browser
<smaug> q+
<jarhar> lwarlow: anything thats rendered within the frame - they already can spoof it. likewise, a right click context menu it trusted but i can hijack that and render a popover that looks identical
<astearns> ack astearns
<jarhar> lwarlow: i would say you already can do all of that stuff, unless the ui escapes the frame, but i dont think it can in any of the browsers
<jarhar> lwarlow: if it blurred the browser ui as well theres something there
<jarhar> smaug: it darkens the background, so there is a distinction with a real context menu
<masonf> q+
<keithamus> q+
<jarhar> smaug: also if you show the browser provided context menu and something on top of that, then its confusing for users to know which portion comes from the browser
<jarhar> smaug: i was testing chrome to see how long the context menu
<dandclark> q+
<jarhar> smaug: if you have a link then it already covers the whole screen, so theres no space for popovers
<lwarlow> q+
<jarhar> smaug: if you have an image which is also a link, then the context menu already covers the whole screen
<smaug> q-
<jarhar> smaug: theres too many list items, they would need to be redesigned
<jarhar> astearns: have you considered allowing authors to give a long press behavior to elements only if there is not a current long press behavior for that element from the browser?
<jarhar> masonf: the primary use case is on links. on all browsers i know, links are the big thing that has a context menu. buttons are in that category
<jarhar> masonf: links are the key use case where they do intersect today
<jarhar> astearns: there possibly could be a path where we provide a web standard way of defining a long press behavior. we can show that its so useful that browsers will decide to allow the author version of a long press for links with an extra button to get to our own instead
<astearns> ack masonf
<jarhar> masonf: we talked about that too. that was in the category of the two clicks, thats just the other direction of it. there was pushback on both of those because users really like the context menu, so making either of them harder felt like a no go
<jarhar> masonf: i actually shared your security concern. you can do this today, and i talked to some google security folks, and they all agreed that you can do this today
<jarhar> masonf: there is a blur that is provided by the browser, but the people in this room that this little blurring is a security signal and nobody else does
<jarhar> masonf: i was convinced by that
<jarhar> masonf: olli mentioned that some context menus are very long. in the chromium prototype, the regular one for links with images it becomes scrollable, so it will be important to make sure these are sorted
<astearns> ack keithamus
<jarhar> keithamus: this is github.com in firefox on ios. if you long press a link you get a preview. this is a native ios behavior. the menu falls off the bottom of the page, and i can drag to show more of the menu or more of the preview
<jarhar> keithamus: this is a common pattern in ios
<jarhar> keithamus: the problem is that the resulting preview is just not very helpful. if i click an avatar, i get a bit of the description, but the button isnt interactive. the button is just going to take me to the page
<jarhar> keithamus: were doing two server requests because we dont cache those. it is multi factorial that we use hovercards: its better ux and better performance, its a cost saving measure
<jarhar> keithamus: most of the data we got about the user we already got in the first request
<jarhar> keithamus: native apps, this is the wikipedia native app. i can long press and get this summary. this is the ideal thing for us
<jarhar> astearns: in the native app, is anything in that preview interactive?
<jarhar> keithamus: not that ive seen
<jarhar> keithamus: social media websites will have a similar feature
<jarhar> keithamus: since they have follow features, maybe theirs are interactive
<astearns> ack dandclark
<keithamus> q+
<jarhar> dandclark: several variants of this have been discussed. ill put in a vote for the two stage model, where the long press gets you the content and then you can press something else to get the context menu from the browser. it solves the space problem because its just one button. it has an additional barrier to get to the native thing, but its not
<jarhar> another long press and its not a new gesture
<jarhar> masonf: i think the best way to spec this - speccing this is going to be an interesting question - but the placement of the context menu is going to be up to the browser
<jarhar> masonf: i think both the ios way and the extra button makes sense
<jarhar> masonf: how and where you render the context menu should still be up to the browser
<astearns> ack lwarlow
<jarhar> lwarlow: the wikipedia native app on android is different. a tap on a link is what shows their preview ui
<jarhar> lwarlow: it has open in new tab and stuff
<jarhar> lwarlow: clicking the whole preview will do the navigation
<jarhar> lwarlow: i do wonder if this should just be that long press does the interesttarget thing
<jarhar> lwarlow: and the browser should just give up their context menu
<jarhar> lwarlow: its the sort of thing that peoplare going to try disabling the browsers native thing anyway and it will have worse ux
<jarhar> lwarlow: like they might just use a button instead of a link because that disables the browser ui
<jarhar> lwarlow: and use user-select:none
<jarhar> lwarlow: all the friction we add to this is probably just another layer of im not going to use this builtin browser feature. which maybe is fine, but worth keeping that in mind
<jarhar> lwarlow: the other alternative is the context menu you get in browsers, can we just let them render inside it? like replace the preview bit at the top? for links you usually get useless stuff at the top
<jarhar> lwarlow: thats how it works in native apps
<jarhar> lwarlow: the wikipedia example is like that
<jarhar> lwarlow: im hesitant on having two bits of ui just on a devs wont use this perspective
<jarhar> keithamus: we can disable the contextmenu, but thats not desirable because people do use the open in new tab thing
<jarhar> keithamus: we want to retain open in new tab, we can replicate it but its not what people want
<jarhar> keithamus: id like to keep exploring two context menus
<jarhar> keithamus: showing context options is important to me and github
<jarhar> masonf: long press does generate a contextmenu event you can preventdefault, but it sounds like in this room there are multiple opinions, so we could leave it up to the browser to choose
<astearns> ack keithamus
<astearns> ack fantasai
<jarhar> fantasai: i thought this was an interesting conversation, but apple does have concerns that are expressed in the issue and i cant explain better than that. just wanted to say there are concerns
<jarhar> astearns: not just apple, i also see martin thompson also expressing concerns
<jarhar> masonf: the concerns are all this feels funny, id love to have more discussion
<tantek> s/thompson/thomson
<jarhar> fantasai: it would be useful to bring this up where tess and anne can participate. unfortunately they had conflicts today
<jarhar> masonf: can we just keep this particular part on the agenda and that could be the venue?
<jarhar> fantasai: tess will be on vacation next time, anne will be available
<jarhar> fantasai: and we have a f2f
<fantasai> s/will/might/
<jarhar> astearns: if there is a meeting in 2 weeks, there probably wont be any css folks in it
<jarhar> masonf: maybe we should cancel the one in two weeks, and it sounded like in 4 weeks could be a good time?
<jarhar> fantasai: you might want to check in if they can make that one specifically
<jarhar> dbaron: if you also want martin thompson thats going to be hard to include australia
<jarhar> masonf: or a breakout on this topic
<jarhar> fantasai: yes