Tablet and Phone Interface Improvements

[hello-vinitha]

Description

This PR makes a series of changes to improve the tablet and phone view for the interface made in the temp/deployment branch. The tablet view should appear as a scaled down view from the desktop version and the mobile interface has a different layout to make the best use of space.

Currently there are some known issues that need be resolved:

• In the tablet customization mode some components disappear when another component is added or a component is moved.
• Button pad buttons sometimes overflow out of the panel depending on the size of the screen.

Testing procedure

• Test that components can be successfully added, removed and rearranged (desktop, tablet)
• Verify mobile responsiveness on different screen sizes (tablet & mobile)
• Verify the correct buttons are being highlighted blue/green when the joint is moving
• Verify the correct buttons are highlighted red when the respective joint is at it's limit
• Verify click-to-pregrasp works on (desktop & tablet)
• Verify that cameras and buttons are sized appropriately for the screen (i.e. buttons/camera are not overflowing or too small, etc.)
• Verify that all action modes work across desktop and mobile

Before opening a pull request

From the top-level of this repository, run:

• pre-commit run --all-files

To merge

• Squash & Merge

[mintyfreshv]

Referring to the button pad, the colors for notifying the person that they’re at their joint limit is variable. Is there a reason why it’s red and green?

[mintyfreshv]

In this photo, you can see that lifting the arm button is highlighted green, but I am not using that button at the moment because at the time of this photo I was using, the lowering the arm button

[hello-vinitha]

@kaikwan @hello-william @mintyfreshv click to pregrasp has been fixed on mobile and desktop. I tested it on 3030 and verified it works as expected when using a chrome browser on a desktop and an IPad

[hello-vinitha]

@hello-william The panel resizing is not working as expected.
https://github.com/user-attachments/assets/9fcbecbf-981b-49e6-a292-b581052a1d02

[hello-william]

let's goooooooooo ☄️

[hello-vinitha]

Referring to the button pad, the colors for notifying the person that they’re at their joint limit is variable. Is there a reason why it’s red and green?

@mintyfreshv red means the joint limit was reach, the button highlights blue/green when that joint is moving.

[hello-vinitha]

All tests passed on 3030. @mintyfreshv please test on your robot.

• Test that components can be successfully added, removed and rearranged (desktop, tablet)
• Verify mobile responsiveness on different screen sizes (tablet & mobile)
• Verify the correct buttons are being highlighted blue/green when the joint is moving
• Verify the correct buttons are highlighted red when the respective joint is at it's limit
• Verify click-to-pregrasp works on (desktop & tablet)
• Verify that cameras and buttons are sized appropriately for the screen (i.e. buttons/camera are not overflowing or too small, etc.)

[mintyfreshv]

ScreenRecording_05-29-2025.15-23-13_1.mov

Cannot delete previously, saved “movement recordings”

Devices where this issue occurs: iPad and desktop version

[mintyfreshv]

ScreenRecording_05-29-2025.15-32-42_1.mov

Part of the customization menu is showing when I slide the interface to the left. Please see video for demonstration.

This is on the iPad

[mintyfreshv]

Inquiry about joystick control input on the mobile tablet version of the interface

https://github.com/user-attachments/assets/cef28f8e-9923-4249-a8fa-499ac12d78e5
I am forgetting, and likely need a recall regarding the functionality of the joystick option on the tablet. Is it meant to be limited in use when it’s on the tablet? Something tells me that this is the case, but I am not sure again.

This is on the iPad

[mintyfreshv]

Issue: wide angle head camera bottom arrow are not in the camera feed

This is on the iPad

[mintyfreshv]

Video 1
https://github.com/user-attachments/assets/966c541b-c0e2-4793-b0ad-55faff82a34f

Video 1 follow up
https://github.com/user-attachments/assets/a87e552d-ae3a-43c9-bf3b-6e68a58e3564

Issue: recording new movement recordings do not work when initiated

Devices where issues occur: iPad and desktop version

Video description: I am limited to about six seconds of video footage. I can send over GitHub. So I split up the videos and edit it down quite a bit. Essentially, I’m giving you an example of recording the robot’s arm fully extended. I’ve labeled it “arm out,” and you will see a video too, that when I click “arm out,” Stretch doesn’t do anything.

However, I had already shave, pre-existing movement recordings before testing today, and those still work. The new ones I created today don’t work though. Also, it may not seem clear in the videos, but I did save the recording, as you see in the drop-down the label “arm out” is present.

[mintyfreshv]

Issue: move to pre-grasp

Description: move to pre-grasp worked for me once, but it was about three or 4 inches away from my target. After trying it again, the feature kept failing. Perhaps, this is what Will has been also facing?

This is on the iPad

[mintyfreshv]

Device: phone interface

Issue: the play button and the controls under recordings. Please see screen capture.

[hello-vinitha]

To Do List based on @mintyfreshv's feedback:

• Movement recorder bugs with recording and deletion (@hello-vinitha)
• Do not allow page scrolling in x & y direction on tablet - this should solve both the joystick and customization menu issue (@hello-william)
• Fix placement of camera pan/tilt buttons when buttons underneath camera view are disabled -- see comment (@hello-william)
• Follow up with @mintyfreshv to investigate if click-to-pregrasp issue is due to not being able to solve IK or input issue on tablet vs. desktop. If due to former, open issue to track separately (@hello-vinitha)
• Fix z-index of button controls on mobile -- see comment (@hello-william)