Strange behavior of a dropdown menu in FilterAssetListTask can make it impossible to test agents

[cyh2004]

import gymnasium as gym
import browsergym.core  # register the openended task as a gym environment
import browsergym.workarena
from bgym import HighLevelActionSet
from time import sleep

env_ids = [id for id in gym.envs.registry.keys() if id.startswith("browsergym/workarena")]
print("\n".join(env_ids))
# start an openended environment
env = gym.make(
    "browsergym/workarena.servicenow.filter-asset-list",
    action_mapping=HighLevelActionSet("coord").to_python_code,
    headless=False
)
# run the environment <> agent loop until termination
obs, info = env.reset()

env.step("mouse_click(62, 75)") # click on the filter button
sleep(3)
env.step("mouse_click(109, 171)") # click on the choose option
sleep(3)
env.step("mouse_click(123, 286)") # click on the option
sleep(3)
env.step("mouse_click(310, 167)") # click on the cond operator
import pdb; pdb.set_trace()

# release the environment
env.close()

I run this code to see the process of workarena.servicenow.filter-asset-list, the last action is to click the cond operator in the filter.(region in the red rectangle)

But after this action, the dropdown menu appears and then disappears

For a agent which takes screenshot as input and outputs action with coordinates like click(300,200), it will stuck on this step of this task

[cyh2004]

This specific dropdown menu has some strange properties:

• It seems to be unable to inspect. When I right-click the menu, it disappears and the option I clicked is chosen.
• It disappears easily. Scrolling the wheel of the mouse or click on a random place makes it disappears.
• It will appear on top of other application's windows.

For the last one, it looks like this:

I insert a breakpoint in the code above, use a terminal to hide the window of the browser, and manually run env.step("mouse_click(310, 167)") again.

I can't figure out whether its a problem with OS or something. Have you tried operating this dropdown menu with actions based on coordinates?

[Joanna0123]

I'm also very interested in this issue. It seems that because of this problem, it's quite difficult for us to test the performance of a GUI agent with only visual observation on WorkArena. Are there any potential solutions to this issue?

[aldro61]

Hey @cyh2004 and @Joanna0123, thanks for raising this concern. I looked into the DOM of list pages and I see that this dropdown is a standard HTML select input. I suspect that these glitches are due to how the browser handles such elements.

This may also be due to some style attributes or JavaScript used in the ServiceNow product. However, the goal of this benchmark is precisely to evaluate agents on real-world enterprise software which is guaranteed to be imperfect. This is the unfortunate reality of the World "Wild" Web :)

Our .cheat functions show that these tasks are solvable via playwright commands. If screenshot-only doesn't work, one potential solution might be to look into hybrid agents that examine both the DOM and a vision modality. Our experiments with such agents have been successful on this UI.

Good luck!

[recursix]

Should we investigate if it's a behavior of bgym/playwright through the coordinate action space? We should make sure there is no bug.

[aldro61]

Good point, it's possible that this is due to a bug in the coordinate action space. The .cheat functions don't test that since they rely on javascript selectors. Do you want to take a look @recursix?

[cyh2004]

I found a relevant issue from playwright repo:

microsoft/playwright#14172 (comment)

The "dropdown" portion of a select is rendered natively by default and not part of the DOM which is what it looks like in your video as well.

(NB: Some UI libraries will do some magic to render the dropdown as a div in a DOM, but assuming they are implemented semantically and robustly, selectOption should work in that case too.)

It can be the reason why I can't click the dropdown menu.

And I found that the dropdown menu beside the problematic one is rendered as a div as well as a select, which makes it behave as expected under any condition

Is it possible that all select element be rendered as a div as well?