[PopupMenu] menu position and menu mouse position return values change when per_pixel_transparency is allowed

[Koyper]

Tested versions

Reproducible in 4.5.x and higher.

System information

MacOS 14.7.5

Issue description

For PopupMenu, per_pixel_transparency is affecting return values of at least get_position and also get_mouse_position as compared to when per_pixel_transparency is disabled. The amount different from each other appears to be a multiple of theme_cache.panel_style->get_margin for top, left, etc., where the theme margins wind up being applied at least twice.

This issue arose while fixing an issue with PopupMenu (#110256) where a state exists that flickers submenus. The cause of that was incorrectly placed autohide areas and mouse position testing that were not adjusted for theme margins.

While I might be able to work out a way to reverse-out the doubled theme margins when per_pixel_transparency is enabled, it would be a lot cleaner to fix the underlying issue, which I think must be in DisplayServer.

In the screen recordings below, the red area is the autohide_area that closes the submenu if the mouse enters it, and the green line extends to the Point2D in the code that determines if the mouse is within the area.

Here is the behavior in the master. Note that both the autohide area and mouse detection point are incorrect. You can see there is a place where the autohide area overlaps the menu item. This allows the flickering state where the mouse point can enter that area which hides the submenu, but then immediately reopens because the mouse is still inside the menu item. The other confounding part of this is that the code to determine whether the mouse is over a menu item computes a different point from the autohide area detection code.

Master - note the black gap below the menu item should not be there, and the end of the green line does not end at the tip of the cursor like it should:

project.with.master.Screen.Recording.2025-11-10.at.1.22.59.PM.mov

Here's the editor with per_pixel_transparency disabled with fixed code where the autohide area is in the right location and the mouse point is at the cursor tip:

with.fix.editor.no.trans.Screen.Recording.2025-11-10.at.1.10.29.PM.mov

Here's the editor with the same fixed code, but only with per_pixel_transparency enabled. Note in the upper left corner of the autohide area the shift that appears to be 2X theme margin, and that the green line is offset from the cursor by the same amount.

with.fix.editor.with.trans.Screen.Recording.2025-11-10.at.1.12.57.PM.mov

Here's a confusing result - when per_pixel_transparency is enabled for a running project with the fix, only the autohide area is offset, not the mouse point.

project.with.fix.trans.mouse.okayScreen.Recording.2025-11-10.at.1.55.34.PM.mov

Steps to reproduce

You'll need to have a debug draw on the autohide area to visualize the offset, probably easiest in _draw_items. Here's the declaration for it, and it contains one or two Rect2, the top and bottom area.

	List<Rect2> autohide_areas;

Minimal reproduction project (MRP)

[Edited]
Here is a place in _activate_submenu where the Rect2 value of this_rect changes when per_pixel_transparency changes.

	const Point2 panel_ofs_start = Point2(panel->get_offset(SIDE_LEFT), panel->get_offset(SIDE_TOP)) * win_scale;
	const Point2 panel_ofs_end = Point2(-panel->get_offset(SIDE_RIGHT), -panel->get_offset(SIDE_BOTTOM)) * win_scale;

	const Point2 this_pos = get_position() + Point2(0, panel_ofs_start.y + theme_cache.panel_style->get_margin(SIDE_TOP) * win_scale);
	Rect2 this_rect(this_pos, get_size());
	print_line("ORIG THIS RECT", this_rect);

The theme_cache and win_scale values do not change, so the wrong value seems to be returned by panel->get_offset

[Koyper]

I edited the MRP info above to include more information that might help track down the problem.

[Koyper]

I started working on a workaround in #110256 that eliminates the use of get_offset, which is what changes with per_pixel_transparency I think because of a dropshadow width shift when transparency is enabled. I will update here if I can't overcome the mouse offset issue.