Modal does not restore focus inside custom element

[angrycat9000]

Describe the bug

Modal does not return focus to the previous element on closing when hosted inside of a custom element shadow root.

To Reproduce

Steps to reproduce the behavior:

1. Create a custom element with a shadow root that hosts a react render root
2. Render a Modal and a button to open in the custom element shadowroot.
3. Tab to the button and press enter to open the modal.
4. Press Escape to close the modal
   • Expect: the original button to have focus
   • Result: the button is not focused

Reproducible Example

Minimal example by using CodeSandbox.

Expected behavior

The original button should have focus.

Screenshots

modal-focus-bug-edit.mp4

Environment (please complete the following information)

• Operating System: windows
• Browser, Version Chrome 133
• react-overlays Version ?

Additional context

document.activeElement returns the custom element since it is the deepest element in the light DOM that has focus. However restoring focus to the custom element in the light DOM isn't enough. There could be multiple elements in the shadow DOM of the custom element. And at this point the shadowRoot.activeElement has lost the reference to the originally focused elements.

the activeElement would need to capture the deepest element piercing the shadow roots when saving the originally focused element.