[Bug]: NVIDIA's isaacsim/isaaclab + omni broken in aspect_rules_py' py_binary/py_test

[kwlzn]

What happened?

Issue Description

When attempting to leverage the aspect_rules_py ruleset in our repo, we found that some third party dependencies in the NVIDIA isaacsim/isaaclab/omni ecosystem break on transitive imports. I have created a minimal repro to demonstrate the exact failure here: https://github.com/kwlzn/bazel_plus_isaaclab

when either the py_binary or py_test rules in this repo are invoked, we see a transitive import failure on the carb package during a from isaaclab.app import AppLauncher:

$ bazel run //:isaaclab_bin
INFO: Analyzed target //:isaaclab_bin (255 packages loaded, 75746 targets configured).
...
INFO: Running command line: bazel-bin/isaaclab_bin
  ⚠ Conflicting symlinks found when attempting to create venv. More than one package provides the file at these paths
   ╭─[1:174]
 1 │ /home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/lib/python3.10/site-packages/isaacsim/__init__.py
   ·                                                                                                                                                                              ─────────────────┬────────────────
   ·                                                                                                                                                                                               ╰── Existing file in virtual environment
 2 │ /home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/rules_python~~pip~pypi_isaac_310_isaacsim_app/site-packages/isaacsim/__init__.py
   ·                                                                                                                                                                                          ─────────────────┬────────────────
   ·                                                                                                                                                                                                           ╰── Next file to link
   ╰────
  help: Set `package_collisions = "ignore"` on the binary or test rule to ignore this warning

Traceback (most recent call last):
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/_main/isaaclab.py", line 3, in <module>
    from isaaclab.app import AppLauncher
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/lib/python3.10/site-packages/isaaclab/__init__.py", line 49, in <module>
    bootstrap_kernel()
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/lib/python3.10/site-packages/isaaclab/__init__.py", line 22, in bootstrap_kernel
    import carb
ModuleNotFoundError: No module named 'carb'

carb is definitely included in the venv contents, but does not appear to be properly setup for imports:

$ ls bazel-bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/lib/python3.10/site-packages/omni/kernel/py/carb
audio                                  dictionary       __init__.py                            profiler  windowing.cpython-310-x86_64-linux-gnu.so
_carb.cpython-310-x86_64-linux-gnu.so  eventdispatcher  input.cpython-310-x86_64-linux-gnu.so  settings  windowing.pyi
_carb.pyi                              events           input.pyi                              tokens
datasource                             imgui.py         l10n.cpython-310-x86_64-linux-gnu.so   variant

$ source bazel-bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/bin/activate
(.isaaclab_bin.venv) 
$ python3
Python 3.10.16 (main, Dec  6 2024, 19:59:16) [Clang 18.1.8 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from isaaclab.app import AppLauncher
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/lib/python3.10/site-packages/isaaclab/__init__.py", line 49, in <module>
    bootstrap_kernel()
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/lib/python3.10/site-packages/isaaclab/__init__.py", line 22, in bootstrap_kernel
    import carb
ModuleNotFoundError: No module named 'carb'
>>> 

Expected Result

the from isaaclab.app import AppLauncher import should be able to import all of it's transitive deps, as demonstrated in this uv-based reproduction:

$ OMNI_KIT_ACCEPT_EULA=YES \
  uv run \
    --with=torch==2.5.1+cu121 \
    --extra-index-url=https://download.pytorch.org/whl/cu121 \
    --with="isaacsim[rl,extscache]==4.5.0" \
    --with "isaaclab[isaacsim,all]==2.0.2" \
    --extra-index-url https://pypi.nvidia.com \
    --python=python3.10 \
    python3
Python 3.10.16 (main, Feb 12 2025, 14:50:02) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from isaaclab.app import AppLauncher
>>> import carb
>>> carb.__file__
'/home/kwilson/.cache/uv/archive-v0/2t2CSki83eVP_Ola6IgSA/lib/python3.10/site-packages/omni/kernel/py/carb/__init__.py'

Version

Development (host) and target OS/architectures: Ubuntu 24.x

Output of bazel --version: 7.6.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: 1.3.2 (latest)

Language(s) and/or frameworks involved: python 3.10

How to reproduce

git clone https://github.com/kwlzn/bazel_plus_isaaclab
cd bazel_plus_isaaclab
bazel run //:isaaclab_bin

Any other information?

No response

[kwlzn]

further debugging attempts here reveal that the transitive carb import (which is backed by native extensions/.so files) seems to be failing due to an inability to dynamically resolve RPATH search paths at import time:

$ ls -al ./lib/python3.10/site-packages/omni/kernel/py/carb/_carb.cpython-310-x86_64-linux-gnu.so
lrwxrwxrwx 1 kwilson kwilson 257 Apr 10 10:36 ./lib/python3.10/site-packages/omni/kernel/py/carb/_carb.cpython-310-x86_64-linux-gnu.so -> /home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/rules_python~~pip~pypi_isaac_310_omniverse_kit/site-packages/omni/kernel/py/carb/_carb.cpython-310-x86_64-linux-gnu.so
$ ldd ./lib/python3.10/site-packages/omni/kernel/py/carb/_carb.cpython-310-x86_64-linux-gnu.so |grep "not found"
        libcarb.so => not found
        libpython3.10.so.1.0 => not found
$ readelf -d ./lib/python3.10/site-packages/omni/kernel/py/carb/_carb.cpython-310-x86_64-linux-gnu.so | grep -E "RPATH|RUNPATH"
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../..:$ORIGIN]
$ ls -al ./lib/python3.10/site-packages/omni/libcarb.so
lrwxrwxrwx 1 kwilson kwilson 215 Apr 10 10:36 ./lib/python3.10/site-packages/omni/libcarb.so -> /home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/isaaclab_bin.runfiles/rules_python~~pip~pypi_isaac_310_omniverse_kit/site-packages/omni/libcarb.so

LD_DEBUG in the working uv case shows this functional E2E:

LD_DEBUG=libs,files \
OMNI_KIT_ACCEPT_EULA=YES \
  uv run \
    --with=torch==2.5.1+cu121 \
    --extra-index-url=https://download.pytorch.org/whl/cu121 \
    --with="isaacsim[rl,extscache]==4.5.0" \
    --with "isaaclab[isaacsim,all]==2.0.2" \
    --extra-index-url https://pypi.nvidia.com \
    --python=python3.10 \
    python3 -c "import sys; print('>>>>>>>>>>>>>>>>>> START', file=sys.stderr); from isaaclab.app import AppLauncher; import carb; print(carb.__file__); print('>>>>>>>>>>>>>>>>>>>>>>>> END', file=sys.stderr)"

but comes up relatively barren in the aspect_rules_py case:

$ source bazel-bin/isaaclab_bin.runfiles/.isaaclab_bin.venv/bin/activate
$ LD_DEBUG=libs,files python3 -c "import sys; print('>>>>>>>>>>>>>>>>>> START', file=sys.stderr); from isaaclab.app import AppLauncher; import carb; print(carb.__file__); print('>>>>>>>>>>>>>>>>>>>>>>>> END', file=sys.stderr)"

I currently believe this is somehow related to the use of symlinks and the dynamic linker on Linux resolving search paths relative to the symlink source vs target. more digging needed, but wondering if this is ringing any bells around similar known issues.

I will also revise the issue title/description scope accordingly.

[kwlzn]

more clues:

from omni.kit_app import KitApp + transitive carb import via omniverse-kit - works in uv:

$ OMNI_KIT_ACCEPT_EULA=YES uv run --extra-index-url https://pypi.nvidia.com --with omniverse-kit --python=python3.10 python3 -c "import sys; print(sys.path); before=sys.path[:]; from omni.kit_app import KitApp; import carb; print(carb.__file__); import omni.kit.app as omni_kit_app; print(omni_kit_app.__file__); print(sys.path); print(set(sys.path[:]) - set(before))"
['', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python310.zip', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/lib-dynload', '/home/kwilson/.cache/uv/archive-v0/Z5Glbe197xA5jNB30FqsO/lib/python3.10/site-packages', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/site-packages']
/home/kwilson/.cache/uv/archive-v0/Z5Glbe197xA5jNB30FqsO/lib/python3.10/site-packages/omni/kernel/py/carb/__init__.py
/home/kwilson/.cache/uv/archive-v0/Z5Glbe197xA5jNB30FqsO/lib/python3.10/site-packages/omni/kernel/py/omni/kit/app/__init__.py
['/home/kwilson/.cache/uv/archive-v0/Z5Glbe197xA5jNB30FqsO/lib/python3.10/site-packages/omni/kernel/py', '', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python310.zip', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/lib-dynload', '/home/kwilson/.cache/uv/archive-v0/Z5Glbe197xA5jNB30FqsO/lib/python3.10/site-packages', '/home/kwilson/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/site-packages']
{'/home/kwilson/.cache/uv/archive-v0/Z5Glbe197xA5jNB30FqsO/lib/python3.10/site-packages/omni/kernel/py'}

and then the same transitive import carb does work in aspect_rules_py after sys.path setup, but fails during a transitive import omni.kit.app (and both have identical relative sys.path mutations under the hood):

$ cat omni_main.py
import sys
print(sys.path)
before = sys.path[:]
try:
    from omni.kit_app import KitApp
except ImportError:
    import carb
    print(f"CARB IMPORT SUCCESSFUL: {carb.__file__})")
    print(sys.path)
    print(set(sys.path[:]) - set(before))
    raise
$ bazel run //:omni_bin
...
['/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu/lib/python310.zip', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu/lib/python3.10', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu/lib/python3.10/lib-dynload', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles/.omni_bin.venv/lib/python3.10/site-packages', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles/_main']
CARB IMPORT SUCCESSFUL: /home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~pip~pypi_isaac_310_omniverse_kit/site-packages/omni/kernel/py/carb/__init__.py)
['/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~pip~pypi_isaac_310_omniverse_kit/site-packages/omni/kernel/py', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu/lib/python310.zip', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu/lib/python3.10', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~python~python_3_10_x86_64-unknown-linux-gnu/lib/python3.10/lib-dynload', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles/.omni_bin.venv/lib/python3.10/site-packages', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles', '/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles/_main']
{'/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/external/rules_python~~pip~pypi_isaac_310_omniverse_kit/site-packages/omni/kernel/py'}
Traceback (most recent call last):
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles/_main/omni_main.py", line 8, in <module>
    from omni.kit_app import KitApp
  File "/home/kwilson/.cache/bazel/_bazel_kwilson/a4b81d9365baf8af8de81d0c7c631c5e/execroot/_main/bazel-out/k8-fastbuild/bin/omni_bin.runfiles/.omni_bin.venv/lib/python3.10/site-packages/omni/kit_app.py", line 94, in <module>
    import omni.kit.app
ModuleNotFoundError: No module named 'omni.kit'

omni.kit.app is a native module and:

$ ls omni/kernel/py/omni/kit/app/
__init__.py  _app.cpython-310-x86_64-linux-gnu.so  _app.pyi  _impl

$ readelf -d omni/kernel/py/omni/kit/app/_app.cpython-310-x86_64-linux-gnu.so |grep "RPATH"
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/../../../..:$ORIGIN]

$ ldd omni/kernel/py/omni/kit/app/_app.cpython-310-x86_64-linux-gnu.so |grep "not found"
        libcarb.so => not found
        libpython3.10.so.1.0 => not found