Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions docs/interpreter.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,8 +273,25 @@ This interpreter provisioning is designed to coexist with `rules_python`:
registration, so these interpreters work with all existing Python rules.
- The `@rules_python//python/config_settings:python_version` flag is kept in
sync with our own version flag via build transitions.
- `py_runtime` and `py_runtime_pair` from `rules_python` are used to create
the runtime providers.
- Runtimes registered with `rules_python`'s `py_runtime` / `py_runtime_pair`
(for example a system interpreter) remain usable by rules_py rules, which
read the runtime fields structurally.
- Build actions that run an interpreter (wheel installation, site-packages
merging) resolve `@aspect_rules_py//py/private/toolchain:exec_tools_toolchain_type`,
registered by `interpreters.toolchain()`. The exec interpreter follows the
Python version flags when the hub provisions that version; otherwise it
falls back to the hub's highest provisioned version — including the hub
rules_py itself registers, so this resolves even in modules that provision
interpreters only through `rules_python`'s `python.toolchain()`. rules_py
registers nothing under `rules_python`'s exec-tools type, leaving it —
including precompiling — entirely to `rules_python`.

Note that runtimes provisioned by `interpreters.toolchain()` carry
`rules_python`'s public `PyRuntimeInfo` (re-exported from
`@aspect_rules_py//py:defs.bzl`), so `rules_python`-defined executables and
their downstream consumers (`py_zipapp_binary`, `py_interpreter`) analyze and
run on them. No coverage tool is bundled, so `rules_python`-rule coverage on
these runtimes is unavailable.

You can migrate incrementally: replace `python.toolchain()` calls with
`interpreters.toolchain()` and remove the `rules_python` interpreter
Expand Down
4 changes: 1 addition & 3 deletions e2e/cases/rules-python-consumers/consumers_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,9 @@ def read(rel):
python3 = read("python3_var.txt")
assert "python_interpreters+python_3_13" in python3, python3

# The exec-tools payload carries rules_python's expected shape: an exec
# runtime from a rules_py-provisioned repo, and a precompiler field (None).
# The exec-tools payload serves an exec runtime from a rules_py-provisioned repo.
facts = read("exec_tools_facts.txt").splitlines()
assert "python_interpreters+" in facts[0], facts
assert facts[1] == "None", facts

assert read("python_launcher.txt") == "3.11"

Expand Down
14 changes: 5 additions & 9 deletions e2e/cases/rules-python-consumers/exec_tools_facts.bzl
Original file line number Diff line number Diff line change
@@ -1,19 +1,15 @@
"""Materialises the resolved exec-tools toolchain payload for assertion.

Reads the fields rules_python's consumers access — `exec_tools.exec_runtime`
and `exec_tools.precompiler` — so the test pins that rules_py's registration
under rules_python's toolchain type keeps their expected shape.
Reads the runtime from rules_py's exec-tools toolchain type so the test pins
that it serves a rules_py-provisioned interpreter.
"""

EXEC_TOOLS_TOOLCHAIN = "@rules_python//python:exec_tools_toolchain_type"
EXEC_TOOLS_TOOLCHAIN = "@aspect_rules_py//py/private/toolchain:exec_tools_toolchain_type"

def _exec_tools_facts_impl(ctx):
exec_tools = ctx.toolchains[EXEC_TOOLS_TOOLCHAIN].exec_tools
exec_runtime = ctx.toolchains[EXEC_TOOLS_TOOLCHAIN].exec_runtime
out = ctx.actions.declare_file(ctx.label.name + ".txt")
ctx.actions.write(out, "{}\n{}\n".format(
exec_tools.exec_runtime.interpreter.path,
exec_tools.precompiler,
))
ctx.actions.write(out, exec_runtime.interpreter.path + "\n")
return [DefaultInfo(files = depset([out]))]

exec_tools_facts = rule(
Expand Down
6 changes: 3 additions & 3 deletions e2e/cases/rules-python-consumers/report_exec_version.bzl
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
"""Runnable that prints the resolved exec-tools runtime version.

Resolves rules_python's exec-tools toolchain directly (no py_* version
Resolves rules_py's exec-tools toolchain directly (no py_* version
transition), so the reported version reflects the version flag only if the flag
is authoritative in the interpreter hub. `bazel run` it and assert on stdout.
"""

_EXEC_TOOLS_TOOLCHAIN = "@rules_python//python:exec_tools_toolchain_type"
_EXEC_TOOLS_TOOLCHAIN = "@aspect_rules_py//py/private/toolchain:exec_tools_toolchain_type"

def _report_exec_version_impl(ctx):
version_info = ctx.toolchains[_EXEC_TOOLS_TOOLCHAIN].exec_tools.exec_runtime.interpreter_version_info
version_info = ctx.toolchains[_EXEC_TOOLS_TOOLCHAIN].exec_runtime.interpreter_version_info
launcher = ctx.actions.declare_file(ctx.label.name + ".sh")
ctx.actions.write(
output = launcher,
Expand Down
2 changes: 1 addition & 1 deletion e2e/cases/rules-python-consumers/test.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
#
# report_exec_version resolves rules_python's exec-tools toolchain without the
# report_exec_version resolves rules_py's exec-tools toolchain without the
# py_* version transition, so it prints the requested version only if the
# version flag is authoritative in the interpreter hub. Both the native flag
# and @rules_python's fallback must select the requested version.
Expand Down
4 changes: 2 additions & 2 deletions e2e/cases/uv-deps-650/crossbuild/toolchain_test.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,10 @@ path into a file, so sh_test scripts can verify which interpreter was selected
under cross-compilation (target platform ≠ exec platform).
"""

EXEC_TOOLS_TOOLCHAIN = "@rules_python//python:exec_tools_toolchain_type"
EXEC_TOOLS_TOOLCHAIN = "@aspect_rules_py//py/private/toolchain:exec_tools_toolchain_type"

def _exec_python_path_impl(ctx):
exec_runtime = ctx.toolchains[EXEC_TOOLS_TOOLCHAIN].exec_tools.exec_runtime
exec_runtime = ctx.toolchains[EXEC_TOOLS_TOOLCHAIN].exec_runtime
out = ctx.actions.declare_file(ctx.label.name + ".txt")
ctx.actions.write(out, exec_runtime.interpreter.path)
return [DefaultInfo(files = depset([out]))]
Expand Down
4 changes: 2 additions & 2 deletions e2e/interpreter-runtime-metadata/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ load(":runtime_metadata_test.bzl", "runtime_metadata_test")
runtime_metadata_test(
name = "regular",
abi_flags = "",
runtime = "@python_3_15_x86_64_unknown_linux_gnu//:py3_runtime",
runtime = "@python_3_15_x86_64_unknown_linux_gnu//:runtime",
)

runtime_metadata_test(
name = "freethreaded",
abi_flags = "t",
runtime = "@python_3_15_x86_64_unknown_linux_gnu_freethreaded//:py3_runtime",
runtime = "@python_3_15_x86_64_unknown_linux_gnu_freethreaded//:runtime",
)
2 changes: 1 addition & 1 deletion e2e/interpreter-runtime-metadata/runtime_metadata_test.bzl
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""Checks metadata on a provisioned PBS Python runtime."""

load("@rules_python//python:py_runtime_info.bzl", "PyRuntimeInfo")
load("@aspect_rules_py//py:defs.bzl", "PyRuntimeInfo")

def _assert_equal(description, expected, actual):
if actual != expected:
Expand Down
12 changes: 12 additions & 0 deletions e2e/rules-python-interop/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -79,3 +79,15 @@ rules_py_test(
main = "pip_deps_test.py",
deps = ["@pip//six"],
)

# Deps from a rules_py uv hub: whl_install's exec-tools requirement must be
# satisfiable with only rules_python-provisioned interpreters, via the
# adapter toolchain rules_py registers itself.
rules_py_test(
name = "uv_deps_test",
srcs = ["uv_deps_test.py"],
dep_group = "py-binary-example",
main = "uv_deps_test.py",
python_version = "3.12",
deps = ["@rpy_pypi//cowsay"],
)
12 changes: 12 additions & 0 deletions e2e/rules-python-interop/MODULE.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,15 @@ pip.parse(
requirements_lock = "//:requirements.txt",
)
use_repo(pip, "pip")

# A rules_py uv hub: whl_install must find an exec-tools runtime even though
# only rules_python provisions interpreters here (the bundled adapter bridges
# rules_python's exec-tools toolchain into rules_py's type).
uv = use_extension("@aspect_rules_py//uv:extensions.bzl", "uv")
uv.declare_hub(hub_name = "rpy_pypi")
uv.project(
hub_name = "rpy_pypi",
lock = "//:uv.lock",
pyproject = "//:pyproject.toml",
)
use_repo(uv, "rpy_pypi")
7 changes: 7 additions & 0 deletions e2e/rules-python-interop/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[project]
name = "py-binary-example"
version = "0.0.0"
requires-python = ">=3.11"
dependencies = [
"cowsay",
]
22 changes: 22 additions & 0 deletions e2e/rules-python-interop/uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions e2e/rules-python-interop/uv_deps_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
"""Asserts a uv-hub wheel imports inside a venv assembled over the
rules_python-provisioned runtime — exercising whl_install's exec-tools
resolution in a module with no rules_py interpreters.
"""

import cowsay

assert cowsay.get_output_string("cow", "moo")
print("OK")
1 change: 1 addition & 0 deletions py/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ bzl_library(
"//py/private:py_wheel",
"//py/private:virtual",
"//py/private/interpreter:current_py_toolchain",
"//py/private/interpreter:runtime",
"//py/private/py_venv:defs",
"@bazel_lib//lib:utils",
"@rules_python//python:packaging_bzl",
Expand Down
7 changes: 7 additions & 0 deletions py/defs.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ which has been registered in the `MODULE.bazel` file, e.g.:
```starlark
interpreters = use_extension("@aspect_rules_py//py:extensions.bzl", "python_interpreters")
interpreters.toolchain(python_version = "3.9")
interpreters.toolchain(python_version = "3.12")
use_repo(interpreters, "python_interpreters")

register_toolchains("@python_interpreters//:all")
```
"""
Expand All @@ -31,6 +33,7 @@ load("//py/private:py_pytest_main.bzl", _py_pytest_main = "py_pytest_main", _pyt
load("//py/private:py_unpacked_wheel.bzl", _py_unpacked_wheel = "py_unpacked_wheel")
load("//py/private:virtual.bzl", _resolutions = "resolutions")
load("//py/private/interpreter:current_py_toolchain.bzl", _current_py_toolchain = "current_py_toolchain")
load("//py/private/interpreter:runtime.bzl", _PyRuntimeInfo = "PyRuntimeInfo")
load(
"//py/private/py_venv:defs.bzl",
_py_binary_with_venv = "py_binary_with_venv",
Expand Down Expand Up @@ -61,6 +64,10 @@ PyLayerTierInfo = _PyLayerTierInfo
# The PyInfo provider used by rules_py
PyInfo = _PyInfo

# The runtime provider carried by rules_py-provisioned interpreter toolchains:
# rules_python's public PyRuntimeInfo, the shared standard-toolchain contract.
PyRuntimeInfo = _PyRuntimeInfo

resolutions = _resolutions

def _resolve_main(name, srcs, main):
Expand Down
6 changes: 6 additions & 0 deletions py/private/interpreter/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,12 @@ bzl_library(
deps = [":sanitize"],
)

bzl_library(
name = "runtime",
srcs = ["runtime.bzl"],
deps = ["@rules_python//python:py_runtime_info_bzl"],
)

bzl_library(
name = "sanitize",
srcs = ["sanitize.bzl"],
Expand Down
55 changes: 36 additions & 19 deletions py/private/interpreter/repository.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -201,9 +201,7 @@ cc_library(
return """\
load("@rules_cc//cc:cc_import.bzl", "cc_import")
load("@rules_cc//cc:cc_library.bzl", "cc_library")
load("@rules_python//python:py_runtime.bzl", "py_runtime")
load("@rules_python//python:py_runtime_pair.bzl", "py_runtime_pair")
load("@rules_python//python:py_exec_tools_toolchain.bzl", "py_exec_tools_toolchain")
load("@aspect_rules_py//py/private/interpreter:runtime.bzl", "py_runtime_toolchain")
load("@rules_python//python/cc:py_cc_toolchain.bzl", "py_cc_toolchain")

package(default_visibility = ["//visibility:public"])
Expand Down Expand Up @@ -244,8 +242,8 @@ filegroup(
deps = {header_interface_deps},
)

py_runtime(
name = "py3_runtime",
py_runtime_toolchain(
name = "runtime",
abi_flags = "{abi_flags}",
files = [":files"],
interpreter = "{python_bin}",
Expand All @@ -256,17 +254,6 @@ py_runtime(
"releaselevel": "{releaselevel}",
"serial": "{serial}",
}},
python_version = "PY3",
)

py_runtime_pair(
name = "runtime_pair",
py2_runtime = None,
py3_runtime = ":py3_runtime",
)

py_exec_tools_toolchain(
name = "exec_tools_toolchain",
)

py_cc_toolchain(
Expand Down Expand Up @@ -433,6 +420,7 @@ config_setting(
# offers a matching pair: https://github.com/aspect-build/rules_py/issues/1095
#
# Second pass: emit toolchain() registrations.
exec_tools_fallbacks = {} # repr(exec_compatible_with) -> (version tuple, name, repo, constraints)
for info, platform_setting_names in toolchain_infos:
extra_config_settings = info.get("config_settings", [])
extra_target_compatible = info.get("target_compatible_with", [])
Expand Down Expand Up @@ -461,7 +449,7 @@ toolchain(
name = "{name}",
target_compatible_with = {target_compatible_with},
target_settings = {target_settings},
toolchain = "@{repo}//:runtime_pair",
toolchain = "@{repo}//:runtime",
toolchain_type = "@bazel_tools//tools/python:toolchain_type",
)

Expand All @@ -483,18 +471,47 @@ toolchain(
content.append("""# Exec tools toolchain: selected by exec platform (not target platform) so
# that build actions using the interpreter (e.g. compileall) get a runnable
# binary on the build host regardless of the target platform being built for.
# Version-gated so the exec interpreter follows the version flags.
toolchain(
name = "{name}_exec_tools",
exec_compatible_with = {exec_compatible_with},
toolchain = "@{repo}//:exec_tools_toolchain",
toolchain_type = "@rules_python//python:exec_tools_toolchain_type",
target_settings = ["{version_setting}"],
toolchain = "@{repo}//:runtime",
toolchain_type = "@aspect_rules_py//py/private/toolchain:exec_tools_toolchain_type",
)
""".format(
name = info["name"],
repo = info["repo"],
exec_compatible_with = exec_compatible_with,
version_setting = version_setting,
))

# Track the platform's highest provisioned version for the
# ungated fallback entry emitted after the loop.
version_key = tuple([int(part) for part in info["python_version"].split(".")])
prior = exec_tools_fallbacks.get(repr(exec_compatible_with))
if prior == None or version_key > prior[0]:
exec_tools_fallbacks[repr(exec_compatible_with)] = (version_key, info["name"], info["repo"], exec_compatible_with)

# Ungated exec-tools fallbacks: build actions only need *a* runnable host
# interpreter, so configurations matching no version-gated entry (e.g. the
# version flags at defaults not provisioned by this hub) fall back to the
# platform's highest provisioned version. Registration is lexicographic by
# name, so the `zz_` prefix sorts these after every gated entry.
for _, name, repo, constraints in exec_tools_fallbacks.values():
content.append("""
toolchain(
name = "zz_{name}_exec_tools_fallback",
exec_compatible_with = {exec_compatible_with},
toolchain = "@{repo}//:runtime",
toolchain_type = "@aspect_rules_py//py/private/toolchain:exec_tools_toolchain_type",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Falling back to the highest provisioned exec interpreter when the active target version is absent is unsafe for compile_pyc. WhlInstall takes the target layout/version from the standard toolchain but runs compileall with this exec runtime; a 3.12 target plus a 3.13-only rules_py hub therefore installs 3.13-magic pyc under python3.12. Require a matching exec version (or explicitly disable/error on pyc compilation when none exists) and add a mixed-version compile_pyc regression.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new major/minor guard in whl_install/rule.bzl correctly disables compileall when target and exec runtimes differ, and the matching/mismatching action-argv tests exercise both paths. That addresses the wrong-magic .pyc failure; resolving.

)
""".format(
name = name,
repo = repo,
exec_compatible_with = constraints,
))

content.append("""
exports_files(
["BUILD.bazel"],
Expand Down
Loading
Loading