Skip to content

[Bug]: avoid downloading foreign whl while running gazelle #1271

Description

@comicfans

What happened?

Hello,

gazelle_index_whl = prebuilds.values()[0] # Effectively random choice :shrug:

    if prebuilds:
        gazelle_index_whl = prebuilds.values()[0]  # Effectively random choice :shrug:
    elif default_target:

this picks random wheel from the index, for example download a win32 whl on linux, which can lead to unnecessary downloads. Would it make more sense to try the host-architecture wheel first, since it is very likely already downloaded for execution?

I created a prototype something like

def pick_gazelle_index_whl(repository_ctx, prebuilds):
    """Pick a wheel for Gazelle indexing without fetching foreign platforms.

    Any wheel of the package works for indexing, but prefer an "any" wheel,
    then a wheel matching the host platform: the build/test is very likely to
    download that host-platform wheel already, so reusing it avoids fetching a
    large foreign-platform wheel purely for indexing.
    """
    bazel_os = repository_ctx.os.name
    bazel_arch = repository_ctx.os.arch

    # repository_ctx os:
    # name = linux, arch = amd64
    # name = linux, arch = aarch64
    # name = "mac os x", arch = aarch64
    #
    # pip platform_tags (from uv.lock):
    # [musllinux_1_2_x86_64]
    # [linux_aarch64, manylinux_2_17_aarch64]
    # [linux_x86_64, manylinux_2_17_x86_64]
    # [win_amd64],
    # [win_arm64]
    # [macosx_10_9_x86_64], 
    # [macosx_11_0_universal2], 
    # [macosx_11_0_arm64]
    # 
    #
    # note: only linux amd64/aarch64 and osx aarch64 are handled here.
    for whl, target in prebuilds.items():
        parsed = parse_whl_name(whl)
        for tag in parsed.platform_tags:
            if tag == "any":
                return target
            elif (bazel_os == "linux") and any([tag.startswith(v) for v in ["linux", "manylinux", "musllinux"]]):
                pip_arch = {"amd64": "x86_64"}.get(bazel_arch, bazel_arch)
                if tag.endswith("_" + pip_arch):
                    return target
            elif (bazel_os == "mac os x") and tag.startswith("macosx"):
                if tag.endswith("universal2") or tag.endswith("arm64"):
                    return target
    return prebuilds.values()[0]

Version

Development (host) and target OS/architectures:

linux/x86_64

Output of bazel --version:

bazel 8.5.1

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file:

Language(s) and/or frameworks involved:

python

How to reproduce

run gazelle_python_manifest on linux 86_64, it will download aarch64 osx/win32 whl files

Any other information?

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions