pip.parse(python_version=) should not be mandatory

[mering]

🐞 bug report

Affected Rule

Bzlmod pip.parse()

Is this a regression?

Yes, was not a problem in WORKSPACE.

Description

The arg documentation says the following:
https://github.com/bazelbuild/rules_python/blob/711186f144af06b431bd416b2d742874de3a2dea/python/private/bzlmod/pip.bzl#L391-L398

It specifically describes If not specified, then the default Python version (as set by the root module or rules_python) will be used. So this attribute should be optional but currently is mandatory.

🔬 Minimal Reproduction

pip.parse() without python_version.

🔥 Exception or Error

ERROR: in tag at <root>/MODULE.bazel:42:10, mandatory attribute python_version isn't being specified

🌍 Your Environment

Operating System:

  
Linux
  

Output of bazel version:

  
2024/01/19 15:28:42 Warning: used fallback version "6.3.2"
Bazelisk version: v1.19.0
INFO: Running bazel wrapper (see //tools/bazel for details), bazel version 6.1.2 will be used instead of system-wide bazel installation.
Build label: 6.1.2
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Apr 18 15:29:54 2023 (1681831794)
Build timestamp: 1681831794
Build timestamp as int: 1681831794
  

Rules_python version:

  
0.27.1
  

Anything else relevant?

[aignas]

Could you explain a little more how having a requirement on this mandatory argument make it more difficult to use rules_python? Are there cases where you cannot do this? Usually requirements.txt files are python version and platform dependent, so I'd be curious to hear out more about your usecase.

[mering]

Could you explain a little more how having a requirement on this mandatory argument make it more difficult to use rules_python? Are there cases where you cannot do this? Usually requirements.txt files are python version and platform dependent, so I'd be curious to hear out more about your usecase.

When you create a module which uses a requirements.txt but is meant to be depended upon by another module, my module doesn't know with which version the dependent module wants to use. So I have to run this for all available Python versions. If I could instead just leave this out and have it determined automatically by the action graph which version is required, it would be much easier to use.

[lalten]

Did you find a solution for this? https://github.com/mvukov/rules_ros2/blob/1e16de8c3797d94b54ea589524fa7be0e45481d9/MODULE.bazel#L34-L67 is a workaround, but I hate it :P

[aignas]

This may be possible once I finish #1837, but it is not in scope, so contributions in a form of PRs are definitely welcome. I would be more than happy to review those. That said, I am curious of the usecase that you are trying to cover by using all of the Python versions and requirements. Is it for testing that your dependencies work on all python versions or is it something else?

[mering]

I am curious of the usecase that you are trying to cover by using all of the Python versions and requirements. Is it for testing that your dependencies work on all python versions or is it something else?

Isn't this required when you don't know which Python version other modules which depend on your module are using?

Let's say, I develop module X which has pip dependencies. When module A wants to depend on X and uses Python version 3.8, this version needs to be declared in X's pip_parse(). When module B wants to depend on X and uses Python version 3.11, this version also needs to be declared in X's pip_parse(). As a developer of module X I don't know which Python versions the users of my module want to use, so I need to declare all of them at the moment. It would be preferable if I would only need to declare my requirements.txt file and have Bazel figure out the correct Python version to use in module X based on the toolchains used in dependent modules.

[aignas]

Since we have #2530, I want to write down extra thoughts here on what we can do:

• In pip.parse make python_version optional.
• If pip.parse is called without python_version and with python_version, throw an error.
• If pip.parse is called without python_version, then create hub repos for registered 3.x python versions.

[aignas]

Does rctx.download support downloading from a git repository? If not, then this is done, since the VCS requirement specifiers will fallback to pip for now.

[mering]

Does rctx.download support downloading from a git repository? If not, then this is done, since the VCS requirement specifiers will fallback to pip for now.

rctx.download is an HTTP downloader. It's possible to run Git over HTTP, so it should be possible, but someone would have to write a Git protocol implementation in Starlark. git_repository rule runs the local git executable (source).

Could we just fall back to the default Python version in case this is not specified but needed? Can't we just use the default Python toolchain for running pip of not explicitly specified/overriden?

[aignas]

Ah shoot, I have commented on the wrong issue - regarding the git I will post the question and your response to the right issue.

I think we could fallback on the default version, however, that would only work in root modules, because the root module defines the default. Otherwise the behaviour of pip.parse in root and non-root modules would be different.

So we actually have only 2 choices here:

• If the python_version in pip.parse is not specified, use the default.
• If the python_version in pip.parse is not specified, use all Python versions from the MINOR_MAPPING.

I somewhat think the first option is simpler but not sure if preferable. In the future when #2629 is merged and when we don't need Python to extract simple wheels, we can do either of the above. My design goal for PyPI related things is to have somewhat immutable repository rule artifacts (i.e. if the defaults change, the rules don't need to be refetched) and then do the selecting between the defaults in the select statements in the hub repo.

Hence my preference for the second approach where we just use all of the registered python toolchains and configure the wheels for all of them.

[mering]

I think the second option is even better. I have often seen a list comprehension with all versions for pip.parse(), so this would make it easier to get it right in such situations.