[FR]: Remove need for runfiles directory to be writable

[jsharpe]

What is the current behavior?

The venv is recreated whenever the binary is executed. This prevents packaging up the binary and installing it to a system outside of bazel as an application deployment that can be run by other users from a common directory e.g. an /apps directory, possibly served read-only over nfs or similar.

Describe the feature

I would like either for the venv creation to fallback to a user writable location e.g. XDG_CACHE or to remove the need to recreate the venv links so it can be done once by an admin user and then subsequent users can just execute from the precreated venv.

[jsharpe]

Another place where this could be used is in an immutable container - the container is built and then at execution time the container image is made immutable and so writing to the runfiles directory would potentially not be possible in this case too.

[jsharpe]

Also the current behaviour would prevent multiple copies of a binary running at the same time since it would delete files being used by the first process on the execution of the second process.

[voxeljorge]

We've run into this issue with when running py_binary targets from this ruleset in an immutable environment. One example is when running code in a os-only aws lambda runtime, the lambda is unpacked in an immutable env and the py_binary fails to run.

[voxeljorge]

@alexeagle this issue is stopping us from migrating a couple of our lambdas to rules_py, and it seems somewhat straightforward to resolve. The runner could accept an env var as proposed above. Either XDG_CACHE or something specifically named after rules_py seems reasonable, any specific preference?

[thesayyn]

Little bit of context and progress here;

We have recently started using uv for venv generation which supports relative paths astral-sh/uv#3717 it's still experimental but it's almost there.

But we are still blocked on python/cpython#83650 which still forces us to use an absolute path pyenv.cfg file, bummer.

However, i have been hitting my head against the wall trying to solve this and we have solved this with @voxeljorge. It's approximately #427 but needs a static symlink from /tmp/venv back to runfiles tree to allow venv being created in /tmp/venv but available in the runfiles tree.

[arrdem]

Some updates here. Removing the need for the virtualenv to be writable requires two things, a relocatable "python" a take at which is #546, and to assemble the site-packages and other required directory trees at build time which is what I was prototyping on last week.

Conceptually it should be as simple as running our existing virtualenv builder tool with a normal build action, rather than at application runtime. Unfortunately due to the external/, bazel-bin/ and bazel-bin/external behavior which Bazel uses it's not quite so simple. The specific problem is that when an application is "built" it sees the .runfiles/ layout of external files which requires a site-packages tree of symlinks to have one set of references up and out to other workspaces in the runfiles. But inbetween build steps -- for instance a jinja2 binary used during building -- those symlinks would need to go into the external and especially the bazel-bin trees which don't exist at runtime.

If we could assume that all external files are source files, which is currently true for rules_python's implementaton of pip dependencies, we could just copy_to_bin [1] everything and then create a consistent tree of symlinks which would be valid at post-build .runfiles/ time. However that simplifying assumption isn't safe given forthcoming rules_python improvements.

So it looks like this feature requires integrating both sources and build results from the current and external workspaces into a single relocatable site-packages tree, which is really only doable via copying rather than symlinking.

For efficiency at .runfiles time, this means we also need to manipulate the final runfiles set to drop references to runfiles which we copy into a a virtualenv. Otherwise we may have multiple copies of source files running around.

The short path to implementing this appears to be adding both some file search machinery and a copying codepath to the existing virtualenv tool, to be activated by a feature flag set by the static virtualenv building action. Hopefully will have something on that this week.