Description of the feature request:
With --experimental_remote_repo_contents_cache, a cache-hit repo is injected into the in-memory RemoteExternalFileSystem and only .bzl / REPO.bazel files are written under <output_base>/external/<repo> (RemoteExternalOverlayFileSystem#shouldPrefetch). Bazel itself and local actions read the rest lazily, but a tool running outside the Bazel server has no supported way to learn the contents of those files: they are not on disk, and no query output exposes them.
I'd like one (ideally the first) of the following:
- Source-file digests in query output. An opt-in flag, e.g.
--proto:source_file_digests, that adds digest (and is_executable) to the SourceFile message for query/cquery --output=proto|streamed_proto|jsonproto. For in-memory repos Bazel already holds the digest from the cached Directory tree (RemoteExternalOverlayFileSystem#getDigest), so this costs no downloads. For on-disk files it could use getFastDigest and fall back to hashing. Restricting it to external repos would be fine if main-repo cost is a concern.
- A supported way to materialize specific repos. e.g.
bazel fetch --repo=@@foo --materialize (name is a strawman), wired to the existing LazyMaterializer#ensureMaterialized. Today bazel fetch --remote_download_all does not materialize; the only thing that works is building each source file as a top-level target (bazel build --remote_download_all @@repo//:path/to/file), which doesn't scale to tens of thousands of files.
- A per-repo tree digest. Expose the root
Directory digest of an injected repo (e.g. in bazel mod show_repo --output=streamed_proto, or a file next to the marker file). A tool with remote cache access could then walk the CAS itself. This is the least convenient for tools but the smallest change.
Independently of which option is chosen, a way to detect the situation would help: currently an external tool cannot distinguish "file is remote-backed and only in memory" from "file does not exist".
Which category does this issue belong to?
External Dependency
What underlying problem are you trying to solve with this feature?
I maintain bazel-diff, which computes a hash per target at two commits and diffs them to find the affected targets. The pattern (shared with target-determinator and similar tools) is:
bazel query/cquery --output=streamed_proto to get the target graph.- For every
SourceFile target, read the file from disk and hash its contents. For labels in external repos the path is $(bazel info output_base)/external/<canonical repo>/<pkg>/<file>.
- Combine rule attribute hashes with the hashes of transitive inputs.
Step 2 silently breaks under the remote repo contents cache. The files aren't on disk, so they hash as "missing", and the result depends on which repos happened to be materialized on that machine. A user reported this in Tinder/bazel-diff#506 (Bazel 9.2.0, RBE, --remote_download_minimal): same commit, identical target graph (200,599 targets), but 51,917 file-not-found warnings with the cache enabled vs. 85 without, and 3,884 target hashes changed with no source change. That turns into a large set of false-positive "affected" targets in CI.
We can (and will) make bazel-diff fail loudly instead of silently, but we cannot make it correct without help from Bazel:
- Turning the flag off for the bazel-diff invocation isn't practical. It's a startup option, so toggling it restarts the server and discards the analysis cache, and it forces the full repo downloads the flag exists to avoid.
- Building every external source file as a top-level target to force materialization is O(files) targets on the command line and downloads everything.
- Reading the repo's cached tree from the remote cache ourselves would mean reimplementing Bazel's repo cache key computation, which is internal and unstable.
Option 1 is the best fit: the tool never needs the bytes, only a stable content identity, and Bazel already has it in memory. It would also let these tools stop re-hashing external files that Bazel has already digested, even when the repo contents cache is off.
Which operating system are you running Bazel on?
Linux amd64 (reported); the behaviour is platform-independent
What is the output of bazel info release?
release 9.2.0
If bazel info release returns development version or (@non-git), tell us how you built Bazel.
No response
What's the output of git remote get-url origin; git rev-parse HEAD ?
Have you found anything relevant by searching the web?
Any other information, logs, or outputs that you want to share?
How bazel-diff would consume each option:
| Option |
bazel-diff change |
Bytes downloaded |
Hashes identical with flag on/off |
| 1. digests in query proto |
Pass the flag on the query we already run; mix SourceFile.digest + exec bit into the target hash instead of reading the file |
none |
yes, if digests are emitted for on-disk repos too |
| 2. materialize command |
Run it for the repos the user asked us to hash file-by-file, then read from output_base/external as today |
full repo |
yes |
| 3. per-repo tree digest |
Hash the tree digest per repo; per-file granularity would need our own CAS client |
none |
only at whole-repo granularity |
Notes on option 1:
- The digest function (
--digest_function) only needs to be stable for a given configuration; consumers mix the bytes in opaquely. Including the function name in the output would let tools detect a mismatch between two runs.
- Symlinks: reporting the digest of the resolved target (what an action would see) is what consumers want.
- Source directories could be left without a digest in a first version.
Description of the feature request:
With
--experimental_remote_repo_contents_cache, a cache-hit repo is injected into the in-memoryRemoteExternalFileSystemand only.bzl/REPO.bazelfiles are written under<output_base>/external/<repo>(RemoteExternalOverlayFileSystem#shouldPrefetch). Bazel itself and local actions read the rest lazily, but a tool running outside the Bazel server has no supported way to learn the contents of those files: they are not on disk, and no query output exposes them.I'd like one (ideally the first) of the following:
--proto:source_file_digests, that addsdigest(andis_executable) to theSourceFilemessage forquery/cquery --output=proto|streamed_proto|jsonproto. For in-memory repos Bazel already holds the digest from the cachedDirectorytree (RemoteExternalOverlayFileSystem#getDigest), so this costs no downloads. For on-disk files it could usegetFastDigestand fall back to hashing. Restricting it to external repos would be fine if main-repo cost is a concern.bazel fetch --repo=@@foo --materialize(name is a strawman), wired to the existingLazyMaterializer#ensureMaterialized. Todaybazel fetch --remote_download_alldoes not materialize; the only thing that works is building each source file as a top-level target (bazel build --remote_download_all @@repo//:path/to/file), which doesn't scale to tens of thousands of files.Directorydigest of an injected repo (e.g. inbazel mod show_repo --output=streamed_proto, or a file next to the marker file). A tool with remote cache access could then walk the CAS itself. This is the least convenient for tools but the smallest change.Independently of which option is chosen, a way to detect the situation would help: currently an external tool cannot distinguish "file is remote-backed and only in memory" from "file does not exist".
Which category does this issue belong to?
External Dependency
What underlying problem are you trying to solve with this feature?
I maintain bazel-diff, which computes a hash per target at two commits and diffs them to find the affected targets. The pattern (shared with target-determinator and similar tools) is:
bazel query/cquery --output=streamed_prototo get the target graph.SourceFiletarget, read the file from disk and hash its contents. For labels in external repos the path is$(bazel info output_base)/external/<canonical repo>/<pkg>/<file>.Step 2 silently breaks under the remote repo contents cache. The files aren't on disk, so they hash as "missing", and the result depends on which repos happened to be materialized on that machine. A user reported this in Tinder/bazel-diff#506 (Bazel 9.2.0, RBE,
--remote_download_minimal): same commit, identical target graph (200,599 targets), but 51,917 file-not-found warnings with the cache enabled vs. 85 without, and 3,884 target hashes changed with no source change. That turns into a large set of false-positive "affected" targets in CI.We can (and will) make bazel-diff fail loudly instead of silently, but we cannot make it correct without help from Bazel:
Option 1 is the best fit: the tool never needs the bytes, only a stable content identity, and Bazel already has it in memory. It would also let these tools stop re-hashing external files that Bazel has already digested, even when the repo contents cache is off.
Which operating system are you running Bazel on?
Linux amd64 (reported); the behaviour is platform-independent
What is the output of
bazel info release?release 9.2.0
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse HEAD?Have you found anything relevant by searching the web?
--experimental_remote_repo_contents_cacheTinder/bazel-diff#506: the downstream report with the numbers above.--experimental_remote_repo_contents_cacheunnecessarily materializes entire repos to local disk #31005: the opposite direction (repos being materialized too eagerly byreadFilein module extensions). A fix there makes fewer files reach disk, so external tools will see this more often, not less.ensureMaterializedBelow; external tools have no equivalent hook.Any other information, logs, or outputs that you want to share?
How bazel-diff would consume each option:
SourceFile.digest+ exec bit into the target hash instead of reading the fileoutput_base/externalas todayNotes on option 1:
--digest_function) only needs to be stable for a given configuration; consumers mix the bytes in opaquely. Including the function name in the output would let tools detect a mismatch between two runs.