Add compiler source download integrity values #1749
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@simuons @liucijus As always, I'm happy to split this into separate pull requests, or at least separate commits, if you'd prefer. Either way, the recommended breakdown would be to review the parts in this order:
Also, @WojciechMazur, if this goes through, |
simuons
approved these changes
Jul 23, 2025
Eliminates the "canonical reproducible form" warnings that often appear for users that enable compiler dependency tracking. Fixes bazel-contrib#1743. Emits a detailed error message if the configured Scala version is unknown to rules_scala, and isn't provided by a `scala_deps.compiler_srcjar` object. This message provides guidance on how to create a suitable `scala_deps.compiler_srcjar` object. Details: - Adds `scripts/update_compiler_sources_integrity.py` to generate `scala/private/macros/compiler_sources_integrity.bzl`. Refactored code from `scripts/update_protoc_integrity.py` into `scripts/lib/update_integrity.py` for reuse by the new script. - Adds `test/shell/test_compiler_sources_integrity.sh` and adds it to `test_rules_scala.sh`. Adds the `test/compiler_sources_integrity` repo for use by the new test suite. The test suite is fast due to the test repo only containing an alias to `@scala_compiler_sources//:src`. - Fixes a problem with the regex in `run_tests` from `test_helper.sh` that prevented the test skipping mechanism from working. --- This eliminates an annoying warning for users, while providing improved build integrity. Win/win! The refactoring of common code into `scripts/lib` eliminated a lot of potential duplication. It's also exposed a common pattern for generating download integrity files for different artifacts.
Fixes breakages in build: - https://buildkite.com/bazel/rules-scala-scala/builds/5652 The lint fix is for `test/compiler_sources_integrity/BUILD`. The unbound variable error on macOS was in `test/shell/test_compiler_sources_integrity.sh`, even though the variable should've been bound. The script sets `-euo pipefail` anyway. Took the missing config setup from `test/shell/test_bzlmod_macros.sh` and added it to `test/shell/test_compiler_sources_integrity.sh`.
The build failed again with an unbound variable error on macOS: - https://buildkite.com/bazel/rules-scala-scala/builds/5653#019775ba-c5ff-4bff-b877-4bb22d22ec9b ``` running test test_emit_no_canonical_reproducible_form_warning_for_default_version Log: .../test/shell/test_compiler_sources_integrity.sh: line 85: build_args[@]: unbound variable build failed ``` Today I learned that older versions of Bash consider empty arrays unset, hence the application of `${build_args[@]+"${build_args[@]}"}`: - https://stackoverflow.com/a/7577209
mbland
force-pushed
the
fix-1743-scala-compiler-source-warning
branch
from
54d92a8 to
0f8d986
Compare
_Always_ run `./test_lint.sh` first! (And then `bazel run //tools:lint_fix` if you have to.)
|
@simuons Thanks, done. Once this lands, I'll create another PR for a version bump. Also, the last_green build is failing again: INFO: Repo +scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20 defined by rule _jvm_import_external in @@//scala:scala_maven_import_external.bzl
ERROR: /workdir/scala/scala_maven_import_external.bzl:145:29: An error occurred during the fetch of repository '+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20':
Traceback (most recent call last):
File "/workdir/scala/scala_maven_import_external.bzl", line 145, column 29, in _jvm_import_external_impl
(repository_ctx.attr.visibility or
Error: unknown attribute visibility
INFO: Repo +scala_deps+scala_proto_rules_scalapb_protoc_bridge_2_12_20 defined by rule _jvm_import_external in @@//scala:scala_maven_import_external.bzl
INFO: Repo +scala_deps+scala_proto_rules_grpc_netty_2_12_20 defined by rule _jvm_import_external in @@//scala:scala_maven_import_external.bzl
ERROR: no such package '@@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20//': unknown attribute visibility
ERROR: /workdir/test/proto/custom_generator/BUILD.bazel:44:22: //test/proto/custom_generator:scalapb_worker_deps_provider depends on @@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20//:scala_proto_rules_scalapb_protoc_gen_2_12_20 in repository @@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20 which failed to fetch. no such package '@@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20//': unknown attribute visibility
ERROR: Analysis of target '//test/proto/custom_generator:scala_proto_deps_toolchain_def' failed; build aborted: Analysis failedPlease feel free to merge this PR before I fix this. If I don't get a chance to fix it in this PR, I'll fix it in the next one with the version bump. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Eliminates the "canonical reproducible form" warnings that often appear for users that enable compiler dependency tracking. Fixes #1743.
Emits a detailed error message if the configured Scala version is unknown to rules_scala, and isn't provided by a
scala_deps.compiler_srcjarobject. This message provides guidance on how to create a suitablescala_deps.compiler_srcjarobject.Details:
Adds
scripts/update_compiler_sources_integrity.pyto generatescala/private/macros/compiler_sources_integrity.bzl. Refactored code fromscripts/update_protoc_integrity.pyintoscripts/lib/update_integrity.pyfor reuse by the new script.Adds
test/shell/test_compiler_sources_integrity.shand adds it totest_rules_scala.sh. Adds thetest/compiler_sources_integrityrepo for use by the new test suite. The test suite is fast due to the test repo only containing an alias to@scala_compiler_sources//:src.Fixes a problem with the regex in
run_testsfromtest_helper.shthat prevented the test skipping mechanism from working.Motivation
This eliminates an annoying warning for users, while providing improved build integrity. Win/win!
The refactoring of common code into
scripts/libeliminated a lot of potential duplication. It's also exposed a common pattern for generating download integrity files for different artifacts.