-
-
Notifications
You must be signed in to change notification settings - Fork 286
Add compiler source download integrity values #1749
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add compiler source download integrity values #1749
Conversation
@simuons @liucijus As always, I'm happy to split this into separate pull requests, or at least separate commits, if you'd prefer. Either way, the recommended breakdown would be to review the parts in this order:
Also, @WojciechMazur, if this goes through, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, @mbland
Eliminates the "canonical reproducible form" warnings that often appear for users that enable compiler dependency tracking. Fixes bazel-contrib#1743. Emits a detailed error message if the configured Scala version is unknown to rules_scala, and isn't provided by a `scala_deps.compiler_srcjar` object. This message provides guidance on how to create a suitable `scala_deps.compiler_srcjar` object. Details: - Adds `scripts/update_compiler_sources_integrity.py` to generate `scala/private/macros/compiler_sources_integrity.bzl`. Refactored code from `scripts/update_protoc_integrity.py` into `scripts/lib/update_integrity.py` for reuse by the new script. - Adds `test/shell/test_compiler_sources_integrity.sh` and adds it to `test_rules_scala.sh`. Adds the `test/compiler_sources_integrity` repo for use by the new test suite. The test suite is fast due to the test repo only containing an alias to `@scala_compiler_sources//:src`. - Fixes a problem with the regex in `run_tests` from `test_helper.sh` that prevented the test skipping mechanism from working. --- This eliminates an annoying warning for users, while providing improved build integrity. Win/win! The refactoring of common code into `scripts/lib` eliminated a lot of potential duplication. It's also exposed a common pattern for generating download integrity files for different artifacts.
Fixes breakages in build: - https://buildkite.com/bazel/rules-scala-scala/builds/5652 The lint fix is for `test/compiler_sources_integrity/BUILD`. The unbound variable error on macOS was in `test/shell/test_compiler_sources_integrity.sh`, even though the variable should've been bound. The script sets `-euo pipefail` anyway. Took the missing config setup from `test/shell/test_bzlmod_macros.sh` and added it to `test/shell/test_compiler_sources_integrity.sh`.
The build failed again with an unbound variable error on macOS: - https://buildkite.com/bazel/rules-scala-scala/builds/5653#019775ba-c5ff-4bff-b877-4bb22d22ec9b ``` running test test_emit_no_canonical_reproducible_form_warning_for_default_version Log: .../test/shell/test_compiler_sources_integrity.sh: line 85: build_args[@]: unbound variable build failed ``` Today I learned that older versions of Bash consider empty arrays unset, hence the application of `${build_args[@]+"${build_args[@]}"}`: - https://stackoverflow.com/a/7577209
54d92a8
to
0f8d986
Compare
_Always_ run `./test_lint.sh` first! (And then `bazel run //tools:lint_fix` if you have to.)
@simuons Thanks, done. Once this lands, I'll create another PR for a version bump. Also, the last_green build is failing again: INFO: Repo +scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20 defined by rule _jvm_import_external in @@//scala:scala_maven_import_external.bzl
ERROR: /workdir/scala/scala_maven_import_external.bzl:145:29: An error occurred during the fetch of repository '+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20':
Traceback (most recent call last):
File "/workdir/scala/scala_maven_import_external.bzl", line 145, column 29, in _jvm_import_external_impl
(repository_ctx.attr.visibility or
Error: unknown attribute visibility
INFO: Repo +scala_deps+scala_proto_rules_scalapb_protoc_bridge_2_12_20 defined by rule _jvm_import_external in @@//scala:scala_maven_import_external.bzl
INFO: Repo +scala_deps+scala_proto_rules_grpc_netty_2_12_20 defined by rule _jvm_import_external in @@//scala:scala_maven_import_external.bzl
ERROR: no such package '@@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20//': unknown attribute visibility
ERROR: /workdir/test/proto/custom_generator/BUILD.bazel:44:22: //test/proto/custom_generator:scalapb_worker_deps_provider depends on @@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20//:scala_proto_rules_scalapb_protoc_gen_2_12_20 in repository @@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20 which failed to fetch. no such package '@@+scala_deps+scala_proto_rules_scalapb_protoc_gen_2_12_20//': unknown attribute visibility
ERROR: Analysis of target '//test/proto/custom_generator:scala_proto_deps_toolchain_def' failed; build aborted: Analysis failed Please feel free to merge this PR before I fix this. If I don't get a chance to fix it in this PR, I'll fix it in the next one with the version bump. |
Description
Eliminates the "canonical reproducible form" warnings that often appear for users that enable compiler dependency tracking. Fixes #1743.
Emits a detailed error message if the configured Scala version is unknown to rules_scala, and isn't provided by a
scala_deps.compiler_srcjar
object. This message provides guidance on how to create a suitablescala_deps.compiler_srcjar
object.Details:
Adds
scripts/update_compiler_sources_integrity.py
to generatescala/private/macros/compiler_sources_integrity.bzl
. Refactored code fromscripts/update_protoc_integrity.py
intoscripts/lib/update_integrity.py
for reuse by the new script.Adds
test/shell/test_compiler_sources_integrity.sh
and adds it totest_rules_scala.sh
. Adds thetest/compiler_sources_integrity
repo for use by the new test suite. The test suite is fast due to the test repo only containing an alias to@scala_compiler_sources//:src
.Fixes a problem with the regex in
run_tests
fromtest_helper.sh
that prevented the test skipping mechanism from working.Motivation
This eliminates an annoying warning for users, while providing improved build integrity. Win/win!
The refactoring of common code into
scripts/lib
eliminated a lot of potential duplication. It's also exposed a common pattern for generating download integrity files for different artifacts.