chore: evaluate Remote Build Execution (RBE) @ Namespace by basvandijk · Pull Request #10579 · dfinity/ic

basvandijk · 2026-06-26T12:28:15Z

Why

To reduce reliance on DFINITY's own Data Centers and to make developing, building and testing the IC more portable we're planning to switch the majority of CI to run on Namespace, a CI provider who DFINITY already uses for ARM64 builds on Linux and MacOS.

In particular we're planning to move the bazel-test-all job (AMD64 bazel build and test) to Namespace. This includes system-tests. Since Namespace does not have access to Farm the system-tests on Namespace will use the Local system-test backend, i.e. each test will spawn its QEMU VMs locally on the worker running the bazel action.

Considering that local system-tests are resource heavy, i.e. they can easily consume 10s of CPUs, they shouldn't be executed on a single worker because that would effectively sequentialise all system-tests which would consume almost 4 hours.

We therefor configure bazel to run its actions on a Remote Build Execution (RBE) cluster @ Namespace. This way builds and tests distribute over many workers and run in parallel. 15 workers for now of shape 32x64 that each can accept 16 action slots.

Before switching to RBE we first need to evaluate its performance and stability. That's this PR.

What

To compare our existing bazel-test-all with RBE we need to have a copy of that job that is as similar as possible with the only difference that it runs bazel on RBE @ Namespace. To this end:

ci-kickoff.yml is extended with a ci-rbe-evaluation job (like ci-main) which calls the new ci-rbe-evaluation.yml workflow.
ci-rbe-evaluation.yml is a slimmed down copy of ci-main.yml with similar jobs like config, infer-bazel-targets and bazel-test-all-rbe. The infer-bazel-targets job will behave the same as in ci-main except that it excludes Farm-based system-tests; the reverse is true for infer-bazel-targets in ci-main which excludes local system-tests.
bazel-test-all-rbe is similar to bazel-test-all except that it runs on a namespace-profile-rbe-driver-amd64-linux-16x32 runner and configures bazel to use Namespace's RBE cluster.
Note that this job doesn't use the same container image as bazel-test-all but instead uses a special optimised variant stored at Namespace's Container Registry (nscr.io).
container-autobuild.yml‎ has been extended with a rbe-worker-image job which generates this special variant. It takes the ic-build container image, mirrors it to nscr.io and optimises it for RBE. The existing update-image-references job will then make sure references to this image are updated in the code .

Evaluation

Since now both bazel-test-all and bazel-test-all-rbe will run on each update to each PR, for each push to master and for each Release Testing run we can compare the difference in performance, stability and debugability.

Note that bazel-test-all-rbe is configured with continue-on-error: true to ensure it doesn't block PRs or releases in case it fails.

Regarding debugability, bazel-test-all-rbe doesn't upload its BEP data to our internal BuildBuddy. Instead it uploads logs of failed runs to Namespace's Artifact Storage from which users can download them.

We'll run with this for a short while until we're confident we can switch to RBE @ Namespace completely.

Unresolved Questions

There are a currently 94 system-tests that are currently configured as Farm-only (backend = "farm") we need to decide how to handle these when we switch to Namespace completely.

Copilot

Pull request overview

This PR introduces an experimental GitHub Actions path to run bazel test using Namespace Bazel Remote Execution (BRE), including automation to build/mirror/optimize a worker image and keep the workflow pinned to an immutable digest.

Changes:

Adds a new experimental workflow to run bazel test on Namespace runners with remote execution enabled.
Extends the container autobuild workflow with jobs to mirror ic-build into nscr.io, optimize it for BRE, and automatically update the pinned worker-image digest used by the new workflow.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
.github/workflows/container-autobuild.yml	Adds jobs to create an optimized Namespace BRE worker image and auto-update the pinned digest reference in workflows.
.github/workflows/bre-namespace-test.yml	New opt-in workflow to run `bazel test` using Namespace remote execution with a pinned worker image.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

github-actions · 2026-06-28T12:19:57Z

Run URL: https://github.com/dfinity/ic/actions/runs/30406537698

New container images with tag: d82f82fd3da23633ea37dee630d0eff9814d04bfc8e46f6e9a1ebd61765d3472
ic-build: sha256:dd11ad77c455bd491169af8c271a8557cdcca6e7f163a39ebf75aedea0214bf2
ic-dev: sha256:82bb8cbc8ba0788687ca2c4654bf381e4d6bda839ad2583c2e25dfae1bacb120

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

docker buildx imagetools inspect --format {{.Manifest.Digest}} returned the full manifest dump (not just the digest) for the multi-platform attestation index that docker/build-push-action emits, producing a garbage worker_image ref. Parse the digest from the nsc base-image upload "Uploaded base image:" line instead, and fail fast if it cannot be determined.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Bazel declares test.xml as an output of every test action. When a test doesn't write $XML_OUTPUT_FILE itself, StandaloneTestStrategy runs a *second* spawn (@bazel_tools//tools/test:test_xml_generator) to derive it from test.log. Under RBE that spawn has to queue for a free worker to do well under a second of work: in run 30158073133 every one of the 152 executed system tests paid a median 435s of queueing for it (p90 1726s). Convert the SystemGroupSummary into a JUnit XML document and write it to $XML_OUTPUT_FILE at the end of the parent process. Bazel only runs the generator when the test didn't produce test.xml itself, so writing it ourselves skips that spawn entirely. Also stop the test-driver e2e scenarios from inheriting XML_OUTPUT_FILE: they spawn the driver as a child process, and those children would each overwrite the test.xml that Bazel expects //rs/tests/idx:test_e2e_scenarios itself to produce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…it XML Every <testcase> was emitted with status="run", including the skipped ones, which contradicts the <skipped/> child element they carry and can confuse consumers that aggregate on the attribute. Derive it from the outcome instead, following GoogleTest's convention that Bazel's own test_xml_generator also emits: "run" for a case that executed, "notrun" for one that didn't. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Addresses review comments on #10908: - Validate that `cpus_oversubscription_factor` and `cpus` are ints >= 1 and `fail()` with a clear message otherwise. Previously a 0 factor produced an obscure division-by-zero at analysis time, a negative one silently produced a nonsense reservation, and a float produced a non-integral `cpu` exec_property like "16.0". - Compute the ceiling with `(cpus + factor - 1) // factor` instead of `-(-cpus // factor)`; now that both are known-positive ints, this is the clearer idiom. Verified the two formulas agree, and that all 369 `_local` targets keep byte-identical `cpu` exec_properties. - Reword the `cpus_oversubscription_factor` docstring, which was grammatically unclear ("factor to divide the requested cpus with which"). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…e-bazel-remote-execution

nmattia

Really exciting! couple questions related to the GHA side of things to start with

…e-bazel-remote-execution

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:7d8e6aa9eae77e13e91ed30835c0fa72e05e86d5067c6d0dd14905e8c53efc36 ic-dev: sha256:06d7680338792de19f456577f37614d344e3feb119b843f535311ec5bdd66eb6 ic-build-worker: nscr.io/c9ptjuknd7oc6/ic-build-worker@sha256:28668b9bb053f840fb12354545ef4a2434cb41b7a7dfcb594afc6cb6686e4c17

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:3e3ee32a5de1fde724d4ab931cd5868dad52fef823c41331494a5b36742a6907 ic-dev: sha256:ad31e83146e452c3b6c31ecbb25c1ccfda79ecd10ec16e8dedb1f6952f1dfd6c ic-build-worker: nscr.io/c9ptjuknd7oc6/ic-build-worker@sha256:145163a97a493d19798422b6b87cabb5b64d8eac50318a14395d6a25d3796439

…e-bazel-remote-execution

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:bead3d28ce7ce4c626e3a6c547cd6fdcbe97e5e8e2f41f719d1fab9726f2bfa3 ic-dev: sha256:15a7b1857cf46812924711a30e18e4eaeebcc8ed7d43c14bf709612a1c2fe6df

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:4aff6230641fb23dbe917e68fd559185164265da34079b531856445f467c2ed4 ic-dev: sha256:0dbf26178f3a0d466fa03c0cb3dd53dab0eadd3b0859caf36f7bca93a855d949

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:dd11ad77c455bd491169af8c271a8557cdcca6e7f163a39ebf75aedea0214bf2 ic-dev: sha256:82bb8cbc8ba0788687ca2c4654bf381e4d6bda839ad2583c2e25dfae1bacb120

…it fails

basvandijk requested a review from Copilot June 26, 2026 12:28

basvandijk changed the title ~~Experiment with Bazel Remote Execution (BRE) on Namespace~~ chore: experiment with Bazel Remote Execution (BRE) on Namespace Jun 26, 2026

github-actions Bot added the chore label Jun 26, 2026

Copilot started reviewing on behalf of basvandijk June 26, 2026 12:28 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread .github/workflows/bre-namespace-test.yml Outdated

basvandijk requested a review from Copilot June 26, 2026 12:33

Copilot started reviewing on behalf of basvandijk June 26, 2026 12:34 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread .github/workflows/container-autobuild.yml Outdated

Comment thread .github/workflows/container-autobuild.yml Outdated

Comment thread .github/workflows/container-autobuild.yml Outdated

Comment thread .github/workflows/bre-namespace-test.yml Outdated

basvandijk requested a review from Copilot June 26, 2026 12:42

Copilot started reviewing on behalf of basvandijk June 26, 2026 12:43 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

basvandijk force-pushed the basvandijk/namespace-bazel-remote-execution branch from 27f3f10 to 95ee442 Compare June 28, 2026 12:15

pr-creation-bot-dfinity-ic Bot temporarily deployed to CREATE_PR June 28, 2026 12:20 Inactive

basvandijk temporarily deployed to CREATE_PR June 28, 2026 12:54 — with GitHub Actions Inactive

basvandijk requested a review from Copilot June 28, 2026 12:54

Copilot started reviewing on behalf of basvandijk June 28, 2026 12:54 View session

basvandijk temporarily deployed to CREATE_PR June 28, 2026 12:56 — with GitHub Actions Inactive

Copilot AI reviewed Jun 28, 2026

View reviewed changes

Comment thread .github/workflows/bre-namespace-test.yml Outdated

Comment thread .github/workflows/bre-namespace-test.yml Outdated

Comment thread .github/workflows/container-autobuild.yml Outdated

basvandijk temporarily deployed to CREATE_PR June 28, 2026 13:10 — with GitHub Actions Inactive

basvandijk requested a review from Copilot June 28, 2026 13:10

Copilot started reviewing on behalf of basvandijk June 28, 2026 13:10 View session

Copilot AI reviewed Jun 28, 2026

View reviewed changes

Comment thread ci/container/TAG Outdated

Comment thread .github/workflows/container-autobuild.yml Outdated

Comment thread .github/workflows/bre-namespace-test.yml Outdated

basvandijk temporarily deployed to CREATE_PR June 28, 2026 13:24 — with GitHub Actions Inactive

basvandijk requested a review from Copilot June 28, 2026 13:24

Copilot started reviewing on behalf of basvandijk June 28, 2026 13:25 View session

Copilot AI reviewed Jun 28, 2026

View reviewed changes

Comment thread .github/workflows/bre-namespace-test.yml

Comment thread .github/workflows/container-autobuild.yml Outdated

Potential fix for pull request finding

dd8e7fb

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

basvandijk temporarily deployed to CREATE_PR June 28, 2026 15:14 — with GitHub Actions Inactive

basvandijk and others added 6 commits July 26, 2026 11:22

fix: don't mix up bazel-targets and bazel-rbe-targets

f716137

oversubscribe cpus for local system-tests

eb4aca2

Merge remote-tracking branch 'origin/master' into basvandijk/namespac…

e86f88c

…e-bazel-remote-execution

nmattia reviewed Jul 27, 2026

View reviewed changes

basvandijk and others added 9 commits July 27, 2026 14:47

Merge remote-tracking branch 'origin/master' into basvandijk/namespac…

ef97445

…e-bazel-remote-execution

trigger container build

c374339

use tee

5f4c444

remove || github.ref_name

f87d0c2

fix

2c9ff05

trigger container rebuild

b143eb3

Merge branch 'master' into basvandijk/namespace-bazel-remote-execution

cbabdf1

basvandijk mentioned this pull request Jul 28, 2026

fix: build e2fsprogs with the hermetic cc toolchain #10931

Merged

basvandijk and others added 13 commits July 28, 2026 22:02

Merge remote-tracking branch 'origin/master' into basvandijk/namespac…

cdd8bc2

…e-bazel-remote-execution

trigger container rebuild

bdbbb19

Simplify rbe-worker-image job

7ea2f9d

simplify

470eff0

Updating container images to tag: d82f82fd3da23633ea37dee630d0eff9814…

d9ba0ba

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:bead3d28ce7ce4c626e3a6c547cd6fdcbe97e5e8e2f41f719d1fab9726f2bfa3 ic-dev: sha256:15a7b1857cf46812924711a30e18e4eaeebcc8ed7d43c14bf709612a1c2fe6df

fix

7550fc9

trigger

83e2018

Updating container images to tag: d82f82fd3da23633ea37dee630d0eff9814…

fe7ad4f

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:4aff6230641fb23dbe917e68fd559185164265da34079b531856445f467c2ed4 ic-dev: sha256:0dbf26178f3a0d466fa03c0cb3dd53dab0eadd3b0859caf36f7bca93a855d949

fix bazel/hermetic_cc_toolchain_cache_dir.patch

a98d704

fix

9460d94

trigger

d422ede

Updating container images to tag: d82f82fd3da23633ea37dee630d0eff9814…

c43ea6a

…d04bfc8e46f6e9a1ebd61765d3472 ic-build: sha256:dd11ad77c455bd491169af8c271a8557cdcca6e7f163a39ebf75aedea0214bf2 ic-dev: sha256:82bb8cbc8ba0788687ca2c4654bf381e4d6bda839ad2583c2e25dfae1bacb120

remove continue-on-error: true such that the workflow turns red when …

6c41243

…it fails

Uh oh!

Conversation

basvandijk commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Evaluation

Unresolved Questions

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

github-actions Bot commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

nmattia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

basvandijk commented Jun 26, 2026 •

edited

Loading

github-actions Bot commented Jun 28, 2026 •

edited

Loading