feat(system-tests): let the driver reach the local ic-gateway by name over verified HTTPS

[basvandijk]

Third PR in the stack. Builds on #11270 (the local ic-gateway serving a dev-CA
certificate on a resolvable domain) and #11265 (nested nodes registering through
it).

On the Local backend, a driver-side client could not reach the gateway the way it
does on Farm. Two workarounds covered for that — resolve_override_for_url for
the name, and root_certificate() + add_root_certificate for the certificate —
and both had to be repeated at every call site. This PR makes name resolution and
trust properties of the environment instead, so all of it goes away. Neither
nns_dapp_test nor the sdk asset helper now carries any backend-specific client
setup: they build a plain client and GET the URL.

1. The driver had no resolver at all

ensure_administrable_netns puts the driver in a network namespace it owns, but
not a mount namespace — so it kept reading the host's /etc/resolv.conf, whose
nameserver is unreachable from that netns. Driver-side DNS was not merely
overridden by the .resolve() callers; it was dead.

It now unshares CLONE_NEWNS as well and bind-mounts a generated resolv.conf
naming IPV6_NAME_SERVERS over /etc/resolv.conf, so the driver resolves
in-group names through the group's own dnsmasq exactly like the VMs do —
dnsmasq already binds those addresses on the group bridge inside that very
netns.

2. dnsmasq answered no wildcards

add_dns_record appends to the --addn-hosts file, which holds exact names
only. Farm's playnet DNS gives the gateway domain an apex record plus * and
*.raw CNAMEs, because clients address canisters as <canister id>.<domain>
and <canister id>.raw.<domain>. Locally those subdomains resolved nowhere —
for clients inside a VM as much as for the driver.

add_wildcard_dns_record answers for a name and every subdomain of it at any
depth: one --address option covering all three Farm records. It takes the whole
address set at once because a record, not an address, is what costs a restart.

A restart is what it costs, since --address is a command-line option and
nothing re-reads those (SIGHUP re-reads the hosts-shaped files and
--servers-file, and a servers-file admits nothing but server/rev-server).
So the records live in a file start_dnsmasq turns back into options,
create_group takes over truncating them, and stop_dnsmasq now waits for
dnsmasq to exit — it binds its listeners before daemonizing, so a restart
racing the old instance for UDP port 53 fails outright. Restarting is safe for
VMs already up: GuestOS nodes are statically configured and never consult the RA,
a SLAAC'd address outlives the gap by far, DHCPv4 leases live in the lease-file
dnsmasq re-reads at startup, and the only real loss is its DNS cache.

3. Trust was configured per client, not per environment

Every dev IC-OS image trusts the dev root CA, via update-ca-certificates in the
output_dev stage of the GuestOS and HostOS Dockerfiles. The driver did not, so
each of its clients had to add the CA to its own roots.

It now installs the CA once, at startup, by bind-mounting
/etc/ssl/certs/ca-certificates.crt with the CA appended over the original — in
the mount namespace section 1 already created. That path is load-bearing through
a chain nothing type-checks (ClientBuilder::build →
rustls_platform_verifier::Verifier → rustls_native_certs →
openssl_probe::probe), so the const carries it in a comment; rs/tests/BUILD.bazel
already builds a bundle at the same path for the colocated driver image for the
same reason.

It does not set SSL_CERT_FILE, which would be the wrong tool:
rustls-native-certs then loads only what that names and skips the platform store
entirely, so it would narrow the driver's trust rather than widen it, breaking
the clients that need public roots (Farm::new, and the log-upload client in
group.rs). std::env::set_var is also unsafe in edition 2024 and racy against
threads building clients, which a mount is not.

It does, however, honour the variable. system_ca_bundle resolves the bundle
the way rustls_native_certs::load_native_certs does — SSL_CERT_FILE when set,
openssl_probe's first Linux candidate otherwise — and the append, the mount and
the read-back all target that path, so the runner's own roots are kept rather than
replaced. An earlier revision refused to run when the variable was set, which
passed locally and failed every _local target on CI, where the Namespace runner
sets it. Hard-coding the default is safe because it is probe()'s first candidate:
if it exists, it is the one chosen, so the mount cannot land on a bundle the
verifier ignores.

The one shape that still refuses is SSL_CERT_DIR without SSL_CERT_FILE:
rustls-native-certs then reads only those directories and no bundle at all, so
there is nothing to append to — and a file cannot be added to a directory the
driver does not own. It fails loudly rather than quietly omitting the CA.

What the mount does not reach: a consumer using OpenSSL CApath alone, which
resolves <subject hash>.0 symlinks and never scans the bundle. Nothing in the
driver's process tree does that, and a plainly named file in the directory would
not have served it either — only a hash-named symlink would, which is what
update-ca-certificates generates and what a bind mount cannot create.

The append is verbatim-then-add, so it can only widen trust. The result is read
back before returning, because every step can succeed while leaving the CA
unreachable — a mount that did not take, a short write, a stale target — and none
of that would surface until a TLS handshake failed six minutes into a test.

Verification

rust-lint.sh, buildifier and bazel build //... --nobuild all clean, and:

target
//rs/tests/sdk:dfx_smoke_test_local	PASSED 486.8s
//rs/tests/nns:nns_dapp_test_local	PASSED 483.2s
//rs/tests/idx:basic_health_test_local	PASSED 368.6s
//rs/tests/sdk:dfx_smoke_test (Farm regression)	PASSED 153.0s
//rs/tests/nested:registration_local	PASSED 824.0s

The CI environment shape is covered too, reproduced locally with
--test_env=SSL_CERT_FILE=<a bundle outside /etc>: dfx_smoke_test_local (gateway)
and basic_health_test_local (no gateway — the majority case) both pass, the
shadowed file is left untouched on the host, and both pass again with the variable
unset.

The two gateway tests are the sharpest: each fetches
https://<canister id>.ic-gateway.ic.net/… with no resolve override and no
roots added, so they fail unless the mount, the wildcard, the trust store and the
certificate all hold. basic_health_test_local is the control — it stands up no
gateway, so it shows the unconditional trust-store install does not disturb the
~366 local targets that never touch one.

The logs confirm the chain rather than just a green tick: dnsmasq is started
exactly twice (create_group plus the one restart, with VMs already booted),
await_status_is_healthy of ic-gateway succeeds against the domain, and the
canister-subdomain asset fetch returns.

A few things were checked directly before writing the code, since each could have
sunk the approach:

• The bind mount works in a nested userns+mntns — the Bazel sandbox layer and
  then the driver's — with the host's files unaffected. _local tests
  deliberately run sandboxed with requires-network absent.
• --address=/d/a answers d, x.d and x.raw.d for both A and AAAA.
• glibc returns the AAAA when the paired A query comes back REFUSED, which is
  exactly what an IPv6-only wildcard produces — and the local gateway has no
  IPv4, so that is the real configuration rather than a corner case.
• The composed bundle verifies the dev CA and still verifies a public root, so
  the append really is additive.

Not done here

No danger_accept_invalid_certs call is removed. All 14 in rs/tests/ dial an
IC node or API boundary node — by IP literal, or by a domain registered with no
DNS server and pinned with .resolve() — or plain HTTP. A trust anchor cannot
help any of them, and none targets the gateway. The reasoning behind them moved
from root_certificate()'s doc to install_dev_root_ca's rather than being
deleted with the method. await_api_bn_healthy_async and status_async likewise
keep their .resolve() calls: an API BN's domain comes from its registry record
and is registered with no DNS server on either backend, and nothing in the tree
overrides uses_dns().

prometheus_vm writes <canister id>.raw.<domain> scrape targets, and those now
resolve — but the ledger-canister and bitcoin-*/dogecoin-* jobs are
scheme: https with no tls_config, unlike the neighbouring nested_hosts and
node_exporter jobs, so they would verify the dev-CA leaf against the Prometheus
UVM's stock trust store and fail. Finishing that means installing the CA there
too; those are mainnet canister ids that only exist on testnets, and it is a
separate, independently testable change.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Enables driver-side DNS resolution and wildcard gateway domains on the local system-test backend.

Changes:

• Installs an isolated resolver configuration in the driver’s mount namespace.
• Adds wildcard DNS records through managed dnsmasq restarts.
• Removes obsolete client-side resolution overrides.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
rs/tests/sdk/src/asset.rs	Removes gateway DNS overrides.
rs/tests/nns/nns_dapp_test.rs	Uses local wildcard DNS resolution.
rs/tests/driver/src/driver/local_backend.rs	Adds resolver mounting and wildcard DNS support.
rs/tests/driver/src/driver/ic_gateway_vm.rs	Registers wildcard gateway records and removes override APIs.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Suppressed comments (2)

rs/tests/driver/src/driver/local_backend.rs:486

• This intra-doc link also refers to the nonexistent SYSTEM_CA_BUNDLE constant; link to DEFAULT_SYSTEM_CA_BUNDLE instead.

    /// of [`SYSTEM_CA_BUNDLE`] with that CA appended over the original.

rs/tests/driver/src/driver/local_backend.rs:482

• When neither SSL variable is set, rustls-native-certs follows openssl_probe::probe() and uses the first existing CA file from a platform list; it does not always select this Debian path. On systems such as RHEL, this either fails startup because the file is absent or modifies a bundle that TLS clients never read. Resolve the bundle from openssl_probe::probe().cert_file (and report when no file is found) so the mounted file matches the verifier's choice.

        Ok(PathBuf::from(DEFAULT_SYSTEM_CA_BUNDLE))

[basvandijk]

placeholder

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

rs/tests/driver/src/driver/mod.rs:7

• Extracting the CA type into this private module leaves the existing [DevRootCa] intra-doc link in ic_gateway_vm.rs:286 unresolved: that module now imports only dev_root_ca, so the type is no longer in scope. Please update the link to the fully qualified crate::driver::dev_root_ca::DevRootCa path.

mod dev_root_ca;

[github-actions]

This pull request changes code owned by the Governance team. Therefore, make sure that
you have considered the following (for Governance-owned code):

1. Update unreleased_changelog.md (if there are behavior changes, even if they are
   non-breaking).

2. Are there BREAKING changes?

3. Is a data migration needed?

4. Security review?

How to Satisfy This Automatic Review

1. Go to the bottom of the pull request page.

2. Look for where it says this bot is requesting changes.

3. Click the three dots to the right.

4. Select "Dismiss review".

5. In the text entry box, respond to each of the numbered items in the previous
   section, declare one of the following:

• Done.

• $REASON_WHY_NO_NEED. E.g. for unreleased_changelog.md, "No
  canister behavior changes.", or for item 2, "Existing APIs
  behave as before.".

Brief Guide to "Externally Visible" Changes

"Externally visible behavior change" is very often due to some NEW canister API.

Changes to EXISTING APIs are more likely to be "breaking".

If these changes are breaking, make sure that clients know how to migrate, how to
maintain their continuity of operations.

If your changes are behind a feature flag, then, do NOT add entrie(s) to
unreleased_changelog.md in this PR! But rather, add entrie(s) later, in the PR
that enables these changes in production.

Reference(s)

For a more comprehensive checklist, see here.

GOVERNANCE_CHECKLIST_REMINDER_DEDUP

No canister changes.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.