Skip to content

deploy red: Rust 1.98 arm64 link arg broke the unpinned cargo-lambda cross-compile #20

Description

@MichaelTaylor3d

Outcome

on.dig.net's main tip was red on deploy, freezing its superproject pointer at 0.5.11. Fixed and
merged as on.dig.net#19 (v0.5.12).

Live service impact: NONE

Checked before deciding urgency, because a failed deploy is the one red of the three that could
mean a degraded live service. https://test.on.dig.net/ answers 404 from the resolver Lambda with
the resolver's own headers (X-Dig-URN, X-Dig-Store, X-Dig-Root in
access-control-expose-headers) — an unregistered-subdomain answer from live code, not an
infrastructure error. The apex on.dig.net has no DNS record by design; *.on.dig.net resolves to
its own CloudFront distribution.

The failed deploy left the live Lambda at the v0.5.10 build rather than breaking it. What had not
reached production is the v0.5.11 URN-parser conformance change (on.dig.net#18). §3.3 therefore did
not apply and no PAUSE.md was written.

Root cause

deploy failed at cargo lambda build --release --arm64:

error: unsupported linker arg: --fix-cortex-a53-843419
error: could not compile `on-dig-net-resolver` (bin "bootstrap")

Nothing in the repository changed. Two floating inputs crossed:

run date rustc cargo-lambda ziglang result
31422779156 08-10 1.97.1 1.9.1 0.16.0 success
32498562051 08-21 1.98.0 1.9.1 0.16.0 failure
32673740675 08-23 1.98.0 1.9.2 0.16.0 success

Rust 1.98.0 (2026-08-20) began emitting -Wl,--fix-cortex-a53-843419 in the link args for
aarch64-unknown-linux-gnu. Zig's linker rejects the flag. cargo-zigbuild filters it as of 0.23.0
(rust-cross/cargo-zigbuild#452); cargo-lambda picked that up in 1.9.2
(cargo-lambda/cargo-lambda#937, "fix: support Rust 1.98 ARM64 builds", published 2026-08-21).
cargo-lambda 1.9.1 bundles cargo-zigbuild 0.20.1, which does not filter it.

The workflow used toolchain: stable with an unpinned pip install cargo-lambda ziglang, so both
sides moved on their own schedules and the repository had no say in either.

The deeper defect, which was not the linker

The arm64 cross-compile existed in no PR gate. Every required check on f6bd45de was green on a
tip that could not deploy, so the break was discoverable only after merge, at deploy time, with the
pointer already frozen and the release commit already cut.

Fixed by adding a lambda-build CI job that runs exactly the build deploy.yml runs, minus AWS, and
adding it to branch protection's required contexts — an added gate that is not required would
not have blocked anything.

Note on the reproduction, recorded because it is the interesting part

The regression gate was pushed first, without the fix, and it passed. That is not the gate
failing: pip install cargo-lambda is unpinned and 1.9.2 had been published in the interim, so by
then the floating install already resolved to the fixed version.

That is the strongest argument for the pin. An unpinned gate and an unpinned deploy resolve
independently, so the gate can pass on one version while the deploy fails on another — precisely how
this reached production with every check green. Both steps are now pinned to
cargo-lambda==1.9.2 and ziglang==0.16.0, which is what makes the gate's green mean something
about the deploy.

Out of scope, reported not fixed

The resolver crate is on the chia 0.26 line (chia-protocol, chia-bls, clvm-* all 0.26.0)
while the ecosystem has moved to 0.36. §2.4b would normally have the touching PR bring those current,
but that is a type-identity change across the resolver's URN and puzzle-hash paths and did not belong
in a PR whose job was to unfreeze a red deploy. Needs its own lane.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions