Skip to content

Interrupted-OTA swap-move coverage: sim torn-write test and CI canary#2648

Open
neilberkman wants to merge 2 commits into
mcu-tools:mainfrom
neilberkman:tardigrade-ci
Open

Interrupted-OTA swap-move coverage: sim torn-write test and CI canary#2648
neilberkman wants to merge 2 commits into
mcu-tools:mainfrom
neilberkman:tardigrade-ci

Conversation

@neilberkman

@neilberkman neilberkman commented Mar 4, 2026

Copy link
Copy Markdown

MCUboot has had real interrupted-OTA regressions in swap-using-move revert handling where a power cut leaves the device stuck or booting the wrong image (#1966 / #2100, #2199). This PR adds coverage for that failure class at two layers.

Simulator: torn-write power-loss testing

The simulator's power-loss testing interrupts boot_go between whole flash operations: when the flash counter expires, flash_area_write in sim/mcuboot-sys/csupport/run.c returns via longjmp before calling the simulated flash write, so every simulated power cut is write-atomic. That is why the revert interruption sweep did not catch #1966: that failure needs power loss inside boot_write_magic during swap-move revert fixup, leaving a partially written magic that reads back as BOOT_MAGIC_BAD and matches no swap table row.

The sim commit adds a torn-write mode:

  • When the interruption counter expires in torn mode, the write programs all but the last byte of its payload before the simulated power loss. Write alignment is a driver interface requirement, not a power-fail atomicity guarantee, so the tear point falls inside the final write unit regardless of max-align.
  • The simulated flash marks torn bytes as retryable: one subsequent write of the same values is allowed, since recovery replays the interrupted operation. The write-once discipline stays enforced everywhere else, with unit tests for the tear, the single replay, a mismatched replay, and erase restoring normal behavior.
  • run_revert_with_torn_writes sweeps the revert path: perform a test upgrade, count the flash operations of an uninterrupted revert, then for each operation clone the post-upgrade state, tear that write during revert, and verify recovery to the original image with clean trailers. Gated to swap-using-move.

Validation: on current main, both the existing revert_with_fails sweep and the new torn-write sweep pass (--features swap-move, and swap-move,max-align-32). With the one-word fix from 4f39356 reverted, the new sweep fails at the fixup_revert magic write, including under max-align-32, while the existing whole-operation sweep still passes. That reproduces #1966 offline in the simulator.

CI: post-merge canary on real firmware

The workflow builds MCUboot swap-using-move for nrf52840dk/nrf52840 against Zephyr main, using the same container image, setup action, and SDK as zephyr_build.yaml, signs test images with the RSA-2048 test key, and runs two power-loss fault-injection sweeps under Renode via tardigrade, pinned by full commit SHA: one across the upgrade, one across the revert of a completed-but-unconfirmed upgrade. It runs post-merge on main when OTA-relevant paths change, weekly, and manually.

  • A run of this exact workflow is green on current main: 941 and 948 fault points, all passing (run).
  • Detection is A/B validated at the pinned harness revision: an exhaustive sweep against the commit before the bootutil: Fix device bricked after power failure during swap-move revert #2100 fix flags 3 power-loss points in the revert fixup where the device ends up stuck on the unconfirmed image (run); the fixed commit runs the same scenario clean (run).
  • The tardigrade action is a stateless composite action: it downloads Renode and Python dependencies, runs the audit CLI, and uploads a JSON report. No external service or telemetry.

The sim test is the primary regression coverage for this class; the canary is the narrower integration layer on top, exercising the real Zephyr-linked binary, image signing, and reset/recovery behavior for one representative configuration.

@neilberkman
neilberkman requested a review from d3zd3z as a code owner March 4, 2026 04:44
@de-nordic
de-nordic requested a review from davidvincze March 5, 2026 20:34
@de-nordic de-nordic added the CI label Mar 5, 2026
@neilberkman neilberkman changed the title ci: add tardigrade OTA resilience testing ci: add OTA resilience canary workflow Mar 5, 2026
@neilberkman neilberkman changed the title ci: add OTA resilience canary workflow ci: add swap-move OTA resilience canary Mar 6, 2026
@neilberkman
neilberkman force-pushed the tardigrade-ci branch 2 times, most recently from 8fa90d5 to 498516c Compare March 6, 2026 12:07
@neilberkman neilberkman changed the title ci: add swap-move OTA resilience canary ci: add interrupted-OTA swap-move canary Mar 6, 2026
@neilberkman
neilberkman force-pushed the tardigrade-ci branch 2 times, most recently from 0b5ea3e to ed2e613 Compare March 13, 2026 07:00

@d3zd3z d3zd3z left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like a good thing to test, although two questions:

  • Does it demonstrate a bug that needs to be fixed?
  • Is there something that could be added to the sim to demonstrate the failure in the offline simulator, which should be doing adversarial testing.
    The build failures likely can be resolved by rebasing onto the latest 'main' which fixes these.

@neilberkman
neilberkman force-pushed the tardigrade-ci branch 2 times, most recently from ccf139a to 1c5b986 Compare July 17, 2026 05:43
@neilberkman
neilberkman force-pushed the tardigrade-ci branch 7 times, most recently from dbfeaae to 951b421 Compare July 17, 2026 06:38
Two tardigrade sweeps on nrf52840dk/nrf52840, built against Zephyr
main with the same container image, setup action, and SDK as the
existing zephyr_build.yaml workflow: power loss across the v1->v2
swap-move upgrade, and power loss across the revert of a
completed-but-unconfirmed upgrade, the failure class of issue mcu-tools#1966
(PR mcu-tools#2100).

The tardigrade action is pinned by full commit SHA to a revision
whose MCUboot revert scenario handling is validated by a passing
profile sweep on that revision's own CI, using the same Renode
portable build source selected here.

Signed-off-by: Neil Berkman <neil@xuku.com>
The simulator's power-loss testing interrupts boot_go between whole
flash operations: when the flash counter expires, flash_area_write
aborts before writing anything. Every simulated power cut is therefore
write-atomic, so states left by a write torn partway through, such as
a partially written trailer magic reading back as BOOT_MAGIC_BAD, are
unreachable. That is why the revert interruption tests did not catch
issue mcu-tools#1966: power loss inside boot_write_magic during swap-move revert
fixup left a bad magic that matched no swap table row and bricked the
device.

Add a torn-write mode to the flash counter: on expiry the interrupted
write programs all but the last byte of its payload before the
simulated power loss.  Write alignment is a driver interface
requirement, not a power-fail atomicity guarantee, so the tear point
falls inside the final write unit regardless of max-align.  The
simulated flash marks torn bytes as retryable: one subsequent write of
the same values is allowed, since recovery replays the interrupted
operation, while the write-once discipline stays enforced everywhere
else.

Add run_revert_with_torn_writes, a revert-only sweep: perform a test
upgrade, count the flash operations of an uninterrupted revert, then
for each operation clone the post-upgrade state, tear that write
during revert, and verify the device recovers to the original image
with clean trailers.  Gated to swap-using-move.

With the fix from commit 4f39356 ("bootutil: Fix device brick after
power failure during swap-move revert") reverted, the new test fails
at the fixup_revert magic write, including under max-align-32, while
the existing whole-operation revert_with_fails sweep still passes; on
current code both pass.

Signed-off-by: Neil Berkman <neil@xuku.com>
@neilberkman neilberkman changed the title ci: add interrupted-OTA swap-move canary Interrupted-OTA swap-move coverage: sim torn-write test and CI canary Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants