Interrupted-OTA swap-move coverage: sim torn-write test and CI canary#2648
Open
neilberkman wants to merge 2 commits into
Open
Interrupted-OTA swap-move coverage: sim torn-write test and CI canary#2648neilberkman wants to merge 2 commits into
neilberkman wants to merge 2 commits into
Conversation
neilberkman
force-pushed
the
tardigrade-ci
branch
from
March 4, 2026 19:42
a467c2d to
0c5f179
Compare
neilberkman
force-pushed
the
tardigrade-ci
branch
from
March 5, 2026 21:55
6a97e6b to
d737a29
Compare
neilberkman
force-pushed
the
tardigrade-ci
branch
from
March 5, 2026 22:03
d737a29 to
baa1fac
Compare
neilberkman
force-pushed
the
tardigrade-ci
branch
2 times, most recently
from
March 6, 2026 12:07
8fa90d5 to
498516c
Compare
neilberkman
force-pushed
the
tardigrade-ci
branch
2 times, most recently
from
March 13, 2026 07:00
0b5ea3e to
ed2e613
Compare
d3zd3z
reviewed
Jul 16, 2026
d3zd3z
left a comment
Member
There was a problem hiding this comment.
This seems like a good thing to test, although two questions:
- Does it demonstrate a bug that needs to be fixed?
- Is there something that could be added to the sim to demonstrate the failure in the offline simulator, which should be doing adversarial testing.
The build failures likely can be resolved by rebasing onto the latest 'main' which fixes these.
neilberkman
force-pushed
the
tardigrade-ci
branch
2 times, most recently
from
July 17, 2026 05:43
ccf139a to
1c5b986
Compare
neilberkman
force-pushed
the
tardigrade-ci
branch
7 times, most recently
from
July 17, 2026 06:38
dbfeaae to
951b421
Compare
Two tardigrade sweeps on nrf52840dk/nrf52840, built against Zephyr main with the same container image, setup action, and SDK as the existing zephyr_build.yaml workflow: power loss across the v1->v2 swap-move upgrade, and power loss across the revert of a completed-but-unconfirmed upgrade, the failure class of issue mcu-tools#1966 (PR mcu-tools#2100). The tardigrade action is pinned by full commit SHA to a revision whose MCUboot revert scenario handling is validated by a passing profile sweep on that revision's own CI, using the same Renode portable build source selected here. Signed-off-by: Neil Berkman <neil@xuku.com>
neilberkman
force-pushed
the
tardigrade-ci
branch
from
July 17, 2026 06:44
951b421 to
4040137
Compare
The simulator's power-loss testing interrupts boot_go between whole flash operations: when the flash counter expires, flash_area_write aborts before writing anything. Every simulated power cut is therefore write-atomic, so states left by a write torn partway through, such as a partially written trailer magic reading back as BOOT_MAGIC_BAD, are unreachable. That is why the revert interruption tests did not catch issue mcu-tools#1966: power loss inside boot_write_magic during swap-move revert fixup left a bad magic that matched no swap table row and bricked the device. Add a torn-write mode to the flash counter: on expiry the interrupted write programs all but the last byte of its payload before the simulated power loss. Write alignment is a driver interface requirement, not a power-fail atomicity guarantee, so the tear point falls inside the final write unit regardless of max-align. The simulated flash marks torn bytes as retryable: one subsequent write of the same values is allowed, since recovery replays the interrupted operation, while the write-once discipline stays enforced everywhere else. Add run_revert_with_torn_writes, a revert-only sweep: perform a test upgrade, count the flash operations of an uninterrupted revert, then for each operation clone the post-upgrade state, tear that write during revert, and verify the device recovers to the original image with clean trailers. Gated to swap-using-move. With the fix from commit 4f39356 ("bootutil: Fix device brick after power failure during swap-move revert") reverted, the new test fails at the fixup_revert magic write, including under max-align-32, while the existing whole-operation revert_with_fails sweep still passes; on current code both pass. Signed-off-by: Neil Berkman <neil@xuku.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MCUboot has had real interrupted-OTA regressions in swap-using-move revert handling where a power cut leaves the device stuck or booting the wrong image (#1966 / #2100, #2199). This PR adds coverage for that failure class at two layers.
Simulator: torn-write power-loss testing
The simulator's power-loss testing interrupts
boot_gobetween whole flash operations: when the flash counter expires,flash_area_writeinsim/mcuboot-sys/csupport/run.creturns via longjmp before calling the simulated flash write, so every simulated power cut is write-atomic. That is why the revert interruption sweep did not catch #1966: that failure needs power loss insideboot_write_magicduring swap-move revert fixup, leaving a partially written magic that reads back asBOOT_MAGIC_BADand matches no swap table row.The sim commit adds a torn-write mode:
max-align.run_revert_with_torn_writessweeps the revert path: perform a test upgrade, count the flash operations of an uninterrupted revert, then for each operation clone the post-upgrade state, tear that write during revert, and verify recovery to the original image with clean trailers. Gated to swap-using-move.Validation: on current main, both the existing
revert_with_failssweep and the new torn-write sweep pass (--features swap-move, andswap-move,max-align-32). With the one-word fix from 4f39356 reverted, the new sweep fails at the fixup_revert magic write, including undermax-align-32, while the existing whole-operation sweep still passes. That reproduces #1966 offline in the simulator.CI: post-merge canary on real firmware
The workflow builds MCUboot swap-using-move for nrf52840dk/nrf52840 against Zephyr main, using the same container image, setup action, and SDK as
zephyr_build.yaml, signs test images with the RSA-2048 test key, and runs two power-loss fault-injection sweeps under Renode via tardigrade, pinned by full commit SHA: one across the upgrade, one across the revert of a completed-but-unconfirmed upgrade. It runs post-merge onmainwhen OTA-relevant paths change, weekly, and manually.The sim test is the primary regression coverage for this class; the canary is the narrower integration layer on top, exercising the real Zephyr-linked binary, image signing, and reset/recovery behavior for one representative configuration.