Skip to content

Commit d55a0f4

Browse files
fix(nns): Drive node provider rewards mint to completion in golden state upgrade test (#10723)
## Problem `test_upgrade_canisters_with_golden_nns_state` was failing in CI with: ``` After upgrading and advancing time, node provider rewards timestamp did not increase. Before: 1781451541, After: 1781451541 ``` (panic at `rs/nns/integration_tests/src/upgrade_canisters_with_golden_nns_state.rs:556`) ## Root cause `advance_time_to_allow_for_voting_and_node_rewards` jumps the state machine clock ~1–2 months forward to trigger the monthly node provider rewards distribution, then ticks a **fixed 100 times** expecting the mint to have completed. That is not enough ticks: - The mint (in the governance heartbeat) can only validate once the **node-rewards canister has synced** its node metrics up to the requested period — its sync runs on an hourly timer, so it needs several ticks to fire and complete after the abrupt time jump. Until then the node-rewards canister rejects the request with `"Metrics and registry are not synced up to to_date"`. - Once it validates, the mint makes **one inter-canister call per day** in the reward period (which spans ~55 days here, because the golden state's last reward is well behind the test's execution date) plus **one ledger transfer per node provider**. When the mint does not finish within the 100-tick budget, the reward timestamp is never updated (the mint error is only logged, not fatal), and the `assert_increased` check on the node provider rewards timestamp fails. This gets worse the staler the pinned golden state is relative to the test's run date. ## Fix Replace the fixed `for _ in 0..100` tick loop with `tick_until_node_provider_rewards_distributed`, which ticks (1s/tick) and polls the reward timestamp until a new reward is actually distributed (timestamp increases past the previous one), up to a generous cap (`MAX_TICKS = 2000`). This makes the wait adaptive to how stale the golden state is, while still failing with the same informative assertion in `check_all` if no reward is ever distributed. Behavior for the passing case is preserved: the distribution is still triggered at the same moment; only the wait is now adaptive instead of a fixed count. ## Testing - `rustfmt` clean. - `bazel build //rs/nns/integration_tests:upgrade_canisters_with_golden_nns_state` passes. - The test itself is a `manual`/nightly golden-state test (requires downloading the golden NNS state) and was not run locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 44c49ca commit d55a0f4

1 file changed

Lines changed: 43 additions & 4 deletions

File tree

rs/nns/integration_tests/src/upgrade_canisters_with_golden_nns_state.rs

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -455,10 +455,17 @@ mod sanity_check {
455455
state_machine.advance_time(std::time::Duration::from_secs(
456456
seconds_to_node_provider_reward_distribution - 1,
457457
));
458-
for _ in 0..100 {
459-
state_machine.advance_time(std::time::Duration::from_secs(1));
460-
state_machine.tick();
461-
}
458+
459+
// Node provider rewards are minted by the governance heartbeat, but only once
460+
// the node-rewards canister has synced its node metrics up to the requested
461+
// period (its sync runs on an hourly timer) and the minting call has fully
462+
// completed. That call makes one inter-canister call per day in the reward
463+
// period and one ledger transfer per node provider, so the number of ticks it
464+
// needs grows with how far the golden state's last reward is behind the current
465+
// time. A fixed number of ticks is therefore not enough, so we tick until the
466+
// reward is actually distributed (i.e. its timestamp increases). If the cap is
467+
// hit without a new reward, the assertion in `check_all` surfaces the failure.
468+
tick_until_node_provider_rewards_distributed(state_machine, before_timestamp);
462469

463470
// Advance time in the state machine by one month to ensure that voting rewards
464471
// are also distributed.
@@ -471,6 +478,38 @@ mod sanity_check {
471478
}
472479
}
473480

481+
/// Ticks the state machine (advancing time by one second per tick) until a new
482+
/// monthly node provider reward has been distributed, i.e. its timestamp has
483+
/// increased past `before_timestamp`, up to a generous cap on the number of ticks.
484+
fn tick_until_node_provider_rewards_distributed(
485+
state_machine: &StateMachine,
486+
before_timestamp: u64,
487+
) {
488+
// Generous upper bound. Reaching the reward distribution requires the
489+
// node-rewards canister to sync (hourly timer) plus a minting call whose length
490+
// scales with the reward period and the number of node providers, so this needs
491+
// to comfortably exceed a few hundred rounds. Since each tick advances time by
492+
// one second, this keeps trying for one hour of simulated time, which is an
493+
// improbably long time for the distribution to take.
494+
const MAX_TICKS: usize = 3600;
495+
// Only poll periodically, since each poll is an ingress call to governance.
496+
const POLL_EVERY_TICKS: usize = 25;
497+
498+
for tick in 0..MAX_TICKS {
499+
state_machine.advance_time(std::time::Duration::from_secs(1));
500+
state_machine.tick();
501+
502+
if (tick + 1) % POLL_EVERY_TICKS == 0 {
503+
let timestamp = nns_get_most_recent_monthly_node_provider_rewards(state_machine)
504+
.unwrap()
505+
.timestamp;
506+
if timestamp > before_timestamp {
507+
return;
508+
}
509+
}
510+
}
511+
}
512+
474513
struct MetricsBeforeAndAfter {
475514
before: Metrics,
476515
after: Metrics,

0 commit comments

Comments
 (0)