feat: rewrite consensus metrics to not assume epoch monotonicity

[ksn6]

Problem and Summary of Changes

consensus_metrics.rs is currently written incorrectly.

We can't assume the invariant that maybe_new_epoch is invoked with a monotonically non-decreasing epoch, given that blocks from different forks may play out of order during replay.

To rectify, this PR introduces a map from epoch to metrics + updates the appropriate structure upon receiving a block.

When it comes to metrics emission, suppose that slot S is the highest possible slot in some epoch called S.epoch. We emit "end of epoch" metrics for S.epoch at the earlier of:

• We have a finalized slot in the following epoch, S.epoch + 1
• If S finalizes (if S doesn't finalize, then default to (1)).

[AshwinSekar]

i'm fine with the overall direction of this change. I'd like to give a chance for @akhi3030 to review this and the histogram change.

He's out this week, but given that we've already removed the problematic assert we should be good to wait until he gets back.

[akhi3030]

Thanks for catching this problem and for the fix. Generally looks good a few small comments though.

[akhi3030]

I don't remember seeing any code so far where we were able to copy the epoch schedule out of the bank. I guess I assumed that it is possible that the epoch schedule can change in future banks (and that is why it is stored in banks). In other words, just want to make sure that this is indeed safe and would the metrics container need access to sharable_banks instead?

[ksn6]

This should be safe - bank.rs sets its epoch schedule by cloning from GenesisConfig while children build their epoch schedules by cloning from their respective parents.

[akhi3030]

LGTM