Trust Quorum Protocol Implementation Tracking

[andrewjstone]

The trust quorum protocol is a sans-io protocol with messages sent between peer nodes. The Node API is expected to be called from sled-agent for things like key management (loading secrets) and trust quorum reconfiguration / status on instruction from Nexus. It's a pretty large protocol, and so tracking the pieces individually is probably useful compared to having one checkbox for the whole protocol in #8262.

All completed functionality includes testing, except where explicitly called out in a checkbox.

• Initial Configuration

  • Nexus request validation
  • Prepare Coordination
    • Creating and splitting rack secret
    • Creating configuration
    • Sending Prepare messages
    • Responding with PrepareAck messages
    • Handling Nexus poll requests for who has acked

• Coordinating Reconfiguration

  • Prepare Coordination (everything in initial configuration + below)
    • Collecting shares and recomputing prior rack secret
    • Encryption of prior rack secrets

• Commit Reconfiguration

  • Handling Commit at prepared nodes
  • PrepareAndCommit from Nexus at unprepared Nodes (nodes that were down/partitioned) during prepare coordination
  • CommitAdvance from peer nodes on stale requests
    • Computation of own key share by retrieving shares from up to date nodes
  • Node expungement (removal from trust quorum configuration)

• Loading Rack Secrets

  • RackSecretLoader functionality and tests
  • Node integration and cluster proptest support

• Fault injection / correction

  • Cluster proptest crash/restart support
  • Liveness testing (reconnect all nodes and finish a full round of configuration at end of cluster proptest)

• LRTQ backwards compatibility

  • Gathering LRTQ shares during Prepare Coordination
  • Nexus API for upgrade from LRTQ