|
| 1 | +--- |
| 2 | +title: "Node Infrastructure" |
| 3 | +description: "How ICP nodes are structured: the IC-OS operating system stack, virtual machine isolation, and Trusted Execution Environments." |
| 4 | +--- |
| 5 | + |
| 6 | +Every node in the Internet Computer network runs **IC-OS**: a custom operating system stack based on Ubuntu Linux and designed specifically for ICP. IC-OS provides a consistent, secure execution environment across all nodes regardless of the underlying hardware, which is a prerequisite for the deterministic execution that consensus requires. |
| 7 | + |
| 8 | +## IC-OS: three operating systems in one |
| 9 | + |
| 10 | +IC-OS is not a single operating system but a layered stack of three systems, each with a distinct role. |
| 11 | + |
| 12 | +### SetupOS |
| 13 | + |
| 14 | +SetupOS is used once: when initializing a new node for the first time. A [node provider](../references/glossary.md#node-provider) boots from a USB drive containing SetupOS, which automatically: |
| 15 | + |
| 16 | +- Verifies that the hardware meets ICP node requirements |
| 17 | +- Tests network connectivity |
| 18 | +- Installs HostOS and GuestOS onto the machine |
| 19 | +- Configures the node with its identity and initial cryptographic keys |
| 20 | + |
| 21 | +After setup completes, the machine reboots into HostOS. SetupOS is not used again unless the node needs to be re-provisioned from scratch. |
| 22 | + |
| 23 | +### HostOS |
| 24 | + |
| 25 | +HostOS runs directly on the physical hardware. Its sole purpose is to configure and run the GuestOS virtual machine. It: |
| 26 | + |
| 27 | +- Launches the GuestOS virtual machine |
| 28 | +- Manages hardware resource allocation |
| 29 | +- Handles GuestOS upgrades pushed by the [Network Nervous System (NNS)](../references/glossary.md#network-nervous-system-nns) |
| 30 | +- Provides a security boundary between the physical hardware and the ICP software stack |
| 31 | + |
| 32 | +HostOS is intentionally minimal. It treats the GuestOS as an untrusted process running in a virtual machine, which limits what a compromised GuestOS can do to the host and what the host can do to the guest. |
| 33 | + |
| 34 | +### GuestOS |
| 35 | + |
| 36 | +GuestOS runs inside a virtual machine on top of HostOS. This is where the ICP software actually executes. GuestOS: |
| 37 | + |
| 38 | +- Runs the [replica](../references/glossary.md#replica) process and the orchestrator (implementing the four-layer protocol stack) |
| 39 | +- Executes canisters and manages their state |
| 40 | +- Participates in consensus with other nodes in the subnet |
| 41 | +- Manages cryptographic key material and threshold signature operations |
| 42 | + |
| 43 | +Running GuestOS in a virtual machine ensures every node presents the same software environment to the replica, regardless of the underlying hardware. It also enables the Trusted Execution Environment (TEE) protection described below. |
| 44 | + |
| 45 | +## Trusted Execution Environments |
| 46 | + |
| 47 | +Running the GuestOS inside a virtual machine provides logical isolation from the host, but a sophisticated attacker with physical access to a node could historically inspect or tamper with GuestOS memory by compromising the HostOS or hypervisor. |
| 48 | + |
| 49 | +Trusted Execution Environments (TEEs) address this by enforcing hardware-level isolation between a virtual machine and its host. Even if the HostOS or hypervisor is compromised, the confidentiality and integrity of GuestOS memory and state are preserved. TEE-enabled nodes are being rolled out across the network as hardware is upgraded. |
| 50 | + |
| 51 | +ICP uses AMD's **Secure Encrypted Virtualization with Secure Nested Paging (SEV-SNP)** as its TEE technology. SEV-SNP provides four capabilities that together make it possible to trust a GuestOS running on a potentially compromised host: |
| 52 | + |
| 53 | +1. **Memory encryption**: protection of GuestOS memory from unauthorized reads or writes by the host |
| 54 | +2. **VM launch measurements**: cryptographic fingerprints that capture how the VM was initialized |
| 55 | +3. **Attestation reports**: verifiable evidence that a VM is running inside a genuine SEV-SNP TEE with a specific configuration |
| 56 | +4. **Sealing keys**: hardware-derived keys that allow data to be securely encrypted for persistent storage |
| 57 | + |
| 58 | + |
| 59 | + |
| 60 | +### Memory encryption |
| 61 | + |
| 62 | +SEV-SNP encrypts all memory pages of the GuestOS virtual machine using keys protected by the CPU's secure processor. A host that gains full control of the machine can only read encrypted blobs from the GuestOS memory: canister state, cryptographic key shares, and other sensitive runtime data remain confidential. |
| 63 | + |
| 64 | +### VM launch measurements |
| 65 | + |
| 66 | +A VM launch measurement is a cryptographic fingerprint of the GuestOS at the moment it starts. The SEV-SNP secure processor computes this measurement from the CPU model and firmware, the guest kernel, the initial ramdisk, and the kernel command-line parameters. Any single-byte change to the GuestOS software or configuration produces a different measurement. |
| 67 | + |
| 68 | +The kernel command-line parameters included in the measurement contain, among other things, the expected hash of the root filesystem, which is verified during early boot. Any modification to the GuestOS (whether in code, configuration, or filesystem contents) therefore leads to a different launch measurement. |
| 69 | + |
| 70 | +For each GuestOS release, the expected launch measurement can be computed in advance and published as part of the release. Nodes running the same GuestOS version produce identical measurements, which provides a basis for verifying that a node is running approved software. |
| 71 | + |
| 72 | +### Attestation reports |
| 73 | + |
| 74 | +An attestation report is a signed document produced by the SEV-SNP secure processor. It contains the VM's launch measurement and the CPU's unique hardware identifier, signed by AMD's root of trust. This gives any verifier (whether another node or an external party) the ability to confirm that: |
| 75 | + |
| 76 | +- The VM is running inside a genuine SEV-SNP TEE |
| 77 | +- The specific software and configuration that were loaded match an approved GuestOS release |
| 78 | + |
| 79 | + |
| 80 | + |
| 81 | +ICP uses attestation in two ways: |
| 82 | + |
| 83 | +- **Node-to-node attestation.** Before sensitive data or secrets are shared between nodes, SEV-SNP-enabled nodes attest each other. This is integral to the upgrade process (see below) and will be extended to all network connections as SEV-SNP adoption expands: each node pair attests the other at connection establishment, ensuring secrets are only exchanged with verified nodes. |
| 84 | +- **External attestation.** SEV-SNP-equipped nodes expose a dedicated attestation endpoint for external verification. Access is restricted by firewall rules and is only available through API boundary nodes. External parties indirectly attest individual nodes through these API boundary nodes, which in turn verify the nodes they communicate with. |
| 85 | + |
| 86 | +### Sealing keys |
| 87 | + |
| 88 | +A sealing key is derived from two inputs: the CPU's unique hardware identifier and the VM's launch measurement. This means: |
| 89 | + |
| 90 | +- Each node produces a unique sealing key, even for the same GuestOS version |
| 91 | +- If the GuestOS changes (for example after an upgrade), the derived key changes and previously encrypted data becomes inaccessible until a secure key handoff completes |
| 92 | + |
| 93 | +ICP uses sealing keys to encrypt the GuestOS disk partitions that contain sensitive runtime data. This ensures that even if an attacker copies the disk to another machine, the data cannot be decrypted: the sealing key depends on the specific CPU and the exact GuestOS configuration. |
| 94 | + |
| 95 | +## Disk encryption |
| 96 | + |
| 97 | +### Partition layout |
| 98 | + |
| 99 | +Each node maintains two partition sets (A and B). This dual layout allows a new GuestOS version to be prepared in the inactive set while the current version continues running, and enables rollback if an upgrade fails. |
| 100 | + |
| 101 | +| Partition | Notes | |
| 102 | +|---|---| |
| 103 | +| EFI | | |
| 104 | +| GRUB | | |
| 105 | +| config | | |
| 106 | +| boot (A) | | |
| 107 | +| root (A) | | |
| 108 | +| **var (A)** | Encrypted; key derived from VM A's launch measurement | |
| 109 | +| boot (B) | | |
| 110 | +| root (B) | | |
| 111 | +| **var (B)** | Encrypted; key derived from VM B's launch measurement | |
| 112 | +| **store** | Encrypted; two keys, one per VM measurement | |
| 113 | + |
| 114 | +Only partitions holding sensitive data are encrypted. The `var` partitions contain runtime data private to the currently active GuestOS. The `store` partition holds persistent data shared across GuestOS versions. System partitions (`boot`, `root`, `config`) are not encrypted: their contents are not confidential, and root filesystem integrity is covered by the root hash embedded in the kernel command-line, which is part of the VM launch measurement. |
| 115 | + |
| 116 | +### From traditional disk encryption to sealing-key-based encryption |
| 117 | + |
| 118 | +ICP nodes have always used disk encryption for data partitions. However, the previous encryption keys were independent of the GuestOS and could in principle be accessed by a malicious GuestOS, leaving a potential attack vector for a highly skilled adversary who could compromise the GuestOS and read the encrypted data. |
| 119 | + |
| 120 | +With SEV-SNP, LUKS passphrases for each encrypted partition are now derived from the SEV-SNP sealing key using HKDF, giving each partition a unique passphrase tied to both the CPU and the exact GuestOS version. This means only the GuestOS that encrypted a partition can decrypt it, and any change in GuestOS version or hardware prevents access to previously encrypted data. |
| 121 | + |
| 122 | + |
| 123 | + |
| 124 | +On reboot, the GuestOS requests the sealing key from the SEV-SNP secure processor. As long as the launch measurement has not changed, the same sealing key is returned, allowing the node to decrypt the partitions. If the launch measurement changes (for example after an upgrade), a different sealing key is generated and the encrypted partitions can no longer be accessed. This is where the upgrade process and remote attestation come in. |
| 125 | + |
| 126 | +## GuestOS upgrades |
| 127 | + |
| 128 | +When a new GuestOS is approved by the NNS, its attributes (root filesystem hash and launch measurement) are published to the NNS registry, which serves as the source of truth for valid GuestOS versions. A malicious GuestOS cannot participate because it will have no entry in the registry. |
| 129 | + |
| 130 | +The upgrade then runs the old and new GuestOS instances in parallel: |
| 131 | + |
| 132 | +1. A proposal to upgrade a subnet or set of nodes is submitted and approved by the ICP community. |
| 133 | +2. The new GuestOS image is downloaded into the inactive partition set while the current GuestOS continues running. |
| 134 | +3. A temporary **Upgrade VM** boots the new GuestOS. It cannot yet access the encrypted `store` or `var` partitions because its sealing key (derived from the new launch measurement) differs from the current one. |
| 135 | +4. The Upgrade VM generates an attestation report containing its launch measurement and sends it to the old GuestOS over a TLS channel. |
| 136 | +5. The old GuestOS verifies the attestation report against the NNS registry to confirm the new GuestOS is an approved release. |
| 137 | +6. Once verified, the old GuestOS shares the disk encryption key with the Upgrade VM. The Upgrade VM re-encrypts the partitions with a key derived from its own sealing key. |
| 138 | +7. Both VMs shut down. The node boots into the upgraded GuestOS, which can now access the data using its own derived key. |
| 139 | + |
| 140 | +This process ensures that disk access transfers only to a verified, NNS-approved GuestOS version, and repeats for every future upgrade. |
| 141 | + |
| 142 | +## Emergency recovery |
| 143 | + |
| 144 | +TEE-enabled GuestOSes are designed to lock everyone out (including node operators) unless a specific, governance-gated recovery process is followed. Recovery is never automatic and always requires an NNS proposal approved by the community. Historically, emergency recoveries have occurred only a few times, and during 2025 not a single one was necessary. |
| 145 | + |
| 146 | +### Manual rollback |
| 147 | + |
| 148 | +Manual rollback is the first option when a node fails after an upgrade. The dual partition layout means the previous GuestOS version still resides on the inactive partition set. |
| 149 | + |
| 150 | +1. The recovery coordinator submits a proposal to the NNS marking the problematic GuestOS version as broken. If approved, nodes refuse to upgrade to that version again even if the subnet record still references it. |
| 151 | +2. Node providers switch the active partition set back to the previous version via the HostOS limited console, without touching the GuestOS or breaking TEE guarantees. |
| 152 | +3. The previous GuestOS boots and the node resumes normal operation. Once a fixed GuestOS version is released and approved, nodes upgrade to it. |
| 153 | + |
| 154 | +### Recovery-GuestOS |
| 155 | + |
| 156 | +When neither partition set boots, manual rollback is insufficient. The encrypted partitions can only be decrypted by a GuestOS with the original launch measurement, so no other GuestOS version can access the data, including a fixed one. |
| 157 | + |
| 158 | +The Internet Computer solves this with a Recovery-GuestOS: a specially crafted image that keeps the same kernel, initramdisk, and kernel command-line as the broken GuestOS (preserving the launch measurement) while replacing the root filesystem with a fixed version. The table below shows how this differs from a standard upgrade image: |
| 159 | + |
| 160 | +| | Upgrade image | Recovery image | |
| 161 | +|---|---|---| |
| 162 | +| Can be reproduced and verified by the community | yes | yes | |
| 163 | +| kernel, initrd, kernel command-line | arbitrary | same as in base image | |
| 164 | +| Root filesystem hash matches `root_hash` kernel parameter | yes | no | |
| 165 | +| Boot partition contains NNS proposal with root filesystem hash | no | yes | |
| 166 | + |
| 167 | +Because the root hash in the kernel command-line no longer matches the recovery root filesystem, a special override is needed: the `BlessAlternativeGuestOsVersion` NNS proposal. During early boot, if the actual root hash does not match the expected hash in the kernel command-line, the integrity checker looks for this proposal. If present, valid, and listing the specific node's chip ID, the recovery root filesystem is mounted while preserving the original launch measurement, and therefore the same disk encryption key. |
| 168 | + |
| 169 | +The full process: |
| 170 | + |
| 171 | +1. The recovery coordinator collects the affected nodes' chip IDs and the base GuestOS launch measurement. |
| 172 | +2. A Recovery-GuestOS branch is prepared in the Internet Computer repository. |
| 173 | +3. A recovery root filesystem is created, and a `BlessAlternativeGuestOsVersion` proposal is submitted to the NNS with the recovery root filesystem hash, base launch measurement, and list of authorized chip IDs. |
| 174 | +4. Once approved, a Recovery-GuestOS upgrade image is built combining the base kernel, initramdisk, kernel command-line, the recovery rootfs, and the signed proposal. |
| 175 | +5. Node operators deploy it via the HostOS limited console. |
| 176 | +6. During early boot, the integrity checker detects the root hash mismatch, verifies the NNS proposal, confirms the node's measurement and chip ID match, and mounts the recovery root filesystem. |
| 177 | +7. The Recovery-GuestOS boots and the node resumes operation, with SEV-SNP privacy guarantees intact. |
| 178 | + |
| 179 | +Because the integrity checker is part of the initramdisk, a malicious actor cannot tamper with it without changing the SEV-SNP launch measurement, preserving the security of the node. |
| 180 | + |
| 181 | +## Further reading |
| 182 | + |
| 183 | +- [Protocol Stack](protocol/index.md): the four-layer architecture (peer-to-peer, consensus, message routing, execution) that runs inside GuestOS |
| 184 | +- [Glossary: replica](../references/glossary.md#replica): the replica process that implements the protocol stack |
| 185 | + |
| 186 | +<!-- Upstream: informed by Learn Hub articles "Overview" (Node Infrastructure), "Trusted Execution Environments" (migrated, source retired) --> |
0 commit comments