TSS storage

[dvdplm]

API Key Service – permanent storage

Status quo

This document does not describe how the Entropy network starts and initiates key shares. Here we assume that the blockchain is up and running and have a working set of secret key shares distributed among a set of TDX nodes, running as Confidential Virtual Machines (CVMs) on untrusted hosts. We further assume that the CVMs are attested, guaranteeing that all nodes can trust that all their peers run the same software.

The software composing the CVM is defined ahead of time as a raw disk image and contains a minimal linux build (x86). Beyond basic OS services like networking, the CVM runs entropy-aks.

The API Key Service ("AKS" below) has two jobs:

1. Manage storage of secret user data, e.g. API keys.
2. Execute remote API calls on behalf of users, using their secrets to authenticate the requests, and return the results to the users.

Secret data CRUD

The blockchain has privileged access (encrypted and authenticated) to a small set of RPC endpoints that allows validator nodes to send secrets, e.g. API keys, on user's behalf to the CVMs running the entropy-aks daemon ("AKS" for short). The secret data submitted by users is encrypted in-flight and opaque to validators and the on-chain transaction data cannot be decrypted/inspected. Secrets only ever reside inside the CVM, running in the TDX enclave.

The relevant RPC endpoints exposed by the AKS are:

• POST /secret?data=: adds a new secret; returns a secret_id.
• PUT /secret?id=&data=: updates the secret with secret_id id
• DELETE /secret?id: deletes a secret
• GET /make-request?secret_id=&remote-url=&verb=&payload=: instructs the AKS to make a remote API call to remote-url using the http verb verb and the secret matching the secret_id to send the payload in payload.

All RPC calls to the AKS from the outside are signed and contain counter measures to replay attacks (likely a block number and/or a random nonce). The API responses are encrypted to the end user's public key before they are send back outside the AKS.

NOTE: This document does not discuss the fact that end users still must do local key management and keep track of their secret_ids. That is an important matter worthy of its own separate discussion.

What do AKS nodes store?

To support the above scenario, the AKS nodes must keep the following data:

• User secrets
• User ACL, one for each remote rpc
• Users public keys
• List of current validators, i.e. the set of outside nodes allowed to do secrets CRUD and their public keys.
• Pending requests and responses.
• Additional bobs and bits, e.g. errors, info about next epoch, current time and current block number.

If each user stores 5 secrets of max 1kb each, with 1kb of ACL data, and 1kb of public key data, we have ~10Kb per user, so 1000 users would be ~10Mb. The rest of the data is negligible and/or ephemeral, so we can assume that the AKS needs to store ~10Mb of data per node during normal operations. When there are network problems and the AKS cannot reach the remote services, the storage needs for pending requests/responses could spike significantly, so at some point we need to look at dynamic rate limiting.

Secrets, ACLs, public keys and epoch info must be replicated to all AKS nodes so that they can all process user requests and ensure resiliance and availability.

Storage options

The amount of data stored is so small that the most performant implementation is to simply store everything in memory, and implement a periodical checkpointing mechanism to save the data to disk; the optimal point in time to do the checkpointing is when synchronizing the state with the other AKS nodes.

The form of the disk-based storage does not matter very much for the current conversation, as long as it is encrypted under a key known only to the AKS node itself. It could be a sqlite database or a bincode-encoded dump of the in-memory data tossed into a file and saved; it could also be a binary blob stored on the blockchain.

Without access to the current state the AKS node is not operational and the node will not accept RPCs at all until the state is available, checked to be valid and in sync with the other nodes.

Store data in an encrypted binary blob on the blockchain

At first glance this option is attractive because it leverages the well-understood properties of the blockchain to replicate and distribute data trustlessly.

The downside is that each AKS node would have to make a blockchain transaction at every block of the entire state. Even if the state size is small (~10Mb), when multiplied by the number of AKS nodes it can get unwieldy very quickly. There is also the matter of synchronizing the AKS operation with the cadence of block production, which is undesirable.

There are many more problems with this approach (e.g. how to reconcile everyone's updates?).

Rely on replication between AKS nodes

The AKS nodes will need a mechanism to ensure they all have the same secrets, ACLs etc so they will already be replicating most of the state between them (TODO: IS THERE AN ACTUAL PLAN FOR THIS WRITTEN DOWN?).
This mechanism relies on attestation that guarantees that all the nodes run the same software and know where to find each other.

If the assumptions above about the state size are correct, the replication mechanism wouldn't have to be very sophisticated. The state is small enough to be sent over the wire in a single message. New AKS nodes joining the network would need this anyway.

Store data in a file on disk

There is a scenario where data about in-flight operations becomes crucial, and having a local data store could be important. Consider a AKS node that has successfully executed a request to a remote service ("hey Coinbase, close my account and send all the coins to 0xDEADBEEF"), but the AKS node crashes before it can send the response back to the user.
In this case, the AKS node would need to be able to recover its state and send the response once it is back online. It would be awkward to include pending responses in the replicated state, as they are not relevant to other nodes and would only bloat the state unnecessarily.

Local, disk-based storage could be useful here.

The scenario described is less of a problem for outgoing pending requests. If the AKS node crashes after receiving the payload to forward, but before sending it, the user would see a timeout/disconnect error but their state would not be invalid and they could simply try again later (or with a different AKS node).

State encryption key(s)

So far we have established that the AKS nodes need to store a small amount of data somewhere, and that the data must be encrypted. The encryption key must not be known to the outside world, including the server operators and blockchain validators.

Furthermore, AKS nodes must be able to communicate among themselves so they can replicate the state. The communication between AKS nodes is based Noise + websockets. New nodes joining the network must have a way to get a copy of the complete state.

In conclusion we have the following requirements:

• A rebooting AKS node is not different from a new AKS node joining the network, modulo pending responses.
• Thanks to attestation and TDX-local Noise protocol encryption, AKS nodes have a secure channel among themselves, private to all external actors.
• It is likely useful to have a local disk-based storage in place in addition to replication, to store pending responses and other ephemeral data. The encryption key for this local data can be safely be the same for all AKS nodes and be included in the state replication message payload.

Attack scenarios

• A TDX compromise is catastrophic; the AKS node would be compromised and with it all user secrets. Game over.
• A dishonest server operator can probably pull off a successful rollback attack on local disk storage, replacing a file with an older version containing a state they like better. This is probably not catastrophic for the whole network, especially if data stored locally is limited to just pending responses and ephemeral (meta-)data.
• Eclipse and DoS attacks are trivial; defense here will rely on the blockchain noticing that AKS nodes are unavailable and having ways of replacing them. If the attacks are temporary this is probably not catastrophic for the network as a whole.
• Bugs in the AKS code can easily be catastrophic. An AKS node's job is akin to that of a browser: make arbitrary http requests and a bad bug in http response parsing could lead to data leaks. This scenario has nothing to do with data storage though.
• Network partitions, while certainly unpleasant, are probably not catastrophic. The AKS nodes will be able to continue operating and serving requests, but they will not be able to replicate the state with the rest of the network. This is a problem for the network as a whole, but not for individual AKS nodes and when the network heals, the AKS nodes can probably patch up the state for most of the data.

Unclear

• Replicate state between AKS nodes or not.
  • If state is not replicated, how can the service scale? When one AKS node is "full", how can it's secrets be sharded to other nodes?
  • Eclipse attacks become much more disruptive to users. Same for DoS and partitions: the service effectively goes down.
• Local persistent storage is either a useful extra or critical, depending on the decision wrt replication.
  • Local persistent storage is useless without at least replicating the encryption key across all/most AKS ndoes.
  • Is there any point in Shamir Secret Sharing (or other fancy crypto) of each AKS node's storage encryption key?
    • Yes, because it would allow the system to survive a corrupt TDX enclave. SGX teaches us that many attacks are possible only with physical access to the machine running the enclave, so even if TDX is broken, it's not a given that all AKS nodes will be compromised immediately.
    • No, not really: a corrupted TDX enclave can just pretend that "Hey fellow AKS nodes, evil AWS killed my disk and I had to reboot, can y'all send me the key again please?".
  • Do all AKS nodes need to store all the secrets?
    • Yes, because the service needs to be able to scale and the secrets need to be replicated. Also: what is even the point of the AKS service if it's not decentralized, censorship resistant and available?
    • No, because the service can scale by adding more AKS nodes and manually sharding the secrets among them. When nodes die or are upgraded, users have to manually re-upload their secrets.

[ameba23]

Naming

The blockchain has privileged access (encrypted and authenticated) to a small set of RPC endpoints that allows validator nodes to send secrets, e.g. API keys, on user's behalf to the CVMs running the entropy-tss service ("TSS" for short)

entropy-tss is a separate service in a separate CVM from the API key service (this repo) which deals with user secrets (API keys). This was the result of the innie-outtie discussion: we went with 'outtie' meaning a completely separate service.

Since this discussion is generally about the storage of user API keys, i think we should replace 'TSS' with 'API Key Service' in this doc.

There is however a similar discussion to be had about concern for the storage for entropy-tss (most importantly the network keyshares), which is what we ended up talking about a lot in the product - crypto - core discussion this week. But i think the requirements are a bit different and we should keep them as two different problems - which maybe end up having the same solution.

Replication of user secrets between instances of the API Key Service

The TSS nodes will need a mechanism to ensure they all have the same secrets, ACLs etc so they will already be replicating most of the state between them (TODO: IS THERE AN ACTUAL PLAN FOR THIS WRITTEN DOWN?).

No, there is not a plan for this, and it is also not written in the design spec that all nodes would store the same secrets.

So before trying to solve this problem, i would ask: 'do they all need the same secrets?' and 'how many nodes do we need?' or put differently 'does having my API key stored on many instances of the API key service improve security or user experience?'.

What is currently implemented:

• Anyone running an instance of the API Key Service can register it with the chain (if they can submit a valid attestation)
• It is possible for clients (users) to query the chain and find details of the available instances.
• Clients can upload API keys to an instance of their choosing, or several instances.

So for example i could choose to deploy my API key to just one instance, to two instances, or to all of them. There is an open issue for discussion on how to choose one.

If we decide there is a great benefit to having many nodes and having all nodes store all users API keys, then it is time to discuss how we replicate data between nodes. I expect there are some great solutions for doing that, but i think we need the 'why' before we get to the 'how'.

Getting to the 'why' - i guess it could be either:

• Because we haven't figured out how to have persistent strorage - meaning that if we figure this out, we wont need replication between nodes.
• Because even with persistent storage, the node could go down and so a single copy is not enough.

Also worth noting, there is currently no financial incentive for a random person run one of these (unlike entropy-tss). So if we were to launch with the current design, there would likely only be a small number of instances run by ourselves - which perhaps would work just fine.

Secure communication

The communication between TSS nodes is based on plain old TLS (terminated inside the TDX enclave itself)

This is not the case and currently something in discussion - TSS nodes run the protocols using the noise protocol over websockets, and for everything else (eg: communication with the user, and backing up / recovering each others database keys) static x25519/chacha20poly1305 over HTTP is used (which i personally would prefer to replace with TLS). But for the purpose of this doc lets just suppose we have secure communication between nodes and between nodes and users.

---

nit: First section, second paragraph - we are not using docker or arch linux for the CVM images - they are raw disk images with a minimal linux build. I wasn't sure whether to just go ahead and edit the issue.

[dvdplm]

@ameba23 I updated the doc based on your (excellent) comments, and added a new section at the bottom as a starting point for our convo tomorrow.

[ameba23]

Notes following the session

Some notes from what we discussed on Friday (in my understanding).

We have two storage-related problems:

1. No out-of-the-box secure persistent storage for CVMs. This effects both the API Key Service for storing user API keys as well as entropy-tss for storing network key-shares.
2. Redundancy of user's API keys. That is, should they be replicated across multiple instances of the API Key service, and if so how many.

Secure persistent storage

We discussed the current backup strategy used by entropy-tss, and looked at some alternatives:

1. David's proposal of having a single encryption key used by all instances, rather than a per-instance encryption key.
2. Using a service providing the SGX seal API as an additional key backup.
3. Using threshold encryption to recover encryption keys. A lot of ideas for this came up, but we saw it as something which needs a lot of research / design work, and ideally we want a solution sooner.

Redundancy of user's API keys

In our product / crypto / core discussion there was the assumption that we want to replicate user's API keys across all instances of the API Key Service. This is not (yet) written in the design spec.

This touches on the question of who will run these API Key Service instances and how are they incentivised. With the current design there is no financial incentive for external people to run these, (unlike entropy-tss) meaning there would likely only be a small number run by Entropy the organization.

We compared having a model with replication to having the client application deploy the API keys to a fixed number of instances. The former is more complicated to implement, and the latter is not resilient to nodes leaving the network.

We did not discuss so much the proposal of storing the API keys on-chain, but this relates to the question of whether all users API keys need to be accessible to all instances of the service. Once we have a clear decision on that in the design spec we can consider this in more detail.

[dvdplm]

Closing in favour of PR #27