AKS SPEC

[dvdplm]

Add a SPEC.md. Let's use this PR to work on the spec. Whether we eventually merge it or not is a question for later, it just seems like the least painful way to manage edits and conversations.

[ameba23]

@dvdplm Having thought about this a bit last night, here's my proposal for the storage replication. It might look overly-complicated but hear me out.

I think we don't need all nodes to store all users API keys. But we do need some duplication, and we need users to reliably be able to find them despite nodes coming and going over time.

My proposal is to take inspiration from kademlia and store secrets on nodes which are 'close to' the secret, using the XOR metric as a measure of 'closeness'.

All secrets have an associated lookup key (the user public key and URL of the service). We can hash this to get a secret ID in the same 'key space' as 'node ids' (which are just public keys / account IDs).

When a client wants to deploy a secret, they get the list of nodes from the chain, and choose the one which is 'closest' to the secret ID using the XOR metric.

On storing a user secret, the node also sends it to their 'neighbors', using the XOR metric to find the closest n nodes to themselves.

If we want n to be big (lots of replication), we can also have nodes propagate this further to their own neighbors until some threshold maximum 'distance' from the secret is reached.

When a new node joins, it requests existing secrets from it's n neighbors. More tricky to implement: when a node leaves, its n neighbors send keys to the new neighbor that now ranks higher in closeness than they did before.

When the user makes a request, they again choose the node closest to the secret ID to make the request to. This node is guaranteed to have their secret regardless of nodes joining or leaving (unless there are zero remaining nodes).

With a big network relative to n, different secrets are spread out in different 'places' in the key spaces. With a small network relative to n, all nodes have all users keys. I think this is worth considering, even if this will always be the case - because it gives us a way of knowing who to send keys to, rather than everyone sending to everyone (which is expensive on a large network).