100.000 Grains per minute

[raptisv]

Hello team, Orleans newbie here!

We have an application that accepts about 100.000 requests per minute. The need is to keep the request's state for about 30 seconds, so we have:

• 100.000 Grains per minute
• each Grain should live for 30 seconds
• there is no need for these Grains to communicate with each other

Assuming resources are unlimited, is Orleans a good fit for our case?

[JorgeCandeias]

Assuming resources are unlimited, a simple two-tier api with a database or redis can do this for you, if you can throw enough money at it. So are resources really unlimited, since you're looking at Orleans?

Activating a grain instance has a non-trivial cost on the system. After that, it becomes very fast to send messages to the grain instance, but the initial activation cost is not something to be overlooked, as the grain directory protocol has to manage and ensure the uniqueness of grain instances across the cluster. A low-resource platform can get overwhelmed if it gets bombarded with more activation requests than it can handle. The quantity of grain instances itself is not much of an issue but the activation rate can be.

Orleans can fulfill the distributed cache role just fine, however it's important to evaluate the system constrains in order to offer an appropriate design. For example, you could design the system so it shards grain instances by the request id itself, as you ask, or you could shard by the hash of the request id, meaning you could have e.g. a fixed 100 long-lived grain instances handling the incoming 100.000 requests per second. This would negate the pressure on the grain directory and keep the system snappy, while still allowing it to scale as more nodes are added. This is a similar sharding approach as you'd use on a redis cluster. This is just one alternative approach, there will likely be more ways to go about it with more information on the domain.

[raptisv]

Hello @JorgeCandeias, thank you for your answer.

So are resources really unlimited, since you're looking at Orleans?

Of course it's not, just wanted to get that parameter out of the way :)

The quantity of grain instances itself is not much of an issue but the activation rate can be.

So the "bottleneck" in this scenario would be the grain directory management.

I am trying to understand how the built-in distributed in-memory directory works exactly. Specifically, how other Silos get notified of the newly activated Grain. I can't find this detail on the documentation. Could you point me on some document or even a piece of code to help me understand?

[JorgeCandeias]

Short story, the other silos don't "get notified of the newly activated grain", they just figure out what exact silo is supposed to know where a certain grain instance is and ask about it as and when they need to.

Long story, the built-in grain directory is implemented as a distributed hash table, as the docs mention. You can find the runtime code here, and all that code helps the LocalGrainDirectory work. This DHT distributes the key lookup workload across nodes by assigning ownership to hash ranges of grain identities (their identity and addresses, not their hosting). Meaning, if you have three nodes, then each node is responsible for knowing where ~33% of the grain instances are hosted (and they can be hosted in some other silo). When node count changes, the ownership ranges are adjusted automatically, which does not affect current placing. In theory, the performance ceiling for the built-in grain directory will increase as your node count increases, assuming similar hardware between nodes. There is some cache magic here and here along with this to reduce double-hops for the most frequently used grain instances.

The important thing here, more than performance, is that the built-in grain directory is eventually consistent, meaning it is possible for duplicate activations to happen, if the cluster becomes unstable due to crashed nodes, faulty network, etc, since the ownership ranges will be inconsistent at that time. Orleans does, however, search for and deactivate duplicates (meaning grain instances that do not belong to the grain directory) as soon as it finds them. If immediate consistency is required, then you can use one of the external grain directories, though then the bottleneck becomes the external system itself.

[raptisv]

Thank you for the detailed explanation @JorgeCandeias. It takes some time to digest but at the end, the "magic" makes sense. Have a nice day :)