The Space Beaver project (from Pigs Can Fly Labs LLC) uses satellite service from swarm.space and SMTP to provide, what is politely called, value-conscious[1] off-grid messaging.[2] The initial draft of the space beaver’s core architecture was built using Scala and Akka, but then we switched to using Ray. By using Ray with Python instead of Akka with Scala we were able to re-use the object relation manager (ORM) from the website, and simplify the deployment. While it is possible to deploy Akka applications on Kubernetes, it is (in Holden’s opinion) substantially more complicated[3] than accomplishing the same task with Ray. In this appendix, we will walk through the general design of the Space Beaver backend, the code for the different actors, and show how to deploy it (and similar applications).
|
Note
|
The code for this case study is at https://github.com/PigsCanFlyLabs/message-backend-ray. |
SpaceBeaver’s core requirement is to serve as a bridge between e-mail (through SMTP), SMS (through bandwidth.com) and the swarm.space satellite APIs. Most of these involve some amount of state, such as running an SMTP server, but the outbound mail messages can be implemented without any state. A rough outline of the design is shown below in Actor Layout.
Now that you’ve seen a rough design, it’s time to explore how the different patterns you’ve learned throughout the book are applied to bring this all together.
The outbound mail client is the one stateless piece of code, since it establishes a connection for each outbound message. Since it’s stateless, we implemented this as a regular remote function, which is created for every incoming request. Ray can then scale up and or down as needed depending on the amount of incoming requests. Being able to scale the number of instances of the remote function containing the e-mail client, which is useful since the client may end up blocking on external hosts.
|
Tip
|
There is some overhead to scheduling each remote function invocation. In our case the expected message rate is not that high. If you have a good idea of what the desired concurrency is you should consider using Ray’s |
However, we want to serialize some settings, like in a settings class, so we wrap it with a special method to pass through a self-reference, despite not being an actor, as shown in Mail Client.
link:message-backend-ray/messaging/mailclient/mailclient.py[role=include]Another reasonable approach would be to make this stateful and maintain a connection across messages.
The remaining components of our system are stateful, either in the context of long-lived network connections or database connections. Since the user actor needs to talk to all of the other actors (and vice versa), to simplify discovering the other actors running in the system, we added a "LazyNamedActorPool"[4], which combines the concept of named actors along with actor pools.
link:message-backend-ray/messaging/utils/utils.py[role=include]The other shared pattern we use is graceful shutdown, where we ask the actors to stop processing new messages. Once the actors stop accepting new messages, the existing messages in the queue will drain out – either to the satellite network or SMTP network as needed. Then the actors can be deleted without having to persist and recover the messages the actor was processing. In the mail server, which we will look at next, this pattern is implemented as shown in Stop For Upgrade.
link:message-backend-ray/messaging/mailserver/mailserver_actor.py[role=include]The mail server actor is responsible for accepting new inbound messages and passing them along to the user actor. This is implemented as an aiosmtpd server handler, as shown in Mail Server Message Handling.
link:message-backend-ray/messaging/mailserver/mailserver_actor.py[role=include]An important part of having a mail server is that external users can make connections to the server. For HTTP services, like the inference server, you can use Ray serve to expose your service. However, the mailserver uses SMTP, which is not currently able to be exposed with Ray serve. So, to allow Kubernetes to route requests to the correct hosts, the mail actor tags itself as shown in Mail Server Kubernetes Labeling.
link:message-backend-ray/messaging/mailserver/mailserver_actor.py[role=include]The satellite actor is similar to the mail server actor, except instead of accepting inbound requests, it gets new messages by polling, and we also send messages through it. Polling is like driving with a six-year-old in the car who keeps asking, "are we there yet?" Except in our case, it is "do you have any new messages?" In Ray, async actors are the best option to implement polling, as the polling loop runs forever, but you still want to be able to process other messages. The satellite actors' polling implementation is shown below in Satellite Actor Polling.
link:message-backend-ray/messaging/satellite/satellite.py[role=include]This polling loop delegates most of the logic out to check_msgs shown in Satellite Check Messages.
link:message-backend-ray/messaging/satellite/satellite.py[role=include]Another interesting pattern we used in the satellite actor is exposing serializable results for testing, but keeping the data in the more efficient async representation in the normal flow. This pattern is shown in the way messages are decoded in Satellite Process Message.
link:message-backend-ray/messaging/satellite/satellite.py[role=include]While the other actors are all async, allowing parallelism within the actor, the user actors are synchronous since the Django Object Relation Mapper (ORM) does not yet handle async. The user actor code is shown relatively completely in User Actor below, so you can see the shared patterns (which were used in the other actors but skipped for brevity).
link:message-backend-ray/messaging/users/user_actor.py[role=include]|
Note
|
Django is a popular Python web development framework, that includes many different components including the ORM we are using. |
To facilitate testing, the actor code was broken into a "base" class and then extended into an actor class. This allowed for testing the mail server independent from its deployment on Ray, as shown in Standalone Mail Test.
link:message-backend-ray/messaging/mailserver/mailserver_tests.py[role=include]While these standalone tests can run with less overhead, it’s a good idea to have some full actor, and you can often speed them up by re-using the Ray context across tests (although when it goes wrong, the debugging is painful), as in Full actor test.
link:message-backend-ray/messaging/mailserver/mailserver_actor_tests.py[role=include]While Ray handles most of the deployment, we need to create a Kubernetes service to make our SMTP and SMS services reachable. On our test cluster, we do this by exposing a load balancer service as shown in SMTP and SMS Services.
link:message-backend-ray/service.yaml[role=include]As shown above, the SMTP and SMS services use different node selectors to route the requests to the correct pods.
The Ray port of the SpaceBeaver messaging backend substantially reduced deployment and packaging complexity while increasing code reuse. Some of this comes from the broad Python ecosystem (popular frontend tools and backend tools), but much of the rest comes from the serverless nature of Ray. The equivalent system in Akka required user intention around scheduling actors, whereas, with Ray, we can leave that up to the scheduler. Of course, there are many benefits to Akka, like the powerful JVM ecosystem, but hopefully, this case study has shown you some interesting ways you can use Ray.