Skip to content

Commit 625eaca

Browse files
ryanofskyryanofsky
committed
Merge #229: Design Documentation Update
cc234be Design doc update (Novo) Pull request description: This PR contains an attempt to improve the design documentation to help new contributors to the repo. It adds more details about `BuildField`, `ReadField` and `PassField`. It explains how Callbacks, ThreadMaps and Async processing work. ACKs for top commit: ryanofsky: Code review ACK cc234be. Thanks for the updates! All the descriptions seem accurate now, and hopefully the diagrams make it clear how the different classes fit together Tree-SHA512: dacb43be97a643b5bd98b21133497940492a82761ab5820c70c937e2c057aec1dbb16c5bc7b12a0eab372c7f9a65c10af3123fc989e649c5b6eaae26ec610227
2 parents a4f9296 + cc234be commit 625eaca

File tree

1 file changed

+185
-2
lines changed

1 file changed

+185
-2
lines changed

‎doc/design.md‎

Lines changed: 185 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ There is also optional support for thread mapping, so each thread making interpr
1313

1414
Libmultiprocess acts as a pure wrapper or layer over the underlying protocol. Clients and servers written in other languages, but using a shared capnproto schema can communicate with interprocess counterparties using libmultiprocess without having to use libmultiprocess themselves or having to know about the implementation details of libmultiprocess.
1515

16-
### Internals
16+
## Core Architecture
1717

1818
The `ProxyClient` and `ProxyServer` generated classes are not directly exposed to the user, as described in [usage.md](usage.md). Instead, they wrap C++ interfaces and appear to the user as pointers to an interface. They are first instantiated when calling `ConnectStream` and `ServeStream` respectively for creating the `InitInterface`. These methods establish connections through sockets, internally creating `Connection` objects wrapping a `capnp::RpcSystem` configured for client and server mode respectively.
1919

@@ -25,7 +25,190 @@ When a generated method on the `ProxyClient` is called, it calls `clientInvoke`
2525

2626
On the server side, the `capnp::RpcSystem` receives the capnp request and invokes the corresponding C++ method through the corresponding `ProxyServer` and the heavily templated `serverInvoke` triggering a `ServerCall`. The return values from the actual C++ methods are copied into capnp responses by `ServerRet` and exceptions are caught and copied by `ServerExcept`. The two are connected through `ServerField`. The main method driving execution of a request is `PassField`, which is invoked through `ServerField`. Instantiated interfaces, or capabilities in capnp speak, are tracked and owned by the server's `capnp::RpcSystem`.
2727

28-
## Interface descriptions
28+
## Request and Response Flow
29+
30+
Method parameters and return values are serialized using Cap'n Proto's Builder objects (for sending) and Reader objects (for receiving). Input parameters flow from the client to the server, while output parameters (return values) flow back from the server to the client.
31+
32+
```mermaid
33+
sequenceDiagram
34+
participant clientInvoke
35+
participant BuildField as BuildField<br/>(Client)
36+
participant ReadField_C as ReadField<br/>(Client)
37+
participant Request as Request<br/>message
38+
participant serverInvoke
39+
participant ReadField as ReadField<br/>(Server)
40+
participant BuildField_S as BuildField<br/>(Server)
41+
participant Response as Response<br/>message
42+
43+
Note over clientInvoke,ReadField: Input Parameter Flow
44+
clientInvoke->>BuildField: BuildField(input_arg)
45+
BuildField->>Request: Serialize input
46+
Request->>serverInvoke: Cap'n Proto message
47+
serverInvoke->>ReadField: Deserialize input
48+
49+
Note over clientInvoke,Response: Output Parameter Flow
50+
serverInvoke-->>BuildField_S: BuildField(output)
51+
BuildField_S-->Response: Serialize output
52+
Response-->>ReadField_C: Cap'n Proto message
53+
ReadField_C-->>clientInvoke: Deserialize output
54+
```
55+
56+
### Detailed Serialization Mechanism
57+
58+
Parameters are represented as Fields that must be set on Cap'n Proto Builder objects (for sending) and read from Reader objects (for receiving).
59+
60+
#### Building Fields
61+
62+
`BuildField` uses a generated parameter `Accessor` to set the appropriate field in the Cap'n Proto Builder object.
63+
64+
```mermaid
65+
sequenceDiagram
66+
participant clientInvoke as clientInvoke or<br/>serverInvoke
67+
participant BuildField
68+
participant Accessor
69+
participant Builder as Params::Builder
70+
71+
Note over clientInvoke,Builder: Serializing Parameters
72+
clientInvoke->>BuildField: BuildField(param1)
73+
BuildField->>Accessor: Use generated field accessor
74+
Accessor->>Builder: builder.setField1(param1)
75+
76+
clientInvoke->>BuildField: BuildField(param2)
77+
BuildField->>Accessor: Use generated field Accessor
78+
Accessor->>Builder: builder.setField2(param2)
79+
```
80+
81+
#### Reading Fields
82+
83+
`ReadField` uses a generated parameter `Accessor` to read the appropriate field from the Cap'n Proto Reader object and reconstruct C++ parameters.
84+
85+
```mermaid
86+
sequenceDiagram
87+
participant serverInvoke as clientInvoke or<br/>serverInvoke
88+
participant ReadField
89+
participant Accessor
90+
participant Reader as Params::Reader
91+
participant ServerCall
92+
93+
Note over serverInvoke,ServerCall: Deserializing Parameters
94+
serverInvoke->>ReadField: Read param1
95+
ReadField->>Accessor: Use generated field accessor
96+
Accessor->>Reader: reader.getField1()
97+
Reader-->>ServerCall: call function with param1
98+
```
99+
100+
## Server-Side Request Processing
101+
102+
The generated server code uses a Russian nesting doll structure to process method fields. Each `ServerField` wraps another `ServerField` (for the next parameter), or wraps `ServerRet` (for the return value), which finally wraps `ServerCall` (which invokes the actual C++ method).
103+
104+
Each `ServerField` invokes `PassField`, which:
105+
1. Calls `ReadField` to deserialize the parameter from the `Params::Reader`
106+
2. Calls the next nested layer's `invoke()` with the accumulated parameters
107+
3. Calls `BuildField` to serialize the parameter back if it's an output parameter
108+
109+
`ServerRet` invokes the next layer (typically `ServerCall`), stores the result, and calls `BuildField` to serialize it into the `Results::Builder`.
110+
111+
`ServerCall` uses the generated `ProxyMethod<MethodParams>::impl` pointer-to-member to invoke the actual C++ method on the wrapped implementation object.
112+
113+
```mermaid
114+
sequenceDiagram
115+
participant serverInvoke
116+
participant SF1 as ServerField<br/>(param 1)
117+
participant SF2 as ServerField<br/>(param 2)
118+
participant SR as ServerRet<br/>(return value)
119+
participant SC as ServerCall
120+
participant PMT as ProxyMethodTraits
121+
participant Impl as Actual C++ Method
122+
123+
serverInvoke->>SF1: SF1::invoke
124+
SF1->>SF2: SF2::invoke
125+
SF2->>SR: SR::invoke
126+
SR->>SC: SC::invoke
127+
SC->>PMT: PMT::invoke
128+
PMT->>Impl: Call impl method
129+
Impl->>PMT: return
130+
PMT->>SC: return
131+
SC->>SR: return
132+
SR->>SF2: return
133+
SF2->>SF1: return
134+
SF1->>serverInvoke: return
135+
```
136+
137+
## Advanced Features
138+
139+
### Callbacks
140+
141+
Callbacks (passed as `std::function` arguments) are intercepted by `CustomBuildField` and converted into Cap'n Proto capabilities that can be invoked across process boundaries. On the receiving end, `CustomReadField` intercepts the capability and constructs a `ProxyCallFn` object with an `operator()` that sends function calls back over the socket to invoke the original callback.
142+
143+
```mermaid
144+
sequenceDiagram
145+
participant CT as Client Thread
146+
participant C as clientInvoke
147+
participant CBF1 as CustomBuildField (Client)
148+
participant S as Socket
149+
participant CRF1 as CustomReadField (Server)
150+
participant Srv as Server Code
151+
participant PCF as ProxyCallFn
152+
153+
C->>CBF1: send function parameter
154+
CBF1->>S: creates a Server for the function and sends a capability
155+
S->>CRF1: receives a capability and creates ProxyCallFn
156+
CRF1->>Srv:
157+
Srv->>PCF: call the callback
158+
PCF-->>CT: sends request to Client
159+
```
160+
161+
### Thread Mapping
162+
163+
Thread mapping enables each client thread to have a dedicated server thread processing its requests, preserving thread-local state and allowing recursive mutex usage across process boundaries.
164+
165+
Thread mapping is initialized by defining an interface method with a `ThreadMap` parameter and/or response. The example below adds `ThreadMap` to the `construct` method because libmultiprocess calls the `construct` method automatically.
166+
167+
```capnp
168+
interface InitInterface $Proxy.wrap("Init") {
169+
construct @0 (threadMap: Proxy.ThreadMap) -> (threadMap :Proxy.ThreadMap);
170+
}
171+
```
172+
173+
- **ThreadMap in parameter**: The client's `CustomBuildField` creates a `ThreadMap::Server` capability and sends it to the server, where `CustomReadField` stores the `ThreadMap::Client` in `connection.m_thread_map`
174+
- **ThreadMap in response**: The server's `CustomBuildField` creates a `ThreadMap::Server` capability and sends it to the client, where `CustomReadField` stores the `ThreadMap::Client` in `connection.m_thread_map`
175+
176+
You can specify ThreadMap in the parameter only, response only, or both:
177+
- **Parameter only**: Server can create threads on the client
178+
- **Response only**: Client can create threads on the server
179+
- **Both (as shown)**: Bidirectional thread creation
180+
181+
When both parameter and response include ThreadMap, both processes end up with `ThreadMap::Client` capabilities pointing to each other's `ThreadMap::Server`, allowing both sides to create threads on the other process.
182+
183+
### Async Processing with Context
184+
185+
By adding a `Context` parameter to a method in the capnp interface file, you enable async processing where the client tells the server to execute the request in a separate worker thread. For example:
186+
187+
```capnp
188+
processData @5 (context :Proxy.Context, data :Data) -> (result :Result);
189+
```
190+
191+
If a method does not have a `Context` parameter, then libmultiprocess will execute IPC requests invoking that method on the I/O event loop thread. This is fine for fast and non-blocking methods, but should be avoided for any methods that are slow or blocking or make any IPC calls(including callbacks to the client), since as long as the method is executing, the Cap'n Proto event loop will not be able to perform any I/O.
192+
193+
When a method has a `Context` parameter:
194+
195+
**Client side** (`CustomBuildField`):
196+
If this is the first asynchronous request made from the current client thread, `CustomBuildField` will:
197+
1. Call `connection.m_thread_map.makeThreadRequest()` to request a dedicated worker thread on the server (stored in `request_threads` map)
198+
2. Set the remote thread capability in `Context.thread`
199+
3. Create a local `Thread::Server` object for the current thread (stored in `callback_threads` map)
200+
4. Set the local thread capability in `Context.callbackThread`
201+
202+
Subsequent requests will resuse the existing thread capabilites held in `callback_threads` and `request_threads`.
203+
204+
**Server side** (`PassField`):
205+
1. Looks up the local `Thread::Server` object specified by `context.thread`
206+
2. The worker thread:
207+
- Stores `context.callbackThread` in its `request_threads` map (so callbacks go to the right client thread)
208+
- Posts the work lambda to that thread's queue via `waiter->post(invoke)`
209+
- Cleans up the `request_threads` entry
210+
211+
## Interface Definitions
29212

30213
As explained in the [usage](usage.md) document, interface descriptions need to be consumed both by the _libmultiprocess_ code generator, and by C++ code that calls and implements the interfaces. The C++ code only needs to know about C++ arguments and return types, while the code generator only needs to know about capnp arguments and return types, but both need to know class and method names, so the corresponding `.h` and `.capnp` source files contain some of the same information, and have to be kept in sync manually when methods or parameters change. Despite the redundancy, reconciling the interface definitions is designed to be _straightforward_ and _safe_. _Straightforward_ because there is no need to write manual serialization code or use awkward intermediate types like [`UniValue`](https://github.com/bitcoin/bitcoin/blob/master/src/univalue/include/univalue.h) instead of native types. _Safe_ because if there are any inconsistencies between API and data definitions (even minor ones like using a narrow int data type for a wider int API input), there are errors at build time instead of errors or bugs at runtime.
31214

0 commit comments

Comments
 (0)