Skip to content

Commit 7756fdc

Browse files
EunovoEunovo
committed
Design doc update
1 parent a4f9296 commit 7756fdc

File tree

1 file changed

+188
-2
lines changed

1 file changed

+188
-2
lines changed

‎doc/design.md‎

Lines changed: 188 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ There is also optional support for thread mapping, so each thread making interpr
1313

1414
Libmultiprocess acts as a pure wrapper or layer over the underlying protocol. Clients and servers written in other languages, but using a shared capnproto schema can communicate with interprocess counterparties using libmultiprocess without having to use libmultiprocess themselves or having to know about the implementation details of libmultiprocess.
1515

16-
### Internals
16+
## Core Architecture
1717

1818
The `ProxyClient` and `ProxyServer` generated classes are not directly exposed to the user, as described in [usage.md](usage.md). Instead, they wrap C++ interfaces and appear to the user as pointers to an interface. They are first instantiated when calling `ConnectStream` and `ServeStream` respectively for creating the `InitInterface`. These methods establish connections through sockets, internally creating `Connection` objects wrapping a `capnp::RpcSystem` configured for client and server mode respectively.
1919

@@ -25,7 +25,193 @@ When a generated method on the `ProxyClient` is called, it calls `clientInvoke`
2525

2626
On the server side, the `capnp::RpcSystem` receives the capnp request and invokes the corresponding C++ method through the corresponding `ProxyServer` and the heavily templated `serverInvoke` triggering a `ServerCall`. The return values from the actual C++ methods are copied into capnp responses by `ServerRet` and exceptions are caught and copied by `ServerExcept`. The two are connected through `ServerField`. The main method driving execution of a request is `PassField`, which is invoked through `ServerField`. Instantiated interfaces, or capabilities in capnp speak, are tracked and owned by the server's `capnp::RpcSystem`.
2727

28-
## Interface descriptions
28+
## Request and Response Flow
29+
30+
Method parameters and return values are serialized using Cap'n Proto's Builder objects (for sending) and Reader objects (for receiving). Input parameters flow from the client to the server, while output parameters (return values) flow back from the server to the client.
31+
32+
```mermaid
33+
sequenceDiagram
34+
participant clientInvoke
35+
participant ReadField_C as ReadField<br/>(Client)
36+
participant BuildField as BuildField<br/>(Client)
37+
participant Socket
38+
participant ReadField as ReadField<br/>(Server)
39+
participant BuildField_S as BuildField<br/>(Server)
40+
participant serverInvoke
41+
42+
Note over clientInvoke,serverInvoke: Input Parameter Flow
43+
clientInvoke->>BuildField: BuildField(input_arg)
44+
BuildField->>Socket: Serialize input
45+
Socket->>ReadField: Cap'n Proto message
46+
ReadField->>serverInvoke: Deserialize input
47+
48+
Note over clientInvoke,serverInvoke: Output Parameter Flow
49+
serverInvoke-->>BuildField_S: BuildField(output)
50+
BuildField_S-->>Socket: Serialize output
51+
Socket-->>ReadField_C: Cap'n Proto message
52+
ReadField_C-->>clientInvoke: Deserialize output
53+
```
54+
55+
### Detailed Serialization Mechanism
56+
57+
Parameters are represented as Fields that must be set on Cap'n Proto Builder objects (for sending) and read from Reader objects (for receiving).
58+
59+
#### Building Fields
60+
61+
`BuildField` uses a `StructField` (identifying which field by index) and a parameter `Accessor` to set the appropriate field in the Cap'n Proto Builder object.
62+
63+
```mermaid
64+
sequenceDiagram
65+
participant clientInvoke
66+
participant BuildField
67+
participant StructField
68+
participant Accessor
69+
participant Builder as Params::Builder
70+
71+
Note over clientInvoke,Builder: Serializing Parameters
72+
clientInvoke->>BuildField: BuildField(param1)
73+
BuildField->>StructField: Make StructField
74+
StructField->>Accessor: Use generated field accessor
75+
Accessor->>Builder: builder.setField1(param1)
76+
77+
clientInvoke->>BuildField: BuildField(param2)
78+
BuildField->>StructField: Make StructField
79+
StructField->>Accessor: Use generated field Accessor
80+
Accessor->>Builder: builder.setField2(param2)
81+
```
82+
83+
The same process is used for building results on the server side with `Results::Builder`.
84+
85+
#### Reading Fields
86+
87+
`ReadField` uses a `StructField` (identifying which field by index) and a parameter `Accessor` to read the appropriate field from the Cap'n Proto Reader object and reconstruct C++ parameters.
88+
89+
```mermaid
90+
sequenceDiagram
91+
participant serverInvoke
92+
participant PassField
93+
participant ReadField
94+
participant StructField
95+
participant Accessor
96+
participant Reader as Params::Reader
97+
participant ServerCall
98+
99+
Note over serverInvoke,ServerCall: Deserializing Parameters
100+
serverInvoke->>PassField: PassField for param1
101+
ReadField->>StructField: Make StructField
102+
StructField->>Accessor: Use generated field accessor
103+
Accessor->>Reader: reader.getField1()
104+
Reader-->>PassField: emplace/construct param1
105+
106+
PassField-->>ServerCall: Reconstructed params
107+
```
108+
109+
The same process is used for reading results on the client side with `Results::Reader`.
110+
111+
## Server-Side Request Processing
112+
113+
The generated server code uses a Russian nesting doll structure to process method fields. Each `ServerField` wraps another `ServerField` (for the next parameter), or wraps `ServerRet` (for the return value), which finally wraps `ServerCall` (which invokes the actual C++ method).
114+
115+
Each `ServerField` invokes `PassField`, which:
116+
1. Calls `ReadField` to deserialize the parameter from the `Params::Reader`
117+
2. Calls the next nested layer's `invoke()` with the accumulated parameters
118+
3. Calls `BuildField` to serialize the parameter back if it's an output parameter
119+
120+
`ServerRet` invokes the next layer (typically `ServerCall`), stores the result, and calls `BuildField` to serialize it into the `Results::Builder`.
121+
122+
`ServerCall` uses the generated `ProxyMethod<MethodParams>::impl` pointer-to-member to invoke the actual C++ method on the wrapped implementation object.
123+
124+
```mermaid
125+
sequenceDiagram
126+
participant serverInvoke
127+
participant SF1 as ServerField<br/>(param 1)
128+
participant SF2 as ServerField<br/>(param 2)
129+
participant SR as ServerRet<br/>(return value)
130+
participant SC as ServerCall
131+
participant PMT as ProxyMethodTraits
132+
participant Impl as Actual C++ Method
133+
134+
serverInvoke->>SF1: SF1::invoke
135+
SF1->>SF2: SF2::invoke
136+
SF2->>SR: SR::invoke
137+
SR->>SC: SC::invoke
138+
SC->>PMT: PMT::invoke
139+
PMT->>Impl: Call impl method
140+
Impl->>PMT: return
141+
PMT->>SC: return
142+
SC->>SR: return
143+
SR->>SF2: return
144+
SF2->>SF1: return
145+
SF1->>serverInvoke: return
146+
```
147+
148+
## Advanced Features
149+
150+
### Callbacks
151+
152+
Callbacks (passed as `std::function` arguments) are intercepted by `CustomBuildField` and converted into Cap'n Proto capabilities that can be invoked across process boundaries. On the receiving end, `CustomReadField` intercepts the capability and constructs a `ProxyCallFn` object with an `operator()` that sends function calls back over the socket to invoke the original callback.
153+
154+
```mermaid
155+
sequenceDiagram
156+
participant CT as Client Thread
157+
participant C as clientInvoke
158+
participant CBF1 as CustomBuildField (Client)
159+
participant S as Socket
160+
participant CRF1 as CustomReadField (Server)
161+
participant Srv as Server Code
162+
participant PCF as ProxyCallFn
163+
164+
C->>CBF1: send function parameter
165+
CBF1->>S: creates a Server for the function and sends a capability
166+
S->>CRF1: receivies capabily creates ProxyCallFn
167+
CRF1->>Srv:
168+
Srv->>PCF: call the callback
169+
PCF-->>CT: sends request to Client
170+
```
171+
172+
### Thread Mapping
173+
174+
Thread mapping enables each client thread to have a dedicated server thread processing its requests, preserving thread-local state and allowing recursive mutex usage across process boundaries.
175+
176+
Thread mapping is initialized by defining an interface method with a `ThreadMap` parameter and/or response, such as:
177+
178+
```capnp
179+
initThreadMap @4 (threadMap: Proxy.ThreadMap) -> (threadMap :Proxy.ThreadMap);
180+
```
181+
182+
- **ThreadMap in parameter**: The client's `CustomBuildField` creates a `ThreadMap::Server` capability and sends it to the server, where `CustomReadField` stores the `ThreadMap::Client` in `connection.m_thread_map`
183+
- **ThreadMap in response**: The server's `CustomBuildField` creates a `ThreadMap::Server` capability and sends it to the client, where `CustomReadField` stores the `ThreadMap::Client` in `connection.m_thread_map`
184+
185+
You can specify ThreadMap in the parameter only, response only, or both:
186+
- **Parameter only**: Server can create threads on the client
187+
- **Response only**: Client can create threads on the server
188+
- **Both (as shown)**: Bidirectional thread creation
189+
190+
When both parameter and response include ThreadMap, both processes end up with `ThreadMap::Client` capabilities pointing to each other's `ThreadMap::Server`, allowing both sides to create threads on the other process.
191+
192+
### Async Processing with Context
193+
194+
By adding a `Context` parameter to a method in the capnp interface file, you enable async processing where the client tells the server to execute the request in a separate worker thread. For example:
195+
196+
```capnp
197+
processData @5 (context :Proxy.Context, data :Data) -> (result :Result);
198+
```
199+
200+
When a method has a `Context` parameter:
201+
202+
**Client side** (`CustomBuildField`):
203+
1. Creates a local `Thread::Server` object for the current thread (stored in `callback_threads` map)
204+
2. Calls `connection.m_thread_map.makeThreadRequest()` to request a dedicated worker thread on the server (stored in `request_threads` map)
205+
3. Sends both thread capabilities in the Context.
206+
207+
**Server side** (`PassField`):
208+
1. Looks up the local `Thread::Server` object specified by `context.thread`
209+
2. The worker thread:
210+
- Stores `context.callbackThread` in its `request_threads` map (so callbacks go to the right client thread)
211+
- Posts the work lambda to that thread's queue via `waiter->post(invoke)`
212+
- Cleans up the `request_threads` entry
213+
214+
## Interface Definitions
29215

30216
As explained in the [usage](usage.md) document, interface descriptions need to be consumed both by the _libmultiprocess_ code generator, and by C++ code that calls and implements the interfaces. The C++ code only needs to know about C++ arguments and return types, while the code generator only needs to know about capnp arguments and return types, but both need to know class and method names, so the corresponding `.h` and `.capnp` source files contain some of the same information, and have to be kept in sync manually when methods or parameters change. Despite the redundancy, reconciling the interface definitions is designed to be _straightforward_ and _safe_. _Straightforward_ because there is no need to write manual serialization code or use awkward intermediate types like [`UniValue`](https://github.com/bitcoin/bitcoin/blob/master/src/univalue/include/univalue.h) instead of native types. _Safe_ because if there are any inconsistencies between API and data definitions (even minor ones like using a narrow int data type for a wider int API input), there are errors at build time instead of errors or bugs at runtime.
31217

0 commit comments

Comments
 (0)