Open
Description
It seems like we have a bottleneck in our C++ SDK. It completely blocks #17333. Java SDK seems to have both a better latency and ~ 15x throughput.
Below is a comparison of SELECT 1 microbenchmark results. There're 3 servers in YDB cluster, and each has 16 cores 6 dynnodes.
С++ version (ydb debug latency) without the option to have own TCP connection per gRPC channel:
> ./ydb -p perf1 debug latency -k Select1 --min-inflight 1 --max-inflight 128
Select1 threads=1, throughput: 1403, per thread: 1403, p50 usec: 709, p90 usec: 731, p99 usec: 771, ok: 7019, error: 0
Select1 threads=2, throughput: 2748, per thread: 1374, p50 usec: 725, p90 usec: 748, p99 usec: 776, ok: 13742, error: 0
Select1 threads=3, throughput: 4134, per thread: 1378, p50 usec: 723, p90 usec: 747, p99 usec: 776, ok: 20671, error: 0
Select1 threads=4, throughput: 5209, per thread: 1302, p50 usec: 740, p90 usec: 797, p99 usec: 1220, ok: 26046, error: 0
Select1 threads=5, throughput: 6088, per thread: 1217, p50 usec: 749, p90 usec: 1149, p99 usec: 1277, ok: 30440, error: 0
Select1 threads=6, throughput: 7501, per thread: 1250, p50 usec: 743, p90 usec: 1126, p99 usec: 1244, ok: 37506, error: 0
Select1 threads=7, throughput: 8972, per thread: 1281, p50 usec: 748, p90 usec: 814, p99 usec: 1230, ok: 44863, error: 0
Select1 threads=8, throughput: 9996, per thread: 1249, p50 usec: 749, p90 usec: 1083, p99 usec: 1246, ok: 49983, error: 0
Select1 threads=9, throughput: 11508, per thread: 1278, p50 usec: 755, p90 usec: 847, p99 usec: 1220, ok: 57540, error: 0
Select1 threads=10, throughput: 12046, per thread: 1204, p50 usec: 764, p90 usec: 996, p99 usec: 1272, ok: 60234, error: 0
Select1 threads=11, throughput: 14230, per thread: 1293, p50 usec: 762, p90 usec: 817, p99 usec: 922, ok: 71154, error: 0
Select1 threads=12, throughput: 15331, per thread: 1277, p50 usec: 754, p90 usec: 832, p99 usec: 1244, ok: 76656, error: 0
Select1 threads=13, throughput: 16570, per thread: 1274, p50 usec: 770, p90 usec: 843, p99 usec: 1010, ok: 82850, error: 0
Select1 threads=14, throughput: 16918, per thread: 1208, p50 usec: 792, p90 usec: 972, p99 usec: 1100, ok: 84590, error: 0
Select1 threads=15, throughput: 18149, per thread: 1209, p50 usec: 813, p90 usec: 902, p99 usec: 996, ok: 90745, error: 0
Select1 threads=16, throughput: 17820, per thread: 1113, p50 usec: 811, p90 usec: 1174, p99 usec: 1327, ok: 89102, error: 0
Select1 threads=32, throughput: 19066, per thread: 595, p50 usec: 1663, p90 usec: 2046, p99 usec: 2592, ok: 95331, error: 0
Select1 threads=64, throughput: 19951, per thread: 311, p50 usec: 3045, p90 usec: 4611, p99 usec: 5828, ok: 99757, error: 0
Select1 threads=128, throughput: 21664, per thread: 169, p50 usec: 5842, p90 usec: 7619, p99 usec: 13811, ok: 108320, error: 0
> java -jar target/select1-benchmark-1.0-SNAPSHOT.jar --jdbc-url "jdbc:ydb:grpc://ydb-vla-dev04-000.search.yandex.net:2135/Root/db1?useQueryService=true" --min-inflight 1 --max-inflight 128 -i 5
May 29, 2025 12:39:27 PM tech.ydb.jdbc.YdbDriver register
INFO: YDB JDBC Driver registered: YDB JDBC Driver 2.3.6(based on SDK 2.3.6)
Running benchmark with parameters:
JDBC URL: jdbc:ydb:grpc://ydb-vla-dev04-000.search.yandex.net:2135/Root/db1?useQueryService=true, Min Inflight: 1, Max Inflight: 128, Interval: 5 seconds, Format: human
Benchmark Results
======================================================================
Inflight RPS P50 (µs) P90 (µs) P99 (µs) P99.9 (µs)
----------------------------------------------------------------------
1 3069.83 253.12 333.71 802.70 1623.11
2 8265.73 236.73 256.86 305.53 437.94
3 12587.47 235.19 249.62 268.68 313.09
4 16573.79 236.70 254.10 298.76 381.53
5 20545.23 240.27 257.14 275.49 309.69
6 24754.91 237.85 256.45 305.95 382.23
7 28927.29 236.91 256.63 309.41 396.65
8 32305.42 242.74 262.26 314.33 440.97
9 36824.69 238.85 258.78 315.61 885.54
10 39999.91 243.20 265.86 336.49 860.75
11 43661.12 244.01 273.46 354.96 515.79
12 47899.68 245.53 269.96 322.86 377.78
13 51111.85 245.13 281.42 362.62 435.24
14 52300.91 253.88 314.04 424.55 517.94
15 58999.32 246.96 276.57 362.37 472.92
16 63157.45 246.05 278.17 350.01 452.11
17 65066.13 252.64 292.17 380.53 476.60
18 68242.09 252.10 302.27 407.52 505.96
19 74512.33 247.82 276.11 356.32 469.49
20 78785.52 246.69 272.39 352.90 517.58
21 79431.42 253.99 297.35 396.21 659.08
22 84916.07 249.16 285.24 395.30 651.83
23 85776.72 254.80 308.42 433.34 669.53
24 91928.08 251.44 286.71 379.22 666.24
25 91938.43 260.12 309.42 410.38 690.47
26 98564.23 252.49 293.98 398.54 671.31
27 99721.29 257.73 307.93 413.10 704.88
28 105392.09 255.97 293.49 388.36 711.61
29 105632.17 259.92 318.81 443.30 717.70
30 108769.17 263.18 315.80 435.19 712.89
31 111875.30 263.70 313.83 424.17 730.62
32 112282.39 266.82 341.20 484.65 751.98
33 115813.77 270.49 334.29 445.73 746.64
34 120519.47 265.50 332.75 459.39 752.86
35 121107.12 271.34 347.35 482.78 776.48
36 127359.48 267.29 332.15 463.80 770.71
37 128362.42 267.75 347.48 525.31 807.90
38 135374.74 263.89 325.46 477.38 773.48
39 139258.28 265.67 320.76 453.11 782.93
40 140523.40 267.25 335.32 476.89 808.29
41 141881.60 272.69 338.37 484.16 839.18
42 140827.04 278.26 362.79 520.02 842.89
43 150021.68 270.17 329.87 486.53 831.50
44 150391.08 276.96 343.16 493.48 843.47
45 157093.67 271.11 330.37 475.18 829.54
46 156761.03 275.64 348.08 507.94 856.42
47 150855.05 280.34 365.07 540.41 900.57
48 159429.54 281.32 361.32 530.52 930.86
49 160307.02 286.68 366.68 528.63 899.27
50 169618.98 277.70 341.21 504.86 923.93
51 170540.16 281.73 351.41 518.01 942.80
52 174220.06 279.73 352.03 528.95 942.28
53 175591.32 282.15 359.14 544.34 974.40
54 174175.94 285.51 372.42 597.78 3185.98
55 172004.97 298.28 403.53 571.66 975.96
56 183309.43 286.74 359.91 536.59 940.79
57 182459.30 292.60 375.25 561.55 974.28
58 188781.80 284.34 374.27 581.61 977.07
59 189132.28 292.70 374.59 566.16 962.51
60 191510.82 294.90 376.52 571.53 963.75
61 193405.67 296.30 378.03 570.80 982.22
62 198040.39 294.83 373.58 565.98 986.22
63 200928.81 293.07 373.58 590.10 1016.97
64 195533.94 306.81 404.23 610.78 1026.62
65 197066.86 302.77 418.52 646.49 1041.52
66 206200.66 298.45 388.17 611.07 1025.89
67 209057.41 299.01 385.30 600.89 1044.93
68 209402.28 302.94 392.27 631.10 1106.80
69 208200.12 309.86 410.38 663.79 1111.23
70 215915.78 300.77 393.50 647.99 1114.34
71 209422.85 317.40 419.33 683.60 1183.53
72 217379.89 308.97 405.29 663.19 1137.45
73 220900.41 306.12 397.15 677.27 1153.41
74 224886.67 305.97 393.53 675.17 1217.23
75 228354.60 307.30 390.92 673.90 1148.35
76 228215.02 311.61 399.08 677.66 1185.38
77 231194.71 311.71 396.22 692.87 1245.55
78 231982.08 312.91 410.08 701.89 1232.21
79 225220.14 321.66 444.39 728.80 1305.74
80 235594.16 315.93 408.45 709.22 1203.70
81 235835.17 318.37 423.87 747.86 1219.77
82 239298.60 318.30 419.25 739.08 1209.21
83 241512.34 319.72 419.31 736.33 1223.38
84 241750.38 323.68 429.31 729.82 1226.71
85 243290.11 324.07 422.99 752.21 1270.61
86 246451.48 325.12 427.78 737.09 1286.39
87 250197.72 323.93 428.80 720.48 1194.76
88 248334.34 329.51 438.24 756.58 1294.80
89 257325.33 322.39 416.48 734.60 1253.50
90 256069.89 327.56 432.21 751.80 1232.66
91 256459.15 329.24 442.10 765.98 1278.22
92 259901.85 328.31 432.62 793.03 1349.67
93 254157.03 338.52 465.07 781.11 1370.04
94 263176.73 332.42 441.46 785.94 1390.96
95 261232.63 335.15 460.78 803.57 1385.00
96 264941.80 336.18 451.42 793.10 1352.06
97 264610.96 336.71 465.60 819.59 1499.39
98 278350.34 329.44 417.93 809.97 1381.71
99 270331.93 340.03 455.63 799.59 1526.17
100 277268.37 334.32 442.30 836.08 1368.66
101 275956.17 339.24 458.29 819.96 1759.12
102 280158.20 338.24 448.05 841.54 1612.39
103 278021.98 338.51 466.02 855.62 1648.08
104 284348.35 340.02 442.15 898.91 1409.71
105 284826.43 342.61 450.65 893.89 1413.34
106 279357.01 346.93 487.06 895.12 1451.95
107 288296.60 345.56 451.93 907.43 1546.67
108 283198.32 349.26 486.20 911.00 1425.10
109 290859.72 343.93 463.32 917.30 1488.93
110 294965.73 344.12 459.04 921.93 1440.72
111 287142.50 355.42 494.46 927.40 1484.46
112 294040.35 351.86 471.17 926.14 1493.19
113 298855.91 349.40 462.42 936.28 1586.71
114 295129.67 355.04 486.82 949.82 1511.49
115 293627.09 358.56 498.07 955.10 1901.59
116 300916.88 354.46 484.87 966.77 1537.18
117 300661.03 356.14 476.36 966.81 1554.41
118 292754.27 364.69 530.01 952.63 1468.27
119 302838.68 359.82 489.03 975.87 1484.81
120 308931.58 353.77 484.46 991.88 1469.81
121 304045.33 366.22 500.16 987.14 1471.40
122 309095.62 363.27 492.51 1001.93 1465.14
123 311228.33 359.79 490.70 1010.26 1488.62
124 314038.55 361.09 492.96 1000.77 1466.14
125 319762.49 357.61 475.74 1018.64 1473.01
126 314835.53 364.68 502.17 1025.84 1473.40
127 311685.68 365.93 530.95 1020.33 1526.02
128 316965.98 369.42 498.33 1056.25 1509.55
======================================================================
In case of Java, if we mutate JDBC url with "client_id=threadnum" to have more channels / TCP connections then results are even better:
> java -jar target/select1-benchmark-1.0-SNAPSHOT.jar --jdbc-url "jdbc:ydb:grpc://ydb-vla-dev04-000.search.yandex.net:2135/Root/db1?useQueryService=true" --min-inflight 1 --max-inflight 128 -i 5
May 29, 2025 12:51:31 PM tech.ydb.jdbc.YdbDriver register
INFO: YDB JDBC Driver registered: YDB JDBC Driver 2.3.6(based on SDK 2.3.6)
Running benchmark with parameters:
JDBC URL: jdbc:ydb:grpc://ydb-vla-dev04-000.search.yandex.net:2135/Root/db1?useQueryService=true, Min Inflight: 1, Max Inflight: 128, Interval: 5 seconds, Format: human
Benchmark Results
======================================================================
Inflight RPS P50 (µs) P90 (µs) P99 (µs) P99.9 (µs)
----------------------------------------------------------------------
1 3077.83 251.64 333.56 823.43 1481.08
2 8284.10 235.79 256.34 306.80 460.88
3 12525.96 236.15 252.30 270.56 306.14
4 16651.07 234.35 250.43 269.00 311.73
5 20953.37 235.72 251.01 269.53 312.71
6 25097.02 236.00 251.22 269.59 316.68
7 28980.64 237.80 254.86 276.02 331.11
8 33052.21 238.55 254.41 273.57 332.38
9 36701.44 241.24 258.86 281.94 381.96
10 40923.70 240.12 257.79 282.31 508.39
11 45064.99 241.19 258.58 278.79 335.67
12 49085.81 241.62 259.19 281.16 338.96
13 52873.46 242.89 260.68 283.00 344.39
14 56393.94 245.20 264.11 289.71 402.62
15 59302.17 249.76 271.77 295.98 433.38
16 64559.27 244.96 262.95 287.12 436.21
17 66627.82 251.91 274.24 301.18 447.22
18 71580.60 248.22 268.02 293.92 448.54
19 74905.74 249.75 270.44 298.38 455.01
20 78602.16 250.39 270.99 297.43 564.64
21 82245.54 251.14 271.74 309.70 632.74
22 85697.97 252.11 272.19 305.94 629.49
23 88840.70 254.67 274.92 306.47 646.45
24 91063.30 258.35 283.41 315.07 660.12
25 94951.22 256.06 287.75 328.59 663.88
26 100224.55 254.40 274.49 306.90 670.80
27 103795.52 255.29 275.44 309.74 669.36
28 106203.99 258.27 281.39 320.22 688.13
29 109915.42 258.20 280.43 315.61 688.27
30 114248.45 257.16 277.37 312.44 686.35
31 117446.43 257.82 279.73 318.56 717.63
32 120418.23 260.09 281.89 321.84 724.05
33 125469.08 257.08 276.82 318.70 720.16
34 126238.95 262.16 288.53 346.27 725.83
35 130344.25 261.40 285.39 373.52 739.97
36 131598.11 265.90 293.22 378.50 748.32
37 133756.22 268.95 295.55 347.52 769.45
38 139457.43 264.69 289.33 372.42 755.02
39 141316.83 266.42 292.15 380.08 775.92
40 145795.08 265.94 292.98 390.64 779.35
41 147857.72 269.43 296.17 368.36 789.21
42 153586.45 265.23 290.97 381.52 806.11
43 155733.23 266.92 293.17 410.89 814.03
44 159889.70 267.51 292.61 398.37 777.40
45 162298.59 268.37 296.61 410.16 816.10
46 163973.31 272.58 300.45 415.31 829.85
47 167063.38 272.95 302.42 423.79 844.67
48 170763.44 272.82 298.98 431.54 851.81
49 174505.62 272.16 297.15 445.38 876.02
50 176644.83 274.36 300.36 454.47 872.78
51 179994.89 272.53 306.33 465.77 880.22
52 182366.17 275.36 304.83 474.97 899.96
53 186074.34 274.80 304.04 489.97 904.71
54 185370.52 275.29 331.53 497.86 936.64
55 189400.22 279.02 310.00 519.84 915.55
56 195804.58 276.20 304.12 510.14 910.81
57 198537.60 277.07 308.09 519.70 926.73
58 199551.42 277.99 315.87 529.51 932.44
59 205533.55 276.80 304.43 538.83 948.75
60 204445.68 282.74 314.94 548.51 967.10
61 203005.60 289.44 329.19 561.05 988.90
62 210205.75 282.29 320.87 562.72 981.33
63 213353.14 282.60 321.17 575.59 1005.47
64 216333.71 282.76 322.48 577.73 1000.07
65 219677.78 282.94 318.79 583.96 1005.89
66 221999.75 285.28 319.77 595.36 1011.43
67 225965.19 284.25 315.50 599.61 1025.11
68 228725.97 284.68 315.36 607.01 1031.12
69 230405.10 285.42 323.03 605.98 1029.59
70 232288.73 288.38 323.85 626.61 1051.08
71 233545.08 288.17 322.21 636.54 1070.03
72 236509.99 290.22 323.92 648.19 1076.81
73 235980.59 290.20 325.64 655.09 1094.93
74 238428.53 293.01 333.02 667.33 1109.57
75 243409.00 292.61 335.35 669.81 1110.58
76 248687.62 291.94 326.16 673.52 1114.19
77 249010.73 294.53 338.88 682.65 1124.13
78 253617.55 292.54 332.78 690.74 1135.28
79 255379.26 294.79 333.02 704.97 1149.76
80 257230.20 296.07 336.53 714.75 1160.58
81 259774.23 296.42 339.80 716.10 1176.31
82 257755.17 298.25 362.73 719.72 1188.45
83 266569.26 295.23 334.81 725.09 1168.57
84 269941.27 296.59 335.02 736.09 1164.78
85 269714.83 297.54 339.17 748.01 1189.68
86 273897.21 297.66 338.65 760.99 1200.32
87 273170.10 302.22 343.28 756.27 1203.69
88 276408.13 302.00 347.67 769.22 1211.70
89 280520.54 300.66 340.88 781.52 1221.86
90 283792.70 301.52 339.68 792.13 1223.14
91 283475.77 301.58 348.93 804.08 1246.92
92 288034.41 302.25 343.64 817.21 1265.79
93 284836.28 308.31 361.04 824.05 1306.77
94 291768.43 304.45 347.29 812.53 1249.50
95 295142.91 303.64 346.01 846.89 1303.48
96 296295.95 305.15 355.60 833.73 1271.40
97 297063.25 304.70 353.81 857.98 1330.48
98 295631.29 310.09 359.87 862.19 1341.07
99 302886.40 307.72 354.34 874.04 1327.59
100 300875.75 312.46 370.93 876.67 1344.26
101 303949.22 312.78 365.68 905.62 1372.30
102 305967.85 313.45 366.62 905.48 1378.94
103 304845.80 315.84 376.08 910.07 1396.34
104 315493.66 310.08 352.89 909.76 1369.03
105 307206.36 318.67 375.80 927.78 1407.21
106 316760.79 312.26 356.78 934.23 1667.64
107 318996.12 314.69 366.19 962.78 1415.33
108 323924.32 311.63 361.47 957.50 1427.14
109 323217.31 315.90 367.63 960.75 1417.99
110 323972.98 316.85 375.11 968.60 1449.90
111 328849.82 316.03 366.91 974.96 1468.36
112 321827.47 324.16 394.05 991.37 1487.36
113 332183.70 316.81 369.78 999.92 1454.06
114 324476.55 322.71 413.60 1011.26 1514.47
115 335155.92 320.30 370.20 1025.58 1646.13
116 336858.17 320.02 381.41 1020.20 1534.46
117 334246.56 321.44 382.24 1029.28 1486.09
118 336603.41 322.25 383.77 1032.37 1475.49
119 338399.32 322.99 393.41 1027.29 1465.04
120 171577.29 323.82 376.82 1035.49 1477.29
121 338928.13 322.62 384.78 1101.88 5283.71
122 345871.12 324.09 384.16 1087.46 1574.96
123 343130.32 328.95 396.82 1071.78 1529.27
124 349743.97 326.40 386.97 1071.19 1518.54
125 351857.52 327.94 387.53 1087.65 1526.17
126 351806.76 329.89 393.57 1090.33 1537.31
127 348380.02 335.65 407.77 1088.93 1539.26
128 359331.47 327.95 387.00 1099.11 1544.41
======================================================================
Finally, in case of C++ it seems that per channel TCP connection seriously improves throughput when we have a single dynnode. But in this particular case (3 server x 6 dynnodes = 18 dynnodes) this option degrades the results.
PlainGrpcPing without option set:
> ./ydb -p perf1 debug latency -k PlainGrpc --min-inflight 1 --max-inflight 128
PlainGrpc threads=1, throughput: 2420, per thread: 2420, p50 usec: 409, p90 usec: 428, p99 usec: 462, ok: 12100, error: 0
PlainGrpc threads=2, throughput: 4786, per thread: 2393, p50 usec: 413, p90 usec: 435, p99 usec: 516, ok: 23934, error: 0
PlainGrpc threads=3, throughput: 6577, per thread: 2192, p50 usec: 420, p90 usec: 530, p99 usec: 835, ok: 32889, error: 0
PlainGrpc threads=4, throughput: 8963, per thread: 2240, p50 usec: 421, p90 usec: 484, p99 usec: 827, ok: 44816, error: 0
PlainGrpc threads=5, throughput: 11036, per thread: 2207, p50 usec: 429, p90 usec: 511, p99 usec: 832, ok: 55180, error: 0
PlainGrpc threads=6, throughput: 13250, per thread: 2208, p50 usec: 429, p90 usec: 520, p99 usec: 831, ok: 66252, error: 0
PlainGrpc threads=7, throughput: 15269, per thread: 2181, p50 usec: 435, p90 usec: 533, p99 usec: 823, ok: 76349, error: 0
PlainGrpc threads=8, throughput: 17220, per thread: 2152, p50 usec: 435, p90 usec: 551, p99 usec: 843, ok: 86103, error: 0
PlainGrpc threads=9, throughput: 19434, per thread: 2159, p50 usec: 437, p90 usec: 550, p99 usec: 798, ok: 97171, error: 0
PlainGrpc threads=10, throughput: 21188, per thread: 2118, p50 usec: 445, p90 usec: 563, p99 usec: 782, ok: 105940, error: 0
PlainGrpc threads=11, throughput: 23063, per thread: 2096, p50 usec: 444, p90 usec: 575, p99 usec: 834, ok: 115319, error: 0
PlainGrpc threads=12, throughput: 24864, per thread: 2072, p50 usec: 449, p90 usec: 584, p99 usec: 815, ok: 124320, error: 0
PlainGrpc threads=13, throughput: 26449, per thread: 2034, p50 usec: 456, p90 usec: 598, p99 usec: 820, ok: 132247, error: 0
PlainGrpc threads=14, throughput: 27920, per thread: 1994, p50 usec: 461, p90 usec: 618, p99 usec: 850, ok: 139604, error: 0
PlainGrpc threads=15, throughput: 29624, per thread: 1974, p50 usec: 464, p90 usec: 632, p99 usec: 870, ok: 148121, error: 0
PlainGrpc threads=16, throughput: 30851, per thread: 1928, p50 usec: 474, p90 usec: 655, p99 usec: 887, ok: 154258, error: 0
PlainGrpc threads=32, throughput: 39125, per thread: 1222, p50 usec: 761, p90 usec: 1167, p99 usec: 1579, ok: 195629, error: 0
PlainGrpc threads=64, throughput: 42700, per thread: 667, p50 usec: 1413, p90 usec: 2232, p99 usec: 3066, ok: 213502, error: 0
PlainGrpc threads=128, throughput: 47295, per thread: 369, p50 usec: 2581, p90 usec: 4104, p99 usec: 5641, ok: 236476, error: 0
With option set:
> ./ydb -p perf1 debug latency -k PlainGrpc --min-inflight 1 --max-inflight 128
PlainGrpc threads=1, throughput: 2425, per thread: 2425, p50 usec: 409, p90 usec: 426, p99 usec: 460, ok: 12126, error: 0
PlainGrpc threads=2, throughput: 4734, per thread: 2367, p50 usec: 419, p90 usec: 440, p99 usec: 475, ok: 23670, error: 0
PlainGrpc threads=3, throughput: 7034, per thread: 2344, p50 usec: 421, p90 usec: 448, p99 usec: 494, ok: 35170, error: 0
PlainGrpc threads=4, throughput: 8924, per thread: 2231, p50 usec: 428, p90 usec: 458, p99 usec: 834, ok: 44621, error: 0
PlainGrpc threads=5, throughput: 11096, per thread: 2219, p50 usec: 427, p90 usec: 461, p99 usec: 840, ok: 55483, error: 0
PlainGrpc threads=6, throughput: 13536, per thread: 2256, p50 usec: 437, p90 usec: 464, p99 usec: 529, ok: 67680, error: 0
PlainGrpc threads=7, throughput: 15698, per thread: 2242, p50 usec: 437, p90 usec: 461, p99 usec: 795, ok: 78492, error: 0
PlainGrpc threads=8, throughput: 17849, per thread: 2231, p50 usec: 439, p90 usec: 464, p99 usec: 800, ok: 89249, error: 0
PlainGrpc threads=9, throughput: 20139, per thread: 2237, p50 usec: 443, p90 usec: 466, p99 usec: 526, ok: 100697, error: 0
PlainGrpc threads=10, throughput: 22209, per thread: 2220, p50 usec: 445, p90 usec: 470, p99 usec: 545, ok: 111048, error: 0
PlainGrpc threads=11, throughput: 24233, per thread: 2203, p50 usec: 448, p90 usec: 475, p99 usec: 552, ok: 121165, error: 0
PlainGrpc threads=12, throughput: 26236, per thread: 2186, p50 usec: 452, p90 usec: 481, p99 usec: 548, ok: 131184, error: 0
PlainGrpc threads=13, throughput: 28181, per thread: 2167, p50 usec: 456, p90 usec: 487, p99 usec: 556, ok: 140906, error: 0
PlainGrpc threads=14, throughput: 30009, per thread: 2143, p50 usec: 461, p90 usec: 495, p99 usec: 570, ok: 150048, error: 0
PlainGrpc threads=15, throughput: 31843, per thread: 2122, p50 usec: 464, p90 usec: 503, p99 usec: 590, ok: 159217, error: 0
PlainGrpc threads=16, throughput: 32999, per thread: 2062, p50 usec: 476, p90 usec: 522, p99 usec: 622, ok: 164995, error: 0
PlainGrpc threads=32, throughput: 34365, per thread: 1073, p50 usec: 894, p90 usec: 1415, p99 usec: 1976, ok: 171825, error: 0
PlainGrpc threads=64, throughput: 33033, per thread: 516, p50 usec: 2064, p90 usec: 4074, p99 usec: 6024, ok: 165169, error: 0
PlainGrpc threads=128, throughput: 31428, per thread: 245, p50 usec: 4583, p90 usec: 9391, p99 usec: 13748, ok: 157140, error: 0