I benchmarked Rocket framework, its result is SHOCKING 😱😞

[rousan]

Questions

I benchmarked it with other frameworks e.g. gotham, hyper and actix-web and Rocket's result was not expected at all.

hyper: Requests/sec: 112557.51 and Latency: 1.77ms

gotham: Requests/sec: 100097.73 and Latency: 1.98ms

actix-web: Requests/sec: 96397.31 and Latency: 2.06ms

Rocket: Requests/sec: 2041.93 😱 and Latency: 3.37ms

See?
For Rocket, it just 2k/sec where hyper or gotham are 112k/sec and 100k/sec.

For the benchmark codebase, please visit: https://github.com/rousan/rust-web-frameworks-benchmark.

And correct me, if I am not using it correctly.

[martynp]

There was a similar discussion on reddit with the async features (https://reddit.app.link/8AGkZT2Zy4). Cursory look seems to have the same order of magnitude differences.

[jebrosen]

Closing as a duplicate of #799, #834, #1023, #1038.

Any comparison between an async and non-async web framework is expected to look something like this, and on top of that rocket 0.4/hyper 0.10 has a few known bugs related to keep-alive that confuse benchmark results.

0.5 (today the async branch) will use the latest version of hyper and show dramatic improvements over 0.4.

[rousan]

@jebrosen , I also did the same benchmark with async branch, getting almost the same results. Can you please share some benchmark results with actix web or other web frameworks?

[jebrosen]

Sure, I can share these results as a sanity check. Both were run with cargo run --release on rousan/rust-web-frameworks-benchmark @ commit 0dcc71c, timed with wrk --latency -t4 -c200 -d8s http://127.0.0.1:8081, on an AMD Ryzen 3700U.

hyper (0.13.2):

Running 8s test @ http://127.0.0.1:8082
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.95ms    0.93ms  20.79ms   96.02%
    Req/Sec    56.87k     6.95k   75.41k    76.88%
  Latency Distribution
     50%  809.00us
     75%    1.07ms
     90%    1.37ms
     99%    4.89ms
  1811869 requests in 8.06s, 152.06MB read
Requests/sec: 224912.14
Transfer/sec:     18.88MB

rocket (async branch, rev 78c8ac8):

Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.72ms    1.92ms  66.46ms   90.48%
    Req/Sec    33.07k     5.47k   46.89k    69.38%
  Latency Distribution
     50%    1.21ms
     75%    2.13ms
     90%    3.55ms
     99%    9.03ms
  1055880 requests in 8.08s, 147.02MB read
Requests/sec: 130756.85
Transfer/sec:     18.21MB

The different number of requests despite the same transfer rate probably has to do with the Rocket server sending an additional Server header and also a Content-Type header (which is about three times the size of the actual response body), compared to hyper.