Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

16-core (WinterJS) vs. 1-core (Node,Bun) benchmarking is not a benchmarking, it's finger painting #54

Closed
uNetworkingAB opened this issue Mar 15, 2024 · 7 comments

Comments

@uNetworkingAB
Copy link

uNetworkingAB commented Mar 15, 2024

Edit:

never mind, I see now that benchmarks are multi-core vs. single-core so completely pointless to even bother communicating anything re. pref.

@uNetworkingAB uNetworkingAB changed the title There are better performance references 16-core (WinterJS) vs. 1-core (Node,Bun) benchmarking is not a benchmarking, it's finger painting Mar 15, 2024
@Arshia001
Copy link
Member

@uNetworkingAB yes, we've heard about the bun issue from others as well. At the time when we were running the benchmarks, we didn't know bun was single-threaded by default. I'll keep this issue open until we update the numbers. CC @syrusakbary

@uNetworkingAB
Copy link
Author

Nodejs, Bun, Deno all 3 are single-threaded by default it's a core design that should be obvious. How are you going to correct this misinformation? The blog post still holds widely invalid numbers

@syrusakbary
Copy link
Member

Hey @uNetworkingAB,

We are testing all runtimes, including Node and Bun with their default settings. Shall they upgrade them, shall the benchmarks be updated. If you want other runtimes to use multiple threads, then I'd suggest prompting that on their own repos.

In general, I'd say that the benchmark that we are currently doing is quite limited (as almost any other benchmark that you will find in the wild).
Mainly because we are testing literally sending the most simple response ("hello") through it, which in general measures the throughput of the event loop & http server, not JS or the integration.

The benchmark will be greatly improved by testing a full end-to-end scenario, such as number of renders of React per second, which is probably the way to go for benchmarking. I've created a ticket for it to make sure we follow up on that: #56.

But, to be as clear as possible: we will not limit the benchmark to use only one thread just because other runtimes can't currently handle it properly.
We believe that, by default, most runtimes should be able to be run multithreaded, specially for WinterCG environments where the context doesn't need to be shared in between requests.

@uNetworkingAB
Copy link
Author

with their default settings. Shall they upgrade them, shall the benchmarks be updated. If you want other runtimes to use multiple threads, then I'd suggest prompting that on their own repos.

So you expect Node.js, a 15 year old single-threaded runtime, to change its default to multi-core in order to properly benchmark against it? You want Node.js, highly used runtime, to change to fit your benchmark rather than take the 5 minutes it takes to set up a Node.js cluster and do apples to apples comparison?

You want the world to fit your benchmark rather than make your benchmark fit the world. Nice confidence, but that's never going to happen

@eeshankeni
Copy link

bro sounds like he doesn't wanna do actual benchmarks cause deep inside he knows his claims are half baked at best

@AddictArts
Copy link

AddictArts commented Mar 25, 2024

It is fair to compare default out of the box running. They just needed to be clear about that and what they compared and how. The dev response didn't say the other runtimes needed to change. They said, "shall they", just a question. I don't believe they have the burden to prove anything, they should make any claim clear and how they got it. I don't believe the blog post did that well, so questioning the results was also fair. I don't know if the replies were bun fans, but too often the reaction when bun is shown as less than, it is over the top and looks bad. I'd rather see bun just "walk the walk", show it, let that speak for itself. It is flattering to see bun as the benchmark to compare to, that means something, free advertising.

@uNetworkingAB
Copy link
Author

Node.js doesn't even have a server by default. It doesn't listen to any port by default and there is no connecting client by default and no attached response.

Winterjs authors explicitly and manually made the competition use a 16th the CPU-time, by explicitly and manually writing the boiler plate code that achieves single thread response, rather than explicitly and manually picking the built-in Cluster module boiler plate code.

None of this is honest, accurate or motivatable.

It's the equivalence of running a manually gered car against an automatically geared one, only running the manual on first fear then posting general numbers representing the car as a whole.

But I have noticed common sense is a rarity nowadays so I wouldn't be shocked people disagree.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants