Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic performance benchmark using criterion for pub sub and req rep #81

Merged
merged 6 commits into from
Oct 12, 2020
Merged

Basic performance benchmark using criterion for pub sub and req rep #81

merged 6 commits into from
Oct 12, 2020

Conversation

TheButlah
Copy link
Contributor

@TheButlah TheButlah commented Oct 12, 2020

Resolves #80

This PR adds a basic performance benchmark using criterion.rs.

Currently, it hangs on the first s.recv() in the recv_msgs() function. I can't figure out why it is hanging here. Any ideas @Alexei-Kornienko?

@TheButlah TheButlah marked this pull request as draft October 12, 2020 12:33
@TheButlah TheButlah changed the title Basic perf bench using criterion for pub sub Basic performance bench using criterion for pub sub Oct 12, 2020
@TheButlah TheButlah changed the title Basic performance bench using criterion for pub sub Basic performance benchmark using criterion for pub sub Oct 12, 2020
@Alexei-Kornienko
Copy link
Collaborator

@TheButlah IMO PUB/SUB (especially 10x of them) could be a bit hard to start with. I would suggest to start with some kind of really simple scenario like 1 REQ - 1 REP. This is much easier to setup and debug. Currently I don't have any idea why it blocks. I can take a closer look on it on a weekend. Currently I have to write code for money not for fun :(

@TheButlah TheButlah changed the title Basic performance benchmark using criterion for pub sub Basic performance benchmark using criterion for pub sub and req rep Oct 12, 2020
@TheButlah TheButlah marked this pull request as ready for review October 12, 2020 15:03
@TheButlah
Copy link
Contributor Author

TheButlah commented Oct 12, 2020

OK, I've added a benchmark for the req rep pattern. As you predicted, it was a lot easier than Pub Sub, and unlike the other one, it actually works. I've left the pub sub benchmark code in this PR, but have commented it out from the Cargo.toml so that it will not be run on cargo bench. We can try to figure out why the SubSocket::recv() blocks another time.

A couple of observations I can make: Even with a sample size of 128 and a benchmark time of 32 seconds with 5 secs warmup time (that gives 512 non-warmup iterations), criteron frequently reports statistically significant changes in performance, when no such changes should be present. I can only assume that this is because of the fluctuating performance of the underlyling tcp sockets on the host computer.

We may want to mock these out from tokio in the future, to avoid this source of nondeterminism

@Alexei-Kornienko
Copy link
Collaborator

@TheButlah I've tested this branch on my PC. I also see fluctuations but it seems that they are mostly related to my workload and Youtube playing in background.. On idle PC I have 3% or less difference between runs. Looks like a good start. We can run the same tests for your codec branch and see is there any difference

@Alexei-Kornienko Alexei-Kornienko merged commit f2eac8b into zeromq:master Oct 12, 2020
@TheButlah TheButlah deleted the basic_perf_bench branch October 12, 2020 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement basic performance benchmark
2 participants