Add scripts/plot.py for comparing latencies, etc. #3327
Conversation
This can be used to visualize performance changes between two or more branches, across several categories (e.g. requests-per-second load). While not perfect I think the output goes a long way to helping the user absorb a lot of information about the data, while avoiding drawing wrong inferences, e.g. paying too much attention to mean, or forgetting how noisey the data is.
|
@nizar-m let me know what you think of this; |
|
Deploy preview for hasura-docs ready! Built with commit 283dec9 |
|
Definitely very interesting plots. Why the Indian oil lamp like plots for latencies, I have no idea. We should have this for our latency plots. |
These are "violin plots" (maybe they have other names too), and technically speaking graph https://en.wikipedia.org/wiki/Kernel_density_estimation but you can think of them like a continuous histogram; so places where the violin is very thick are areas that are very dense with samples (above these are unsurprisingly right around the median represented by the thin black line). |
- no need to add category and variant labels on command line - produce an html table of median and min values, with highlighting based on pct difference - Flag --baseline-variant affects above - color sample dots to highlight drift - put output files in input data directory
|
Closing this for now, as I think it's going to live in its own repo. |
|
Review app https://hge-ci-pull-3327.herokuapp.com is deleted |
|
Development continues here for this: https://github.com/hasura/latency-plotting |
This can be used to visualize performance changes between two or more
branches, across several categories (e.g. requests-per-second load).
While not perfect I think the output goes a long way to helping the user
absorb a lot of information about the data, while avoiding drawing wrong
inferences, e.g. paying too much attention to mean, or forgetting how
noisey the data is.
This will hopefully be integrated into some benchmarking scripts, and improved, probably in collaboration with Nizar's work in e.g.
tirumaraiselvan#70 , and...
#3310 (review)
Wanted to share this ASAP though.
Steps to test and verify
See README.md
Example
The graph offers:
Tentative plan is to use this to plot internal service timing, and combine it with some hdr histogram timing from wrk2 (the client), so we can see where throughput is actually making things fall over.