-
Notifications
You must be signed in to change notification settings - Fork 469
ci: update benchmark slo + reporting dependencies #15339
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Only run SLO check if the benchmark jobs all succeeded. Only run the reporting if the SLO check job ran.
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 261 ± 5 ms. The average import time from base is: 269 ± 5 ms. The import time difference between this PR and base is: -8.0 ± 0.2 ms. Import time breakdownThe following import paths have shrunk:
|
igoragoli
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good from APM DCS Performance.
Summarizing a discussion I had with @brettlangdon for future readers:
when: always is useful on other other repos where it's possible to merge a PR/release with a failing benchmarking job, so we rely on the SLO check for a general gating mechanism.
On the other hand, on dd-trace-py this is not possible, so if a microbenchmarking job fails, the SLO check is not necessary since we are not going to be able to merge the PR anyways.
Description
Only run SLO check if the benchmark jobs all succeeded.
Only run the reporting if the SLO check job ran.
Right now if a benchmarks fails (programming error, or infra issues, etc), we still run the SLO check, which we know will fail because we don't report all benchmark results, then we also run the benchmark reporter which will succeed. If you then go back and re-run the failed benchmark job (assuming flaky infra, for example), then you have to know to manually re-run the SLO check and the reporting job as well).
This should simplify and clarify some of it a little bit.
Testing
Risks
Additional Notes