Perf release gate #9068

bric3 · 2025-06-30T16:59:23Z

What Does This Do

Check then notify if the release do not meet expected thresholds.

Motivation

Ensure performance thresholds.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-30T17:33:00Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752246830	1752248657
git_commit_sha	`53f5d20`	`67435a9`
release_version	1.52.0-SNAPSHOT~53f5d20482	1.52.0-SNAPSHOT~67435a9ab7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752250690	1752250690
ci_job_id	1025486071	1025486071
ci_pipeline_id	70325228	70325228
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-rbcipry8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-rbcipry8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 8 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (997.575 ms) : 0, 997575
Total [baseline] (8.568 s) : 0, 8568381
Agent [candidate] (994.156 ms) : 0, 994156
Total [candidate] (8.578 s) : 0, 8577694
section iast
Agent [baseline] (1.132 s) : 0, 1132138
Total [baseline] (9.297 s) : 0, 9297156
Agent [candidate] (1.131 s) : 0, 1131278
Total [candidate] (9.253 s) : 0, 9253305

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	997.575 ms	-
Agent	iast	1.132 s	134.563 ms (13.5%)
Total	tracing	8.568 s	-
Total	iast	9.297 s	728.775 ms (8.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	994.156 ms	-
Agent	iast	1.131 s	137.122 ms (13.8%)
Total	tracing	8.578 s	-
Total	iast	9.253 s	675.611 ms (7.9%)

gantt
    title insecure-bank - break down per module: candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (689.507 ms) : 0, 689507
BytebuddyAgent [candidate] (686.363 ms) : 0, 686363
GlobalTracer [baseline] (242.079 ms) : 0, 242079
GlobalTracer [candidate] (241.901 ms) : 0, 241901
AppSec [baseline] (30.29 ms) : 0, 30290
AppSec [candidate] (30.228 ms) : 0, 30228
Debugger [baseline] (5.985 ms) : 0, 5985
Debugger [candidate] (6.014 ms) : 0, 6014
Remote Config [baseline] (672.181 µs) : 0, 672
Remote Config [candidate] (675.39 µs) : 0, 675
Telemetry [baseline] (8.2 ms) : 0, 8200
Telemetry [candidate] (8.203 ms) : 0, 8203
section iast
BytebuddyAgent [baseline] (807.719 ms) : 0, 807719
BytebuddyAgent [candidate] (806.161 ms) : 0, 806161
GlobalTracer [baseline] (231.923 ms) : 0, 231923
GlobalTracer [candidate] (231.9 ms) : 0, 231900
IAST [baseline] (26.273 ms) : 0, 26273
IAST [candidate] (24.819 ms) : 0, 24819
AppSec [baseline] (31.106 ms) : 0, 31106
AppSec [candidate] (31.558 ms) : 0, 31558
Debugger [baseline] (5.716 ms) : 0, 5716
Debugger [candidate] (7.511 ms) : 0, 7511
Remote Config [baseline] (577.34 µs) : 0, 577
Remote Config [candidate] (586.869 µs) : 0, 587
Telemetry [baseline] (7.948 ms) : 0, 7948
Telemetry [candidate] (7.884 ms) : 0, 7884

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (996.852 ms) : 0, 996852
Total [baseline] (10.739 s) : 0, 10739208
Agent [candidate] (1.002 s) : 0, 1002075
Total [candidate] (10.691 s) : 0, 10691005
section appsec
Agent [baseline] (1.176 s) : 0, 1175715
Total [baseline] (10.773 s) : 0, 10772711
Agent [candidate] (1.175 s) : 0, 1175437
Total [candidate] (10.734 s) : 0, 10734101
section iast
Agent [baseline] (1.135 s) : 0, 1135320
Total [baseline] (10.919 s) : 0, 10918534
Agent [candidate] (1.13 s) : 0, 1130247
Total [candidate] (10.816 s) : 0, 10815984
section profiling
Agent [baseline] (1.242 s) : 0, 1242053
Total [baseline] (10.942 s) : 0, 10942408
Agent [candidate] (1.242 s) : 0, 1242432
Total [candidate] (10.88 s) : 0, 10880495

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	996.852 ms	-
Agent	appsec	1.176 s	178.863 ms (17.9%)
Agent	iast	1.135 s	138.468 ms (13.9%)
Agent	profiling	1.242 s	245.201 ms (24.6%)
Total	tracing	10.739 s	-
Total	appsec	10.773 s	33.504 ms (0.3%)
Total	iast	10.919 s	179.326 ms (1.7%)
Total	profiling	10.942 s	203.2 ms (1.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.002 s	-
Agent	appsec	1.175 s	173.362 ms (17.3%)
Agent	iast	1.13 s	128.172 ms (12.8%)
Agent	profiling	1.242 s	240.357 ms (24.0%)
Total	tracing	10.691 s	-
Total	appsec	10.734 s	43.096 ms (0.4%)
Total	iast	10.816 s	124.979 ms (1.2%)
Total	profiling	10.88 s	189.49 ms (1.8%)

gantt
    title petclinic - break down per module: candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (687.633 ms) : 0, 687633
BytebuddyAgent [candidate] (692.378 ms) : 0, 692378
GlobalTracer [baseline] (243.066 ms) : 0, 243066
GlobalTracer [candidate] (243.127 ms) : 0, 243127
AppSec [baseline] (30.427 ms) : 0, 30427
AppSec [candidate] (30.511 ms) : 0, 30511
Debugger [baseline] (6.017 ms) : 0, 6017
Debugger [candidate] (6.039 ms) : 0, 6039
Remote Config [baseline] (682.392 µs) : 0, 682
Remote Config [candidate] (682.258 µs) : 0, 682
Telemetry [baseline] (8.241 ms) : 0, 8241
Telemetry [candidate] (8.256 ms) : 0, 8256
section appsec
BytebuddyAgent [baseline] (710.864 ms) : 0, 710864
BytebuddyAgent [candidate] (710.817 ms) : 0, 710817
GlobalTracer [baseline] (235.002 ms) : 0, 235002
GlobalTracer [candidate] (235.012 ms) : 0, 235012
IAST [baseline] (23.641 ms) : 0, 23641
IAST [candidate] (23.523 ms) : 0, 23523
AppSec [baseline] (170.839 ms) : 0, 170839
AppSec [candidate] (170.642 ms) : 0, 170642
Debugger [baseline] (5.729 ms) : 0, 5729
Debugger [candidate] (5.76 ms) : 0, 5760
Remote Config [baseline] (606.701 µs) : 0, 607
Remote Config [candidate] (603.344 µs) : 0, 603
Telemetry [baseline] (8.073 ms) : 0, 8073
Telemetry [candidate] (8.132 ms) : 0, 8132
section iast
BytebuddyAgent [baseline] (807.962 ms) : 0, 807962
BytebuddyAgent [candidate] (805.74 ms) : 0, 805740
GlobalTracer [baseline] (233.413 ms) : 0, 233413
GlobalTracer [candidate] (231.698 ms) : 0, 231698
IAST [baseline] (26.905 ms) : 0, 26905
IAST [candidate] (27.481 ms) : 0, 27481
AppSec [baseline] (30.048 ms) : 0, 30048
AppSec [candidate] (29.364 ms) : 0, 29364
Debugger [baseline] (7.501 ms) : 0, 7501
Debugger [candidate] (6.662 ms) : 0, 6662
Remote Config [baseline] (588.275 µs) : 0, 588
Remote Config [candidate] (575.729 µs) : 0, 576
Telemetry [baseline] (7.98 ms) : 0, 7980
Telemetry [candidate] (7.928 ms) : 0, 7928
section profiling
BytebuddyAgent [baseline] (676.376 ms) : 0, 676376
BytebuddyAgent [candidate] (676.4 ms) : 0, 676400
GlobalTracer [baseline] (360.683 ms) : 0, 360683
GlobalTracer [candidate] (361.271 ms) : 0, 361271
AppSec [baseline] (30.92 ms) : 0, 30920
AppSec [candidate] (31.526 ms) : 0, 31526
Debugger [baseline] (11.303 ms) : 0, 11303
Debugger [candidate] (11.271 ms) : 0, 11271
Remote Config [baseline] (676.274 µs) : 0, 676
Remote Config [candidate] (671.298 µs) : 0, 671
Telemetry [baseline] (10.246 ms) : 0, 10246
Telemetry [candidate] (8.896 ms) : 0, 8896
ProfilingAgent [baseline] (103.295 ms) : 0, 103295
ProfilingAgent [candidate] (103.858 ms) : 0, 103858
Profiling [baseline] (103.319 ms) : 0, 103319
Profiling [candidate] (103.883 ms) : 0, 103883

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752246830	1752248657
git_commit_sha	`53f5d20`	`67435a9`
release_version	1.52.0-SNAPSHOT~53f5d20482	1.52.0-SNAPSHOT~67435a9ab7

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752250370	1752250370
ci_job_id	1025486073	1025486073
ci_pipeline_id	70325228	70325228
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-zm4xa90u 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-zm4xa90u 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-594.184µs; -290.361µs] or [-6.700%; -3.274%]	unstable [-42.791op/s; +96.979op/s] or [-8.181%; +18.542%]	8.427ms	550.125op/s	8.869ms	523.031op/s
scenario:load:petclinic:no_agent:high_load	better [-3.183ms; -2.548ms] or [-8.236%; -6.594%]	unstable [+0.751op/s; +18.474op/s] or [+0.621%; +15.268%]	35.780ms	130.613op/s	38.646ms	121.000op/s
scenario:load:petclinic:iast:high_load	better [-1.813ms; -0.957ms] or [-3.976%; -2.098%]	unstable [-5.528op/s; +9.330op/s] or [-5.320%; +8.980%]	44.219ms	105.800op/s	45.604ms	103.899op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482
    dateFormat X
    axisFormat %s
section baseline
no_agent (38.646 ms) : 38343, 38949
.   : milestone, 38646,
appsec (46.886 ms) : 46480, 47291
.   : milestone, 46886,
code_origins (45.829 ms) : 45434, 46223
.   : milestone, 45829,
iast (45.604 ms) : 45205, 46003
.   : milestone, 45604,
profiling (48.435 ms) : 47970, 48899
.   : milestone, 48435,
tracing (43.317 ms) : 42950, 43685
.   : milestone, 43317,
section candidate
no_agent (35.78 ms) : 35493, 36067
.   : milestone, 35780,
appsec (46.616 ms) : 46202, 47031
.   : milestone, 46616,
code_origins (45.676 ms) : 45293, 46058
.   : milestone, 45676,
iast (44.219 ms) : 43822, 44616
.   : milestone, 44219,
profiling (48.833 ms) : 48386, 49281
.   : milestone, 48833,
tracing (43.785 ms) : 43429, 44141
.   : milestone, 43785,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	38.646 ms [38.343 ms, 38.949 ms]	-
appsec	46.886 ms [46.48 ms, 47.291 ms]	8.24 ms (21.3%)
code_origins	45.829 ms [45.434 ms, 46.223 ms]	7.183 ms (18.6%)
iast	45.604 ms [45.205 ms, 46.003 ms]	6.959 ms (18.0%)
profiling	48.435 ms [47.97 ms, 48.899 ms]	9.789 ms (25.3%)
tracing	43.317 ms [42.95 ms, 43.685 ms]	4.672 ms (12.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	35.78 ms [35.493 ms, 36.067 ms]	-
appsec	46.616 ms [46.202 ms, 47.031 ms]	10.836 ms (30.3%)
code_origins	45.676 ms [45.293 ms, 46.058 ms]	9.895 ms (27.7%)
iast	44.219 ms [43.822 ms, 44.616 ms]	8.439 ms (23.6%)
profiling	48.833 ms [48.386 ms, 49.281 ms]	13.053 ms (36.5%)
tracing	43.785 ms [43.429 ms, 44.141 ms]	8.005 ms (22.4%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.361 ms) : 4312, 4411
.   : milestone, 4361,
iast (9.173 ms) : 9023, 9322
.   : milestone, 9173,
iast_FULL (13.936 ms) : 13659, 14212
.   : milestone, 13936,
iast_GLOBAL (10.668 ms) : 10477, 10860
.   : milestone, 10668,
profiling (8.869 ms) : 8722, 9015
.   : milestone, 8869,
tracing (7.558 ms) : 7453, 7663
.   : milestone, 7558,
section candidate
no_agent (4.485 ms) : 4427, 4542
.   : milestone, 4485,
iast (9.378 ms) : 9228, 9529
.   : milestone, 9378,
iast_FULL (13.44 ms) : 13174, 13706
.   : milestone, 13440,
iast_GLOBAL (10.426 ms) : 10227, 10624
.   : milestone, 10426,
profiling (8.427 ms) : 8291, 8562
.   : milestone, 8427,
tracing (7.378 ms) : 7276, 7481
.   : milestone, 7378,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.361 ms [4.312 ms, 4.411 ms]	-
iast	9.173 ms [9.023 ms, 9.322 ms]	4.811 ms (110.3%)
iast_FULL	13.936 ms [13.659 ms, 14.212 ms]	9.574 ms (219.5%)
iast_GLOBAL	10.668 ms [10.477 ms, 10.86 ms]	6.307 ms (144.6%)
profiling	8.869 ms [8.722 ms, 9.015 ms]	4.508 ms (103.4%)
tracing	7.558 ms [7.453 ms, 7.663 ms]	3.197 ms (73.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.485 ms [4.427 ms, 4.542 ms]	-
iast	9.378 ms [9.228 ms, 9.529 ms]	4.894 ms (109.1%)
iast_FULL	13.44 ms [13.174 ms, 13.706 ms]	8.955 ms (199.7%)
iast_GLOBAL	10.426 ms [10.227 ms, 10.624 ms]	5.941 ms (132.5%)
profiling	8.427 ms [8.291 ms, 8.562 ms]	3.942 ms (87.9%)
tracing	7.378 ms [7.276 ms, 7.481 ms]	2.893 ms (64.5%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752246830	1752248657
git_commit_sha	`53f5d20`	`67435a9`
release_version	1.52.0-SNAPSHOT~53f5d20482	1.52.0-SNAPSHOT~67435a9ab7

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1752250915	1752250915
ci_job_id	1025486074	1025486074
ci_pipeline_id	70325228	70325228
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-7xhsy493 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-7xhsy493 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.02 s) : 15020000, 15020000
.   : milestone, 15020000,
appsec (14.706 s) : 14706000, 14706000
.   : milestone, 14706000,
iast (18.601 s) : 18601000, 18601000
.   : milestone, 18601000,
iast_GLOBAL (17.946 s) : 17946000, 17946000
.   : milestone, 17946000,
profiling (15.714 s) : 15714000, 15714000
.   : milestone, 15714000,
tracing (14.838 s) : 14838000, 14838000
.   : milestone, 14838000,
section candidate
no_agent (15.559 s) : 15559000, 15559000
.   : milestone, 15559000,
appsec (14.852 s) : 14852000, 14852000
.   : milestone, 14852000,
iast (18.374 s) : 18374000, 18374000
.   : milestone, 18374000,
iast_GLOBAL (17.885 s) : 17885000, 17885000
.   : milestone, 17885000,
profiling (15.547 s) : 15547000, 15547000
.   : milestone, 15547000,
tracing (14.889 s) : 14889000, 14889000
.   : milestone, 14889000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.02 s [15.02 s, 15.02 s]	-
appsec	14.706 s [14.706 s, 14.706 s]	-314.0 ms (-2.1%)
iast	18.601 s [18.601 s, 18.601 s]	3.581 s (23.8%)
iast_GLOBAL	17.946 s [17.946 s, 17.946 s]	2.926 s (19.5%)
profiling	15.714 s [15.714 s, 15.714 s]	694.0 ms (4.6%)
tracing	14.838 s [14.838 s, 14.838 s]	-182.0 ms (-1.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.559 s [15.559 s, 15.559 s]	-
appsec	14.852 s [14.852 s, 14.852 s]	-707.0 ms (-4.5%)
iast	18.374 s [18.374 s, 18.374 s]	2.815 s (18.1%)
iast_GLOBAL	17.885 s [17.885 s, 17.885 s]	2.326 s (14.9%)
profiling	15.547 s [15.547 s, 15.547 s]	-12.0 ms (-0.1%)
tracing	14.889 s [14.889 s, 14.889 s]	-670.0 ms (-4.3%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~67435a9ab7, baseline=1.52.0-SNAPSHOT~53f5d20482
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.476 ms) : 1465, 1488
.   : milestone, 1476,
appsec (2.422 ms) : 2372, 2472
.   : milestone, 2422,
iast (2.198 ms) : 2136, 2260
.   : milestone, 2198,
iast_GLOBAL (2.234 ms) : 2171, 2297
.   : milestone, 2234,
profiling (2.062 ms) : 2010, 2113
.   : milestone, 2062,
tracing (2.018 ms) : 1970, 2066
.   : milestone, 2018,
section candidate
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (2.416 ms) : 2366, 2466
.   : milestone, 2416,
iast (2.198 ms) : 2136, 2260
.   : milestone, 2198,
iast_GLOBAL (2.24 ms) : 2178, 2303
.   : milestone, 2240,
profiling (2.059 ms) : 2008, 2111
.   : milestone, 2059,
tracing (2.028 ms) : 1979, 2077
.   : milestone, 2028,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	2.422 ms [2.372 ms, 2.472 ms]	945.41 µs (64.0%)
iast	2.198 ms [2.136 ms, 2.26 ms]	721.771 µs (48.9%)
iast_GLOBAL	2.234 ms [2.171 ms, 2.297 ms]	757.782 µs (51.3%)
profiling	2.062 ms [2.01 ms, 2.113 ms]	585.305 µs (39.6%)
tracing	2.018 ms [1.97 ms, 2.066 ms]	541.869 µs (36.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.472 ms [1.461 ms, 1.484 ms]	-
appsec	2.416 ms [2.366 ms, 2.466 ms]	943.864 µs (64.1%)
iast	2.198 ms [2.136 ms, 2.26 ms]	725.546 µs (49.3%)
iast_GLOBAL	2.24 ms [2.178 ms, 2.303 ms]	768.138 µs (52.2%)
profiling	2.059 ms [2.008 ms, 2.111 ms]	587.092 µs (39.9%)
tracing	2.028 ms [1.979 ms, 2.077 ms]	555.395 µs (37.7%)

ddyurchenko

Looks good from my side! 🎉
Need also @igoragoli review for the final approval.

igoragoli

Thanks @bric3! 🙌

There are just some points regarding startup:petclinic.* thresholds that need to be addressed before merging.

.gitlab/benchmarks/bp-runner.fail-on-breach.yml

.gitlab/macrobenchmarks.yml

…can debug

igoragoli

Looks good, thanks Brice!

I think it's a good idea to include the source for the SLOs in the thresholds file, nice.

bric3 · 2025-07-11T14:12:06Z

.gitlab/macrobenchmarks.yml

-      when: always
-    - when: manual
-      allow_failure: true
+    - when: on_success  # TODO: PLEASE revert before merging the PR


todo:

To revert before merging

ddyurchenko · 2025-07-14T15:30:27Z

Still see some yellow in results, so while job won't block releases, it will send warning messages in Java guild channel.
Augusto practically finalized the changes to reporting (so now confidence intervals and thresholds are clearly displayed, incl. warning threshold). I propose to rebuild Java image, so the changes are included, and update the SLOs once more for p50 latency, p99 latency and startup time mean value (execution_time metric), so they are no longer in yellow zone.

SLO breach check  | 
SLO breach check  | #### high_load--only-tracing
SLO breach check  | 
SLO breach check  | - 🟩 `throughput` 1[250](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1025486085#L250).55 op/s > 1100.00 op/s
SLO breach check  | 
SLO breach check  | #### high_load--otel-latest
SLO breach check  | 
SLO breach check  | - 🟩 `throughput` 1245.11 op/s > 1100.00 op/s
SLO breach check  | 
SLO breach check  | #### normal_operation--only-tracing
SLO breach check  | 
SLO breach check  | - 🟩 `agg_http_req_duration_p50` 2.12 ms < 2.36 ms
SLO breach check  | - 🟨 `agg_http_req_duration_p99` 7.10 ms < 7.89 ms
SLO breach check  | 
SLO breach check  | #### normal_operation--otel-latest
SLO breach check  | 
SLO breach check  | - 🟨 `agg_http_req_duration_p50` 2.12 ms < 2.34 ms
SLO breach check  | - 🟨 `agg_http_req_duration_p99` 8.75 ms < 9.50 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:appsec:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟨 `execution_time` 235.01 ms < [260](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1025486085#L260).00 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:iast:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟩 `execution_time` 231.70 ms < 260.00 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:profiling:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟨 `execution_time` 361.27 ms < 368.00 ms
SLO breach check  | 
SLO breach check  | #### startup:petclinic:tracing:GlobalTracer
SLO breach check  | 
SLO breach check  | - 🟨 `execution_time` 243.13 ms < 260.00 ms
SLO breach check  | 
SLO breach check  | ---
SLO breach check  | 
SLO breach check  | Legend:
SLO breach check  | - 🟩 pass
SLO breach check  | - 🟥 breach
SLO breach check  | - 🟨 warning
SLO breach check  | - (unstable) unstable

igoragoli · 2025-07-16T12:16:09Z

Augusto practically finalized the changes to reporting

Changes for reporting done! 🙂

I'm updating the registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:latest image (the one used in the check-slo-breaches job across the board) here: https://gitlab.ddbuild.io/DataDog/benchmarking-platform-tools/-/jobs/1031270236

ddyurchenko · 2025-07-16T13:35:43Z

Thanks @igoragoli ! I reran the job https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1031460223, based on its results, will provide suggestions to @bric3 .

ddyurchenko · 2025-07-16T13:39:14Z

.gitlab/benchmarks/bp-runner.fail-on-breach.yml

+        #   https://datadoghq.atlassian.net/wiki/x/LgI1LgE#Specification
+        # Measurements
+        #   https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario
+        scenarios:


Based on the results of https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1031460223, I suggest to update the SLOs to the following values:

# Standard macrobenchmarks # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fonly-tracing&trendsType=scenario - name: normal_operation/only-tracing thresholds: - agg_http_req_duration_p50 < 2.36 ms - agg_http_req_duration_p99 < 7.89 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fotel-latest&trendsType=scenario - name: normal_operation/otel-latest thresholds: - agg_http_req_duration_p50 < 2.5 ms - agg_http_req_duration_p99 < 10 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=high_load%2Fonly-tracing&trendsType=scenario - name: high_load/only-tracing thresholds: - throughput > 1100.0 op/s # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=high_load%2Fotel-latest&trendsType=scenario - name: high_load/otel-latest thresholds: - throughput > 1100.0 op/s # Startup macrobenchmarks # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Atracing%3AGlobalTracer&trendsType=scenario # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aappsec%3AGlobalTracer&trendsType=scenario # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aiast%3AGlobalTracer&trendsType=scenario - name: "startup:petclinic:(tracing|appsec|iast):GlobalTracer" thresholds: - execution_time < 280 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aprofiling%3AGlobalTracer&trendsType=scenario - name: "startup:petclinic:profiling:GlobalTracer" thresholds: - execution_time < 420 ms

ddyurchenko

Need to update some of SLO thresholds before merge.

bric3 requested a review from a team as a code owner June 30, 2025 16:59

bric3 requested review from colin-higgins and removed request for a team June 30, 2025 16:59

bric3 added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jun 30, 2025

bric3 marked this pull request as draft June 30, 2025 16:59

bric3 changed the title ~~chore(ci): Basic slo breach prototype~~ Perf release gate Jun 30, 2025

ddyurchenko approved these changes Jul 1, 2025

View reviewed changes

ddyurchenko requested a review from igoragoli July 1, 2025 13:43

bric3 requested a review from a team July 1, 2025 14:07

igoragoli requested changes Jul 1, 2025

View reviewed changes

.gitlab/benchmarks/bp-runner.fail-on-breach.yml Outdated Show resolved Hide resolved

.gitlab/benchmarks/bp-runner.fail-on-breach.yml Outdated Show resolved Hide resolved

.gitlab/macrobenchmarks.yml Show resolved Hide resolved

bric3 requested a review from a team July 2, 2025 15:24

bric3 force-pushed the bdu/r-gate branch 7 times, most recently from 7e61d82 to feb60ef Compare July 10, 2025 12:48

bric3 added 3 commits July 11, 2025 11:35

chore(ci): Basic slo breach prototype

fdd30e5

chore(ci): PR review

6407c42

chore: Collect reports from benchmarks

736aefd

bric3 force-pushed the bdu/r-gate branch from feb60ef to 736aefd Compare July 11, 2025 09:35

ddyurchenko added 2 commits July 11, 2025 15:28

tweak: Include benchmarkType in file name

361aaf6

tweak: Store artifacts that were used for checking regression, so we …

92f9783

…can debug

ddyurchenko self-requested a review July 11, 2025 14:39

chore(ci): Tweak thresholds

67435a9

bric3 marked this pull request as ready for review July 11, 2025 15:45

igoragoli approved these changes Jul 11, 2025

View reviewed changes

bric3 commented Jul 11, 2025

View reviewed changes

ddyurchenko reviewed Jul 16, 2025

View reviewed changes

ddyurchenko requested changes Jul 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Perf release gate #9068

Perf release gate #9068

bric3 commented Jun 30, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Jun 30, 2025 •

edited

Loading

Uh oh!

ddyurchenko left a comment

Uh oh!

igoragoli left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igoragoli left a comment

Uh oh!

bric3 Jul 11, 2025

Uh oh!

ddyurchenko commented Jul 14, 2025

Uh oh!

igoragoli commented Jul 16, 2025 •

edited

Loading

Uh oh!

ddyurchenko commented Jul 16, 2025

Uh oh!

ddyurchenko Jul 16, 2025

Uh oh!

ddyurchenko left a comment

Uh oh!

Uh oh!

Perf release gate #9068

Are you sure you want to change the base?

Perf release gate #9068

Conversation

bric3 commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

ddyurchenko left a comment

Choose a reason for hiding this comment

Uh oh!

igoragoli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

igoragoli left a comment

Choose a reason for hiding this comment

Uh oh!

bric3 Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

ddyurchenko commented Jul 14, 2025

Uh oh!

igoragoli commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddyurchenko commented Jul 16, 2025

Uh oh!

ddyurchenko Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

ddyurchenko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bric3 commented Jun 30, 2025 •

edited

Loading

pr-commenter bot commented Jun 30, 2025 •

edited

Loading

igoragoli commented Jul 16, 2025 •

edited

Loading