Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix KafkaStreamsTest #8638

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Fix KafkaStreamsTest #8638

wants to merge 2 commits into from

Conversation

smola
Copy link
Member

@smola smola commented Mar 28, 2025

What Does This Do

Fix KafkaStreamsTest. This test was broken after it was marked as @Flaky. Not detected because these do not run in the main CI pipelines.

Test with:

./gradlew :dd-java-agent:instrumentation:kafka-streams-0.11:latestDepTest

Motivation

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@smola smola requested a review from a team March 28, 2025 14:49
@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Mar 28, 2025
@pr-commenter
Copy link

pr-commenter bot commented Mar 28, 2025 •
edited
Loading

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/kafka-streams-test-fix
git_commit_date 1743421733 1743422520
git_commit_sha 8e29e20 9e6fe03
See matching parameters
Baseline Candidate
ci_job_date 1743424284 1743424284
ci_job_id 872205160 872205160
ci_pipeline_id 60498200 60498200
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 0 unstable metrics.

scenario Δ mean throughput
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce better
[+1320.631op/s; +4996.499op/s] or [+1.038%; +3.928%]
See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaProduceBenchmark.benchProduce unsure
[-41832.693op/s; -1959.328op/s] or [-2.297%; -0.108%]
scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce same

@pr-commenter
Copy link

pr-commenter bot commented Mar 28, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/kafka-streams-test-fix
git_commit_date 1744017548 1744020319
git_commit_sha d1abd07 e17d650
release_version 1.48.0-SNAPSHOT~d1abd07be5 1.48.0-SNAPSHOT~e17d65063b
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1744023286 1744023286
ci_job_id 883411615 883411615
ci_pipeline_id 61248186 61248186
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-gmgtqgjr-project-304-concurrent-0-wma2mfk1 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-gmgtqgjr-project-304-concurrent-0-wma2mfk1 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 69 metrics, 2 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054850
Total [baseline] (10.52 s) : 0, 10519652
Agent [candidate] (1.055 s) : 0, 1055173
Total [candidate] (10.475 s) : 0, 10474690
section appsec
Agent [baseline] (1.201 s) : 0, 1200685
Total [baseline] (10.801 s) : 0, 10801001
Agent [candidate] (1.2 s) : 0, 1199905
Total [candidate] (10.74 s) : 0, 10739919
section iast
Agent [baseline] (1.191 s) : 0, 1191020
Total [baseline] (11.066 s) : 0, 11065563
Agent [candidate] (1.183 s) : 0, 1183388
Total [candidate] (11.06 s) : 0, 11059739
section profiling
Agent [baseline] (1.292 s) : 0, 1291813
Total [baseline] (10.921 s) : 0, 10921383
Agent [candidate] (1.287 s) : 0, 1286945
Total [candidate] (10.906 s) : 0, 10905850
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.055 s -
Agent appsec 1.201 s 145.835 ms (13.8%)
Agent iast 1.191 s 136.17 ms (12.9%)
Agent profiling 1.292 s 236.963 ms (22.5%)
Total tracing 10.52 s -
Total appsec 10.801 s 281.349 ms (2.7%)
Total iast 11.066 s 545.91 ms (5.2%)
Total profiling 10.921 s 401.73 ms (3.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.055 s -
Agent appsec 1.2 s 144.732 ms (13.7%)
Agent iast 1.183 s 128.215 ms (12.2%)
Agent profiling 1.287 s 231.772 ms (22.0%)
Total tracing 10.475 s -
Total appsec 10.74 s 265.229 ms (2.5%)
Total iast 11.06 s 585.049 ms (5.6%)
Total profiling 10.906 s 431.159 ms (4.1%) 
gantt
    title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (720.181 ms) : 0, 720181
BytebuddyAgent [candidate] (720.443 ms) : 0, 720443
GlobalTracer [baseline] (239.264 ms) : 0, 239264
GlobalTracer [candidate] (239.565 ms) : 0, 239565
AppSec [baseline] (54.542 ms) : 0, 54542
AppSec [candidate] (54.135 ms) : 0, 54135
Debugger [baseline] (4.387 ms) : 0, 4387
Debugger [candidate] (4.388 ms) : 0, 4388
Remote Config [baseline] (688.819 µs) : 0, 689
Remote Config [candidate] (709.126 µs) : 0, 709
Telemetry [baseline] (14.904 ms) : 0, 14904
Telemetry [candidate] (15.042 ms) : 0, 15042
section appsec
BytebuddyAgent [baseline] (742.939 ms) : 0, 742939
BytebuddyAgent [candidate] (742.049 ms) : 0, 742049
GlobalTracer [baseline] (236.532 ms) : 0, 236532
GlobalTracer [candidate] (236.743 ms) : 0, 236743
AppSec [baseline] (176.286 ms) : 0, 176286
AppSec [candidate] (176.538 ms) : 0, 176538
Debugger [baseline] (4.274 ms) : 0, 4274
Debugger [candidate] (4.243 ms) : 0, 4243
Remote Config [baseline] (637.477 µs) : 0, 637
Remote Config [candidate] (642.518 µs) : 0, 643
Telemetry [baseline] (8.312 ms) : 0, 8312
Telemetry [candidate] (8.252 ms) : 0, 8252
IAST [baseline] (21.563 ms) : 0, 21563
IAST [candidate] (21.207 ms) : 0, 21207
section iast
BytebuddyAgent [baseline] (847.224 ms) : 0, 847224
BytebuddyAgent [candidate] (840.625 ms) : 0, 840625
GlobalTracer [baseline] (230.744 ms) : 0, 230744
GlobalTracer [candidate] (229.996 ms) : 0, 229996
AppSec [baseline] (55.934 ms) : 0, 55934
AppSec [candidate] (55.837 ms) : 0, 55837
Debugger [baseline] (4.112 ms) : 0, 4112
Debugger [candidate] (4.114 ms) : 0, 4114
Remote Config [baseline] (589.463 µs) : 0, 589
Remote Config [candidate] (589.783 µs) : 0, 590
Telemetry [baseline] (8.671 ms) : 0, 8671
Telemetry [candidate] (8.695 ms) : 0, 8695
IAST [baseline] (22.841 ms) : 0, 22841
IAST [candidate] (22.672 ms) : 0, 22672
section profiling
BytebuddyAgent [baseline] (711.561 ms) : 0, 711561
BytebuddyAgent [candidate] (709.671 ms) : 0, 709671
GlobalTracer [baseline] (361.017 ms) : 0, 361017
GlobalTracer [candidate] (358.558 ms) : 0, 358558
AppSec [baseline] (54.21 ms) : 0, 54210
AppSec [candidate] (53.312 ms) : 0, 53312
Debugger [baseline] (4.313 ms) : 0, 4313
Debugger [candidate] (4.272 ms) : 0, 4272
Remote Config [baseline] (701.536 µs) : 0, 702
Remote Config [candidate] (685.799 µs) : 0, 686
Telemetry [baseline] (9.007 ms) : 0, 9007
Telemetry [candidate] (8.925 ms) : 0, 8925
ProfilingAgent [baseline] (103.067 ms) : 0, 103067
ProfilingAgent [candidate] (103.644 ms) : 0, 103644
Profiling [baseline] (103.092 ms) : 0, 103092
Profiling [candidate] (103.67 ms) : 0, 103670
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.061 s) : 0, 1061314
Total [baseline] (8.714 s) : 0, 8714091
Agent [candidate] (1.054 s) : 0, 1053522
Total [candidate] (8.65 s) : 0, 8649773
section iast
Agent [baseline] (1.189 s) : 0, 1189122
Total [baseline] (9.23 s) : 0, 9230045
Agent [candidate] (1.191 s) : 0, 1190946
Total [candidate] (9.278 s) : 0, 9277642
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.194 s) : 0, 1194393
Total [baseline] (9.227 s) : 0, 9227499
Agent [candidate] (1.201 s) : 0, 1201345
Total [candidate] (9.246 s) : 0, 9246245
section iast_TELEMETRY_OFF
Agent [baseline] (1.179 s) : 0, 1179311
Total [baseline] (9.233 s) : 0, 9232894
Agent [candidate] (1.179 s) : 0, 1178755
Total [candidate] (9.229 s) : 0, 9228842
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.061 s -
Agent iast 1.189 s 127.808 ms (12.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.194 s 133.079 ms (12.5%)
Agent iast_TELEMETRY_OFF 1.179 s 117.997 ms (11.1%)
Total tracing 8.714 s -
Total iast 9.23 s 515.954 ms (5.9%)
Total iast_HARDCODED_SECRET_DISABLED 9.227 s 513.408 ms (5.9%)
Total iast_TELEMETRY_OFF 9.233 s 518.803 ms (6.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent iast 1.191 s 137.424 ms (13.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.201 s 147.823 ms (14.0%)
Agent iast_TELEMETRY_OFF 1.179 s 125.233 ms (11.9%)
Total tracing 8.65 s -
Total iast 9.278 s 627.869 ms (7.3%)
Total iast_HARDCODED_SECRET_DISABLED 9.246 s 596.472 ms (6.9%)
Total iast_TELEMETRY_OFF 9.229 s 579.069 ms (6.7%) 
gantt
    title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (724.512 ms) : 0, 724512
BytebuddyAgent [candidate] (719.243 ms) : 0, 719243
GlobalTracer [baseline] (240.729 ms) : 0, 240729
GlobalTracer [candidate] (238.802 ms) : 0, 238802
AppSec [baseline] (55.239 ms) : 0, 55239
AppSec [candidate] (54.574 ms) : 0, 54574
Debugger [baseline] (4.397 ms) : 0, 4397
Debugger [candidate] (4.416 ms) : 0, 4416
Remote Config [baseline] (705.106 µs) : 0, 705
Remote Config [candidate] (695.25 µs) : 0, 695
Telemetry [baseline] (14.785 ms) : 0, 14785
Telemetry [candidate] (14.991 ms) : 0, 14991
section iast
BytebuddyAgent [baseline] (846.005 ms) : 0, 846005
BytebuddyAgent [candidate] (847.061 ms) : 0, 847061
GlobalTracer [baseline] (230.463 ms) : 0, 230463
GlobalTracer [candidate] (230.441 ms) : 0, 230441
IAST [baseline] (22.583 ms) : 0, 22583
IAST [candidate] (22.66 ms) : 0, 22660
AppSec [baseline] (55.826 ms) : 0, 55826
AppSec [candidate] (56.437 ms) : 0, 56437
Debugger [baseline] (4.113 ms) : 0, 4113
Debugger [candidate] (4.105 ms) : 0, 4105
Remote Config [baseline] (599.196 µs) : 0, 599
Remote Config [candidate] (600.641 µs) : 0, 601
Telemetry [baseline] (8.676 ms) : 0, 8676
Telemetry [candidate] (8.746 ms) : 0, 8746
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (848.584 ms) : 0, 848584
BytebuddyAgent [candidate] (854.608 ms) : 0, 854608
GlobalTracer [baseline] (231.276 ms) : 0, 231276
GlobalTracer [candidate] (232.235 ms) : 0, 232235
IAST [baseline] (23.229 ms) : 0, 23229
IAST [candidate] (23.231 ms) : 0, 23231
AppSec [baseline] (56.759 ms) : 0, 56759
AppSec [candidate] (56.426 ms) : 0, 56426
Debugger [baseline] (4.159 ms) : 0, 4159
Debugger [candidate] (4.214 ms) : 0, 4214
Remote Config [baseline] (606.966 µs) : 0, 607
Remote Config [candidate] (615.771 µs) : 0, 616
Telemetry [baseline] (8.775 ms) : 0, 8775
Telemetry [candidate] (8.915 ms) : 0, 8915
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (837.825 ms) : 0, 837825
BytebuddyAgent [candidate] (837.512 ms) : 0, 837512
GlobalTracer [baseline] (229.613 ms) : 0, 229613
GlobalTracer [candidate] (229.329 ms) : 0, 229329
IAST [baseline] (22.06 ms) : 0, 22060
IAST [candidate] (22.237 ms) : 0, 22237
AppSec [baseline] (55.901 ms) : 0, 55901
AppSec [candidate] (55.871 ms) : 0, 55871
Debugger [baseline] (4.077 ms) : 0, 4077
Debugger [candidate] (4.077 ms) : 0, 4077
Remote Config [baseline] (583.413 µs) : 0, 583
Remote Config [candidate] (589.369 µs) : 0, 589
Telemetry [baseline] (8.513 ms) : 0, 8513
Telemetry [candidate] (8.441 ms) : 0, 8441
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-04-07T10:25:01 2025-04-07T10:32:47
git_branch master smola/kafka-streams-test-fix
git_commit_date 1744017548 1744020319
git_commit_sha d1abd07 e17d650
release_version 1.48.0-SNAPSHOT~d1abd07be5 1.48.0-SNAPSHOT~e17d65063b
start_time 2025-04-07T10:24:46 2025-04-07T10:32:32
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1744022368 1744022368
ci_job_id 883411616 883411616
ci_pipeline_id 61248186 61248186
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-gtma1pc4-project-304-concurrent-0-ey8ixzlw 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-gtma1pc4-project-304-concurrent-0-ey8ixzlw 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.343 ms) : 1324, 1362
.   : milestone, 1343,
appsec (1.739 ms) : 1716, 1763
.   : milestone, 1739,
appsec_no_iast (1.772 ms) : 1748, 1796
.   : milestone, 1772,
code_origins (1.695 ms) : 1668, 1723
.   : milestone, 1695,
iast (1.534 ms) : 1510, 1558
.   : milestone, 1534,
profiling (1.51 ms) : 1487, 1534
.   : milestone, 1510,
tracing (1.517 ms) : 1493, 1541
.   : milestone, 1517,
section candidate
no_agent (1.384 ms) : 1365, 1403
.   : milestone, 1384,
appsec (1.756 ms) : 1733, 1779
.   : milestone, 1756,
appsec_no_iast (1.739 ms) : 1714, 1765
.   : milestone, 1739,
code_origins (1.671 ms) : 1643, 1699
.   : milestone, 1671,
iast (1.536 ms) : 1512, 1560
.   : milestone, 1536,
profiling (1.55 ms) : 1526, 1575
.   : milestone, 1550,
tracing (1.521 ms) : 1497, 1544
.   : milestone, 1521,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.343 ms [1.324 ms, 1.362 ms] -
appsec 1.739 ms [1.716 ms, 1.763 ms] 396.044 µs (29.5%)
appsec_no_iast 1.772 ms [1.748 ms, 1.796 ms] 429.145 µs (32.0%)
code_origins 1.695 ms [1.668 ms, 1.723 ms] 352.3 µs (26.2%)
iast 1.534 ms [1.51 ms, 1.558 ms] 191.328 µs (14.2%)
profiling 1.51 ms [1.487 ms, 1.534 ms] 167.147 µs (12.4%)
tracing 1.517 ms [1.493 ms, 1.541 ms] 173.794 µs (12.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.384 ms [1.365 ms, 1.403 ms] -
appsec 1.756 ms [1.733 ms, 1.779 ms] 372.153 µs (26.9%)
appsec_no_iast 1.739 ms [1.714 ms, 1.765 ms] 355.514 µs (25.7%)
code_origins 1.671 ms [1.643 ms, 1.699 ms] 287.242 µs (20.8%)
iast 1.536 ms [1.512 ms, 1.56 ms] 152.054 µs (11.0%)
profiling 1.55 ms [1.526 ms, 1.575 ms] 166.325 µs (12.0%)
tracing 1.521 ms [1.497 ms, 1.544 ms] 136.712 µs (9.9%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5
    dateFormat X
    axisFormat %s
section baseline
no_agent (383.958 µs) : 364, 404
.   : milestone, 384,
iast (515.764 µs) : 494, 538
.   : milestone, 516,
iast_FULL (741.014 µs) : 719, 763
.   : milestone, 741,
iast_GLOBAL (562.145 µs) : 540, 584
.   : milestone, 562,
iast_HARDCODED_SECRET_DISABLED (517.077 µs) : 495, 539
.   : milestone, 517,
iast_INACTIVE (464.692 µs) : 444, 486
.   : milestone, 465,
iast_TELEMETRY_OFF (501.378 µs) : 480, 523
.   : milestone, 501,
tracing (466.765 µs) : 445, 488
.   : milestone, 467,
section candidate
no_agent (385.849 µs) : 366, 406
.   : milestone, 386,
iast (526.691 µs) : 505, 549
.   : milestone, 527,
iast_FULL (732.966 µs) : 711, 755
.   : milestone, 733,
iast_GLOBAL (563.378 µs) : 542, 585
.   : milestone, 563,
iast_HARDCODED_SECRET_DISABLED (522.78 µs) : 501, 544
.   : milestone, 523,
iast_INACTIVE (471.357 µs) : 449, 493
.   : milestone, 471,
iast_TELEMETRY_OFF (503.129 µs) : 481, 525
.   : milestone, 503,
tracing (471.524 µs) : 450, 493
.   : milestone, 472,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 383.958 µs [363.691 µs, 404.225 µs] -
iast 515.764 µs [493.851 µs, 537.676 µs] 131.805 µs (34.3%)
iast_FULL 741.014 µs [719.135 µs, 762.893 µs] 357.056 µs (93.0%)
iast_GLOBAL 562.145 µs [540.057 µs, 584.233 µs] 178.187 µs (46.4%)
iast_HARDCODED_SECRET_DISABLED 517.077 µs [495.112 µs, 539.042 µs] 133.119 µs (34.7%)
iast_INACTIVE 464.692 µs [443.55 µs, 485.834 µs] 80.734 µs (21.0%)
iast_TELEMETRY_OFF 501.378 µs [479.742 µs, 523.014 µs] 117.42 µs (30.6%)
tracing 466.765 µs [445.147 µs, 488.383 µs] 82.807 µs (21.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 385.849 µs [365.687 µs, 406.01 µs] -
iast 526.691 µs [504.644 µs, 548.737 µs] 140.842 µs (36.5%)
iast_FULL 732.966 µs [710.843 µs, 755.089 µs] 347.117 µs (90.0%)
iast_GLOBAL 563.378 µs [541.607 µs, 585.15 µs] 177.53 µs (46.0%)
iast_HARDCODED_SECRET_DISABLED 522.78 µs [501.06 µs, 544.5 µs] 136.931 µs (35.5%)
iast_INACTIVE 471.357 µs [449.376 µs, 493.339 µs] 85.509 µs (22.2%)
iast_TELEMETRY_OFF 503.129 µs [481.403 µs, 524.854 µs] 117.28 µs (30.4%)
tracing 471.524 µs [450.436 µs, 492.612 µs] 85.675 µs (22.2%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/kafka-streams-test-fix
git_commit_date 1744017548 1744020319
git_commit_sha d1abd07 e17d650
release_version 1.48.0-SNAPSHOT~d1abd07be5 1.48.0-SNAPSHOT~e17d65063b
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1744022838 1744022838
ci_job_id 883411617 883411617
ci_pipeline_id 61248186 61248186
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-gmgtqgjr-project-304-concurrent-1-3p221vrj 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-gmgtqgjr-project-304-concurrent-1-3p221vrj 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.961 s) : 14961000, 14961000
.   : milestone, 14961000,
appsec (15.0 s) : 15000000, 15000000
.   : milestone, 15000000,
iast (18.936 s) : 18936000, 18936000
.   : milestone, 18936000,
iast_GLOBAL (18.195 s) : 18195000, 18195000
.   : milestone, 18195000,
profiling (15.14 s) : 15140000, 15140000
.   : milestone, 15140000,
tracing (14.924 s) : 14924000, 14924000
.   : milestone, 14924000,
section candidate
no_agent (14.834 s) : 14834000, 14834000
.   : milestone, 14834000,
appsec (15.121 s) : 15121000, 15121000
.   : milestone, 15121000,
iast (19.488 s) : 19488000, 19488000
.   : milestone, 19488000,
iast_GLOBAL (17.9 s) : 17900000, 17900000
.   : milestone, 17900000,
profiling (15.074 s) : 15074000, 15074000
.   : milestone, 15074000,
tracing (15.04 s) : 15040000, 15040000
.   : milestone, 15040000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.961 s [14.961 s, 14.961 s] -
appsec 15.0 s [15.0 s, 15.0 s] 39.0 ms (0.3%)
iast 18.936 s [18.936 s, 18.936 s] 3.975 s (26.6%)
iast_GLOBAL 18.195 s [18.195 s, 18.195 s] 3.234 s (21.6%)
profiling 15.14 s [15.14 s, 15.14 s] 179.0 ms (1.2%)
tracing 14.924 s [14.924 s, 14.924 s] -37.0 ms (-0.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.834 s [14.834 s, 14.834 s] -
appsec 15.121 s [15.121 s, 15.121 s] 287.0 ms (1.9%)
iast 19.488 s [19.488 s, 19.488 s] 4.654 s (31.4%)
iast_GLOBAL 17.9 s [17.9 s, 17.9 s] 3.066 s (20.7%)
profiling 15.074 s [15.074 s, 15.074 s] 240.0 ms (1.6%)
tracing 15.04 s [15.04 s, 15.04 s] 206.0 ms (1.4%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~e17d65063b, baseline=1.48.0-SNAPSHOT~d1abd07be5
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.493 ms) : 1481, 1504
.   : milestone, 1493,
appsec (2.399 ms) : 2352, 2446
.   : milestone, 2399,
iast (2.17 ms) : 2111, 2229
.   : milestone, 2170,
iast_GLOBAL (2.211 ms) : 2151, 2270
.   : milestone, 2211,
profiling (2.478 ms) : 2297, 2660
.   : milestone, 2478,
tracing (2.006 ms) : 1960, 2052
.   : milestone, 2006,
section candidate
no_agent (1.481 ms) : 1470, 1493
.   : milestone, 1481,
appsec (2.393 ms) : 2346, 2439
.   : milestone, 2393,
iast (2.166 ms) : 2107, 2225
.   : milestone, 2166,
iast_GLOBAL (2.204 ms) : 2144, 2263
.   : milestone, 2204,
profiling (2.011 ms) : 1964, 2059
.   : milestone, 2011,
tracing (2.002 ms) : 1956, 2047
.   : milestone, 2002,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.493 ms [1.481 ms, 1.504 ms] -
appsec 2.399 ms [2.352 ms, 2.446 ms] 906.549 µs (60.7%)
iast 2.17 ms [2.111 ms, 2.229 ms] 677.156 µs (45.4%)
iast_GLOBAL 2.211 ms [2.151 ms, 2.27 ms] 718.001 µs (48.1%)
profiling 2.478 ms [2.297 ms, 2.66 ms] 985.687 µs (66.0%)
tracing 2.006 ms [1.96 ms, 2.052 ms] 513.057 µs (34.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.481 ms [1.47 ms, 1.493 ms] -
appsec 2.393 ms [2.346 ms, 2.439 ms] 911.311 µs (61.5%)
iast 2.166 ms [2.107 ms, 2.225 ms] 684.972 µs (46.2%)
iast_GLOBAL 2.204 ms [2.144 ms, 2.263 ms] 722.365 µs (48.8%)
profiling 2.011 ms [1.964 ms, 2.059 ms] 530.165 µs (35.8%)
tracing 2.002 ms [1.956 ms, 2.047 ms] 520.521 µs (35.1%)

@pr-commenter
Copy link

pr-commenter bot commented Mar 28, 2025 •
edited
Loading

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/kafka-streams-test-fix
git_commit_date 1744017548 1744020319
git_commit_sha d1abd07 e17d650
See matching parameters
Baseline Candidate
ci_job_date 1744021955 1744021955
ci_job_id 883411620 883411620
ci_pipeline_id 61248186 61248186
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
jdkVersion 11.0.25 11.0.25
jmhVersion 1.36 1.36
jvm /usr/lib/jvm/java-11-openjdk-amd64/bin/java /usr/lib/jvm/java-11-openjdk-amd64/bin/java
jvmArgs -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant
vmName OpenJDK 64-Bit Server VM OpenJDK 64-Bit Server VM
vmVersion 11.0.25+9-post-Ubuntu-1ubuntu122.04 11.0.25+9-post-Ubuntu-1ubuntu122.04

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 2 metrics, 0 unstable metrics.

scenario Δ mean throughput
scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume worse
[-9114.475op/s; -3223.201op/s] or [-4.880%; -1.726%]
See unchanged results
scenario Δ mean throughput
scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume same
scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume unsure
[-14400.569op/s; -938.098op/s] or [-4.562%; -0.297%]

@smola smola marked this pull request as ready for review April 1, 2025 08:12
@smola smola requested a review from a team as a code owner April 1, 2025 08:12
amarziali
amarziali reviewed Apr 1, 2025
View reviewed changes
...va-agent/instrumentation/kafka-streams-0.11/src/latestDepTest/groovy/KafkaStreamsTest.groovy Outdated
@@ -27,6 +27,7 @@ import spock.lang.Shared

import java.util.concurrent.LinkedBlockingQueue
import java.util.concurrent.TimeUnit
import java.util.regex.Pattern

@Flaky("https://github.com/DataDog/dd-trace-java/issues/3865")
Copy link
Collaborator

@amarziali amarziali Apr 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test does not seem to run because it's marked as flaky. Should this be removed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amarziali To be honest, I don't know if my fixes fix the flakiness or not. Just the permanent errors. Do you think we should remove @Flaky here and try? (we can add it back if needed)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK Got it you ran the test locally and fixed it. I think we can have a try otherwise we will never spot that kind of issues in CI

@smola smola force-pushed the smola/kafka-streams-test-fix branch from 9e6fe03 to e17d650 Compare April 7, 2025 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amarziali amarziali amarziali left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
comp: testing Testing tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@smola @amarziali