Ensure Propagators are still run when `DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore` #8604

mhlidd · 2025-03-21T14:40:13Z

What Does This Do

Ensures that extraction still occurs for all incoming headers, and if DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore, then start a new tracecontext when it is being built from remote context in CoreTracer.

Motivation

This PR is made to fix the following failures from system-tests CI where other headers are not being seen as part of the span.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: APMAPI-1251

pr-commenter · 2025-03-21T19:19:53Z

Benchmarks

Startup

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-03-31T13:48:59	2025-03-31T13:56:45
git_branch	master	mhlidd/fix_client_ip_tag
git_commit_date	1743425619	1743427774
git_commit_sha	`32ca4c6`	`a4fa02b`
release_version	1.48.0-SNAPSHOT~32ca4c67e9	1.48.0-SNAPSHOT~a4fa02b05b
start_time	2025-03-31T13:48:45	2025-03-31T13:56:31

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1743429805	1743429805
ci_job_id	872433442	872433442
ci_pipeline_id	60509348	60509348
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-aopymtch-project-304-concurrent-3-d9igel1h 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-aopymtch-project-304-concurrent-3-d9igel1h 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~a4fa02b05b, baseline=1.48.0-SNAPSHOT~32ca4c67e9
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.369 ms) : 1349, 1389
.   : milestone, 1369,
appsec (1.727 ms) : 1704, 1751
.   : milestone, 1727,
appsec_no_iast (1.732 ms) : 1708, 1755
.   : milestone, 1732,
code_origins (1.67 ms) : 1642, 1698
.   : milestone, 1670,
iast (1.516 ms) : 1490, 1542
.   : milestone, 1516,
profiling (1.572 ms) : 1548, 1597
.   : milestone, 1572,
tracing (1.509 ms) : 1484, 1534
.   : milestone, 1509,
section candidate
no_agent (1.371 ms) : 1352, 1390
.   : milestone, 1371,
appsec (1.748 ms) : 1724, 1771
.   : milestone, 1748,
appsec_no_iast (1.744 ms) : 1720, 1767
.   : milestone, 1744,
code_origins (1.69 ms) : 1662, 1717
.   : milestone, 1690,
iast (1.503 ms) : 1478, 1527
.   : milestone, 1503,
profiling (1.532 ms) : 1507, 1556
.   : milestone, 1532,
tracing (1.522 ms) : 1497, 1546
.   : milestone, 1522,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.369 ms [1.349 ms, 1.389 ms]	-
appsec	1.727 ms [1.704 ms, 1.751 ms]	358.266 µs (26.2%)
appsec_no_iast	1.732 ms [1.708 ms, 1.755 ms]	362.554 µs (26.5%)
code_origins	1.67 ms [1.642 ms, 1.698 ms]	300.942 µs (22.0%)
iast	1.516 ms [1.49 ms, 1.542 ms]	146.838 µs (10.7%)
profiling	1.572 ms [1.548 ms, 1.597 ms]	203.259 µs (14.8%)
tracing	1.509 ms [1.484 ms, 1.534 ms]	139.927 µs (10.2%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.371 ms [1.352 ms, 1.39 ms]	-
appsec	1.748 ms [1.724 ms, 1.771 ms]	376.347 µs (27.4%)
appsec_no_iast	1.744 ms [1.72 ms, 1.767 ms]	372.154 µs (27.1%)
code_origins	1.69 ms [1.662 ms, 1.717 ms]	318.403 µs (23.2%)
iast	1.503 ms [1.478 ms, 1.527 ms]	131.159 µs (9.6%)
profiling	1.532 ms [1.507 ms, 1.556 ms]	160.147 µs (11.7%)
tracing	1.522 ms [1.497 ms, 1.546 ms]	150.522 µs (11.0%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~a4fa02b05b, baseline=1.48.0-SNAPSHOT~32ca4c67e9
    dateFormat X
    axisFormat %s
section baseline
no_agent (391.305 µs) : 371, 411
.   : milestone, 391,
iast (514.077 µs) : 492, 536
.   : milestone, 514,
iast_FULL (734.8 µs) : 713, 757
.   : milestone, 735,
iast_GLOBAL (566.328 µs) : 544, 589
.   : milestone, 566,
iast_HARDCODED_SECRET_DISABLED (523.417 µs) : 501, 546
.   : milestone, 523,
iast_INACTIVE (458.866 µs) : 438, 480
.   : milestone, 459,
iast_TELEMETRY_OFF (501.913 µs) : 480, 523
.   : milestone, 502,
tracing (461.144 µs) : 440, 482
.   : milestone, 461,
section candidate
no_agent (379.738 µs) : 360, 400
.   : milestone, 380,
iast (516.444 µs) : 494, 538
.   : milestone, 516,
iast_FULL (732.837 µs) : 711, 755
.   : milestone, 733,
iast_GLOBAL (572.564 µs) : 549, 596
.   : milestone, 573,
iast_HARDCODED_SECRET_DISABLED (519.496 µs) : 498, 541
.   : milestone, 519,
iast_INACTIVE (471.121 µs) : 449, 493
.   : milestone, 471,
iast_TELEMETRY_OFF (502.673 µs) : 481, 524
.   : milestone, 503,
tracing (476.238 µs) : 455, 497
.   : milestone, 476,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	391.305 µs [371.401 µs, 411.209 µs]	-
iast	514.077 µs [492.026 µs, 536.127 µs]	122.772 µs (31.4%)
iast_FULL	734.8 µs [712.77 µs, 756.83 µs]	343.495 µs (87.8%)
iast_GLOBAL	566.328 µs [543.87 µs, 588.787 µs]	175.023 µs (44.7%)
iast_HARDCODED_SECRET_DISABLED	523.417 µs [501.325 µs, 545.508 µs]	132.112 µs (33.8%)
iast_INACTIVE	458.866 µs [438.107 µs, 479.625 µs]	67.561 µs (17.3%)
iast_TELEMETRY_OFF	501.913 µs [480.466 µs, 523.361 µs]	110.608 µs (28.3%)
tracing	461.144 µs [440.186 µs, 482.103 µs]	69.839 µs (17.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	379.738 µs [359.691 µs, 399.784 µs]	-
iast	516.444 µs [494.468 µs, 538.419 µs]	136.706 µs (36.0%)
iast_FULL	732.837 µs [710.841 µs, 754.833 µs]	353.099 µs (93.0%)
iast_GLOBAL	572.564 µs [549.359 µs, 595.769 µs]	192.826 µs (50.8%)
iast_HARDCODED_SECRET_DISABLED	519.496 µs [497.658 µs, 541.335 µs]	139.758 µs (36.8%)
iast_INACTIVE	471.121 µs [449.419 µs, 492.823 µs]	91.383 µs (24.1%)
iast_TELEMETRY_OFF	502.673 µs [480.953 µs, 524.393 µs]	122.935 µs (32.4%)
tracing	476.238 µs [455.158 µs, 497.319 µs]	96.501 µs (25.4%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	mhlidd/fix_client_ip_tag
git_commit_date	1743425619	1743427774
git_commit_sha	`32ca4c6`	`a4fa02b`
release_version	1.48.0-SNAPSHOT~32ca4c67e9	1.48.0-SNAPSHOT~a4fa02b05b

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1743430242	1743430242
ci_job_id	872433443	872433443
ci_pipeline_id	60509348	60509348
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zvu8n-98-project-304-concurrent-5-vvl6lu21 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zvu8n-98-project-304-concurrent-5-vvl6lu21 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~a4fa02b05b, baseline=1.48.0-SNAPSHOT~32ca4c67e9
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.473 ms) : 1461, 1485
.   : milestone, 1473,
appsec (2.331 ms) : 2288, 2375
.   : milestone, 2331,
iast (2.112 ms) : 2056, 2167
.   : milestone, 2112,
iast_GLOBAL (2.151 ms) : 2096, 2207
.   : milestone, 2151,
profiling (1.966 ms) : 1922, 2010
.   : milestone, 1966,
tracing (1.951 ms) : 1909, 1993
.   : milestone, 1951,
section candidate
no_agent (1.47 ms) : 1458, 1481
.   : milestone, 1470,
appsec (2.331 ms) : 2288, 2375
.   : milestone, 2331,
iast (2.115 ms) : 2059, 2170
.   : milestone, 2115,
iast_GLOBAL (2.163 ms) : 2107, 2219
.   : milestone, 2163,
profiling (1.962 ms) : 1919, 2006
.   : milestone, 1962,
tracing (1.929 ms) : 1887, 1971
.   : milestone, 1929,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.461 ms, 1.485 ms]	-
appsec	2.331 ms [2.288 ms, 2.375 ms]	858.434 µs (58.3%)
iast	2.112 ms [2.056 ms, 2.167 ms]	638.835 µs (43.4%)
iast_GLOBAL	2.151 ms [2.096 ms, 2.207 ms]	678.388 µs (46.1%)
profiling	1.966 ms [1.922 ms, 2.01 ms]	492.953 µs (33.5%)
tracing	1.951 ms [1.909 ms, 1.993 ms]	478.001 µs (32.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.458 ms, 1.481 ms]	-
appsec	2.331 ms [2.288 ms, 2.375 ms]	861.885 µs (58.6%)
iast	2.115 ms [2.059 ms, 2.17 ms]	645.306 µs (43.9%)
iast_GLOBAL	2.163 ms [2.107 ms, 2.219 ms]	693.308 µs (47.2%)
profiling	1.962 ms [1.919 ms, 2.006 ms]	492.607 µs (33.5%)
tracing	1.929 ms [1.887 ms, 1.971 ms]	459.523 µs (31.3%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~a4fa02b05b, baseline=1.48.0-SNAPSHOT~32ca4c67e9
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.675 s) : 14675000, 14675000
.   : milestone, 14675000,
appsec (15.048 s) : 15048000, 15048000
.   : milestone, 15048000,
iast (18.147 s) : 18147000, 18147000
.   : milestone, 18147000,
iast_GLOBAL (17.996 s) : 17996000, 17996000
.   : milestone, 17996000,
profiling (14.963 s) : 14963000, 14963000
.   : milestone, 14963000,
tracing (15.007 s) : 15007000, 15007000
.   : milestone, 15007000,
section candidate
no_agent (15.342 s) : 15342000, 15342000
.   : milestone, 15342000,
appsec (15.243 s) : 15243000, 15243000
.   : milestone, 15243000,
iast (18.596 s) : 18596000, 18596000
.   : milestone, 18596000,
iast_GLOBAL (18.124 s) : 18124000, 18124000
.   : milestone, 18124000,
profiling (15.158 s) : 15158000, 15158000
.   : milestone, 15158000,
tracing (14.878 s) : 14878000, 14878000
.   : milestone, 14878000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.675 s [14.675 s, 14.675 s]	-
appsec	15.048 s [15.048 s, 15.048 s]	373.0 ms (2.5%)
iast	18.147 s [18.147 s, 18.147 s]	3.472 s (23.7%)
iast_GLOBAL	17.996 s [17.996 s, 17.996 s]	3.321 s (22.6%)
profiling	14.963 s [14.963 s, 14.963 s]	288.0 ms (2.0%)
tracing	15.007 s [15.007 s, 15.007 s]	332.0 ms (2.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.342 s [15.342 s, 15.342 s]	-
appsec	15.243 s [15.243 s, 15.243 s]	-99.0 ms (-0.6%)
iast	18.596 s [18.596 s, 18.596 s]	3.254 s (21.2%)
iast_GLOBAL	18.124 s [18.124 s, 18.124 s]	2.782 s (18.1%)
profiling	15.158 s [15.158 s, 15.158 s]	-184.0 ms (-1.2%)
tracing	14.878 s [14.878 s, 14.878 s]	-464.0 ms (-3.0%)

amarziali · 2025-03-28T08:23:01Z

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java

@@ -1511,23 +1511,27 @@ private DDSpanContext buildSpanContext() {
      if (parentContext != null
          && parentContext.isRemote()
          && Config.get().getTracePropagationBehaviorExtract()


can we fetch this value once since it's called several time and it's used in several places? It will save few calls

my understanding from @dougqh is that this chain of calls gets constant folded, so storing the value locally doesn't actually save much - that said it can make the code more readable

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java

amarziali

Thanks for having addressed the comments

mhlidd added 3 commits March 21, 2025 10:23

saving changes

de431dd

cleanup and adding tests

6566856

Merge branch 'master' into mhlidd/fix_client_ip_tag

4f67bd4

mhlidd marked this pull request as ready for review March 21, 2025 18:29

mhlidd requested a review from a team as a code owner March 21, 2025 18:29

mhlidd requested review from smola, mcculls and amarziali March 21, 2025 18:29

mhlidd added type: bug tag: no release notes Changes to exclude from release notes comp: context propagation Trace context propagation and removed type: bug labels Mar 24, 2025

amarziali reviewed Mar 28, 2025

View reviewed changes

dd-trace-core/src/main/java/datadog/trace/core/CoreTracer.java Outdated Show resolved Hide resolved

mhlidd added 2 commits March 28, 2025 10:22

addressing PR comments

3f8ed71

Merge branch 'master' into mhlidd/fix_client_ip_tag

665e632

mhlidd requested a review from amarziali March 28, 2025 18:20

mcculls approved these changes Mar 31, 2025

View reviewed changes

amarziali approved these changes Mar 31, 2025

View reviewed changes

optimizing calls to config

a4fa02b

smola approved these changes Mar 31, 2025

View reviewed changes

mhlidd merged commit 848f35a into master Mar 31, 2025
253 checks passed

mhlidd deleted the mhlidd/fix_client_ip_tag branch March 31, 2025 18:05

github-actions bot added this to the 1.48.0 milestone Mar 31, 2025

mhlidd mentioned this pull request Mar 31, 2025

[Java] Re-enabling tests from #4355 DataDog/system-tests#4440

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure Propagators are still run when `DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore` #8604

Ensure Propagators are still run when `DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore` #8604

mhlidd commented Mar 21, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Mar 21, 2025 •

edited

Loading

amarziali Mar 28, 2025

mcculls Mar 31, 2025

amarziali left a comment

Ensure Propagators are still run when DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore #8604

Ensure Propagators are still run when DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore #8604

Conversation

mhlidd commented Mar 21, 2025 • edited by jira bot Loading

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented Mar 21, 2025 • edited Loading

Benchmarks

Startup

Load

Parameters

Summary

Dacapo

Parameters

Summary

amarziali Mar 28, 2025

Choose a reason for hiding this comment

mcculls Mar 31, 2025

Choose a reason for hiding this comment

amarziali left a comment

Choose a reason for hiding this comment

Ensure Propagators are still run when `DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore` #8604

Ensure Propagators are still run when `DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=ignore` #8604

mhlidd commented Mar 21, 2025 •

edited by jira bot

Loading

pr-commenter bot commented Mar 21, 2025 •

edited

Loading