Pull upstream changes (#415)

* Upstream sync 6989e88..afb9638

* Changes DuckType to bypass ToString method to the target instance (#2430)

* Changes DuckType defaults to bypass ToString method to the target instance.

* Changes

* Change the implementation to be part of the IDuckType interface adding an special case to avoid ToString throwing on null target instances.

* Revert ci_library.language tag to language (as other integrations) (#2433)

* [CIApp] - Ensure the Origin tag value on the public SpanContext .ctor (#2434)

* Ensure the Origin tag value on the public SpanContext .ctor when running in CI Visibility mode.

* Changes from review.

* Setup Github Actions for the profiler CI (#2415)

Plug the profiler CI and add its jobs in the noop-pipeline when required

* Github Actions jobs are recognized by their name (or key if the name is not present).
When there is a matrix (example: x64,x86 / net45,net5.0), there will be as many jobs as number of combination of the matrix.
The final name will be "name (XX, YY)" (ex: "build (x64, net45)").
* Make run.sh executable

* update PR template (#2429)

* AOT improvements (#2428)

* Update `.vsconfig` (#2407)

Update .vsconfig

Co-authored-by: Lucas Pimentel-Ordyna <[email protected]>

* exclude `.vsconfig` from builds
* exclude `.vsconfig` from profiler pipeline

* Reduce test flakiness caused by TcpPortProvider.GetOpenPort (#2438)

* Add Auth header to HttpClient that downloads Azure DevOps build artifacts (#2436)

add auth header to httpclient

* Fix MongoDB integration when BsonDocument throws an exception (#2435)

* Add try/catch to fix MogoDB regression when BsonDocument throws and exception.

* Add snapshot tests for MongoDb (#2440)

* Add a VerifyHelper for verifying full snapshots

Primarily ensures a consistent ordering of traces, and of spans within each trace (can be an issue when you have any concurrent behaviour)

* Make scrubbing easier for VerifyHelper

* Update MongoDb tests to use snapshot testing instead

* Update snapshots to remove `ismaster` and `hello` spans

(Based on changes in #2216)

* Update snapshots to match current results

Based on changes in #2430 and #2435

* Remove buildinfo and getlasterrror spans from snapshots

These spans are generated as part of server monitoring, but we will have a non-deterministic number of them, which will lead to flaky snapshots.

As a workaround, remove them from the snapshots, and do some basic assertions against them instead.

Also, adds some extra scrubbing for differences between localhost snapshot generation and cli (e.g. host, aggregate queries etc)

* Add arm64 scrubber for hostname

* Add additional snapshot and scrubbers to support 2.5.x + 2.6.x

Co-authored-by: Andrew Lock <[email protected]>

* Move EnumNgenModuleMethodsInliningThisMethod call from the background thread and use the CLR thread instead. (#2442)

* Add regression tests for running in partial trust environments (#2338)

We previously had a regression test called `Sandbox.ManualTracing` to make sure that a simple usage of custom instrumentation did not cause issues due to code permissions. This PR adds the correct test category to make sure it runs in CI.

This PR also adds a `Sandbox.AutomaticInstrumentation` to run automatic instrumentation in a similar partial trust scenario. However, this application is expected to fail so we expect a failure in its test runs.

* Update protobuf version to 3.19.4 (#2443)

The dependabot tool in the dd-trace-dotnet repository reported an
issue with version <3.15 of protobuf
https://github.com/DataDog/dd-trace-dotnet/security/dependabot/2

* Disable sending telemetry by default (#2431)

* Disable telemetry by default in all cases

* Adding more info on instrumenting specific methods (#2441)

* Adding more info on instrumenting specific methods

Penultimate parameter must be exception on end method
Instrumenting child classes
Instrumenting properties

* Update AutomaticInstrumentation.md

changing phrasing

* response to comments

* Response to comment on ref and in keywords
Ref or in are the same, in is safer for not modifying arguments
Remove spaces

* Trigger deployment to the reliability environment (#2362)

* trigger gitlab pipeline from ours actually

* fix

* remove useless variables

* Change variable name to DEPLOY_TO_REL_ENV

to make it more explicit

* Add IIS checks to the CLI (#2416)

* Revert "Reduce test flakiness caused by TcpPortProvider.GetOpenPort (#2438)" (#2449)

This reverts commit bee319f.

* Update the http.status_code on root-level aspnet.request spans after TransferRequest calls (#2419)

When HttpServerUtility.TransferRequest is called, this results in two separate IIS pipeline requests where the HttpContext.Response object for the initial (unused) request has a 200 status code and the HttpContext.Response object for the second (transferred) request has the correct status. The issue was that the root span would be marked with a http.status_code=200, which does not corrrespond to the actual status code. This overrides the http.status_code tag of the root span with the correct status code.

* [Doc] Describe the best scenarios to upgrade from v1 to v2 (#2417)

* first version of the migration doc

* Trying to avoid triggering some OCD by switching to .NET everywhere

* Apply suggestions from code review

Co-authored-by: Lucas Pimentel-Ordyna <[email protected]>

* Apply suggestions from code review

Co-authored-by: Zach Montoya <[email protected]>

* Apply suggestions from code review

Co-authored-by: Andrew Lock <[email protected]>

* Apply suggestions from code review

Co-authored-by: Zach Montoya <[email protected]>

* Update v1_to_v2.md

Co-authored-by: Lucas Pimentel-Ordyna <[email protected]>
Co-authored-by: Zach Montoya <[email protected]>
Co-authored-by: Andrew Lock <[email protected]>

* Refactorings to avoid merge conflicts with the Live Debugger (#2452)

* Refactored the CallTargetTokens to maximize code sharing with the Debugger and squashed various refactorings including renamings to avoid merge conflicts with the Debugger slydog

* [Test Package Versions Bump] Updating package versions (#2444)

* [Test Package Versions Bump]

* CouchbaseNetClient 3.2.6 dropped support for netcoreapp2.1

That causes failures in .NET Core 2.1 and .NET Core 3.0

Co-authored-by: andrewlock <[email protected]>
Co-authored-by: Andrew Lock <[email protected]>

* Add a warning to the CLI when a version conflict is detected (#2448)

Co-authored-by: Andrew Lock <[email protected]>
Co-authored-by: Zach Montoya <[email protected]>

* Log a message when the process is killed (#2456)

* Update GraphQL and Security ASP.NET Core tests to reduce flakiness (#2451)

* Update GraphQLTests to reduce flakiness

Previously, we were trying to find an open port, and then use that for all requests.

With this change, we rely on ASP.NET Core's own "find an open port" mechanism, and then read that port from the output. This _should_ be more reliable, and hopefully will reduce the number of cases of "unable to bind port" that we see

* Update Security ASP.NET Core tests to reduce flakiness

Previously, we were trying to find an open port, and then use that for all requests.

With this change, we rely on ASP.NET Core's own "find an open port" mechanism, and then read that port from the output. This _should_ be more reliable, and hopefully will reduce the number of cases of "unable to bind port" that we see

* Update tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/GraphQLTests.cs

Co-authored-by: Kevin Gosse <[email protected]>

Co-authored-by: Kevin Gosse <[email protected]>

* add support for trace-level tags (vertical propagation) (#2432)

* add TraceTags and tests

* add Tracer.Tags

* add trace-level tags to root spans during serialization

* fix xml-doc comment

* make one ctor private

* fix off-by-one error

* add comments

* fix xml-doc comment, redux

* keep only tags with "_dd.p.*" prefix

* don't allocation the string if not needed

* comments

* no need for volatile read

* improved validation, caching, and tests

* implement IEnumerable, add GetEnumerator() and Count

* don't use AsList(), rename overloads

* remove AsList()

* remove IEnumerable to prevent accidental allocations, keep GetEnumerator

* add missing lock

* expand comments

* check for prefix earlier

* more comments

* fix build errors in tests

* New tags useful for throughput testing (#2439)

* Replace [Fact] and [Theory] with [SkippableFact] and [SkippableTheory] (#2453)

We have known issues where .NET Core 2.1 specifically can throw a seg fault. We handle this, and rethrow as a `SkipException`, but these are only handled if we use `[SkippableFact]` and `[SkippableTheory]`.

For simplicity, and to avoid missing any, updated all the integration test projects to _always_ use the skippable attributes.

* Fix incorrect timeout in BatchingSinkTests (#2462)

* Fix incorrect timeout in BatchingSinkTests

This should be 10s not 3hrs

* Add some more logs to track down the flake

* Fix missing `http.status_code` tag on ASP.NET Core spans with errors (#2458)

* Add tests for case where an exception is set on a span and a non-200 status code is set

Shows that the tests are failing for the "legacy" (feature-flag disabled) use case.

* Fix missing HTTP status code tag when span is manually marked as an error

* Enable telemetry in AAS by default (#2454)

The intake is going to be enabled to support the feature, so it's ok to be enabled

* Refactor no-op pipeline to not use so many hosts (#2467)

Currently the no-op pipeline is generating a different job for each github status update check it needs to run. As we now have the profiler in there as well, the no-op pipeline is running on basically every PR. That means 20ish agents are required to run these checks. While they only run for a very short period of time (~2s), there's still the provisioning time etc to think about.

With this approach, instead of using a matrix to generate multiple jobs, we output a single semi-colon delimited list of jobs to update in GitHub, and do it all in one hit.

This also added some refactoring of the GenerateNoopStages code. Functionally it's basically the same, but we output a list of jobs instead of a json blob

* Fix typos in code coverage report generator (#2468)

* Add support for aspnetcore sites hosted out of process (#2450)

Co-authored-by: Zach Montoya <[email protected]>

* Increase coverage in mongodb integration tests (#2469)

To avoid erroneous coverage descriptions in PR

* Refactor all EnumNgenModuleMethodsInliningThisMethod call sites (#2470)

* Refactor all EnumNgenModuleMethodsInliningThisMethod call sites and make sure it's only called from two ICorProfilerCallBacks: ModuleLoadFinished and JITCachedFunctionSearchStarted

* fix

* Disable runtime metrics tests for netcoreapp 3.1 (#2457)

* Disable runtime metrics tests for netcoreapp 3.1

Co-authored-by: Pierre Bonet <[email protected]>

* Make IntegrationRegistry and DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS case insensitive (#2447)

* Make IntegrationRegistry case insensitive

* Make DD_LOGS_DIRECT_SUBMISSION_INTEGRATIONS case insensitive

Fixes AIT-767

* Support cleaning filenames in http-client + aspnet (#2471)

The HttpClient integration (and the ASP.NET/ASP.NET Core integration when the feature flag is disabled) all do "cleaning" of URLs to try and reduce the cardinality introduced by number/guid route parameters. This works by examining each "segment" in a URL, and using heuristics to see if we should replace it with `?`.

This works as expected for the following paths:
- `/some/value/123` => `/some/value/?`
- `/some/value/123/123-456 => `/some/value/?/?`
- `/some/value/123/example.png => `/some/value/?/example.png`

However, it treats the whole final segment as a single unit, which means filenames with extensions will never be obfuscated, e.g.

- `/some-filevalue/123.png`
- `/some-filevalue/123.456`
- `/some-filevalue/123.c`

That can lead to high cardinality, which in turn can cause problems for customers with too many endpoints in their account.

To work around it, we treat the filename as a separate segment from the file extension, and don't try and obfuscate the extension

* Always set response header tags in ASP.NET Core (#2480)

* Add proper spdlog-based logging to NativeLoader (#2481)

* Revert "Add proper spdlog-based logging to NativeLoader (#2481)" (#2485)

This reverts commit 08d734e.
Just temporarily, as we are about to release and want more testing of this first

* [Version Bump] 2.5.0 (#2488)

Co-authored-by: andrewlock <[email protected]>

* Improve error handling in CLI (#2463)

* Use mermaid for flow chart (#2476)

* Use mermaid for flow chart

* Combine states

* Add proper spdlog-based logging to NativeLoader (#2481) (#2489)

Co-authored-by: Omer Raviv <[email protected]>

* Increase Redis test matrix (#2472)

* Try reducing Redis flakiness

Based on some recommendations here: StackExchange/StackExchange.Redis#1120 (comment)

* Make sure we test against the latest StackExchange versions

* Ignore common test failures (#2477)

* We get occasional SegFaults in NativeProfilerChecks, but we're not doing anything about them at the moment, so stop failing the build

* Ignore AbandonedMutexException thrown by Coverlet

* Ignore AbandonedMutexException thrown by Coverlet in smoke tests

* Ensure the native loader works with 3rd party dispatcher (e.g.: Contrast) (#2478)

Get the Native Loader module file path from the HMODULE instead of COR[E]_PROFILER_XXX env variables

* Remove TODO from stylecop.json (#2490)

* Fix typo in docker-compose.yml that is causing CI failure in resolving IncludeMinorPackageVersions (#2475)

* Add `ExpandRouteTemplatesEnabled` setting for ASP.NET Core (#2496)

* Add ExpandRouteParametersEnabled setting

Implement for ASP.NET Core and add tests

In order to keep the consistent behaviour for routes like `/`, `/Home`, `/Home/Index`, we can't simply short-circuit the route parameter replacement. Instead, we have to replace each individual segment, which is unfortunate, but I don't think it can be avoided

* Update implementation to use {routeTemplate} placeholders instead of '?' (a bit nicer)

* Annotate test data for clarity

* Doc updates

Co-authored-by: Lucas Pimentel-Ordyna <[email protected]>

* Rename ExpandRouteParameters -> ExpanRouteTemplates

Co-authored-by: Lucas Pimentel-Ordyna <[email protected]>

* Shared Loader - Add support for more rewriting points (#2487)

* Add support for more rewriting points

* Fixes

* Fix variable name to something meaningful (#2504)

* Add clarification to automatic instrumentation docs (#2473)

* Adapt verify files and tests to remove appsec enable blocking tests (#2505)

* redundant docs removal, verification files update, redundant scripts/workflow removal

* doc new configuration key

* skip failing runner tests

* SignalFx distribution requiers more permisions
than upstream

* disable appdomain test
separate task created

* fix lib name and public token

* adjust log paths

Co-authored-by: Tony Redondo <[email protected]>
Co-authored-by: Gregory LEOCADIE <[email protected]>
Co-authored-by: Lucas Pimentel-Ordyna <[email protected]>
Co-authored-by: Robert Pickering <[email protected]>
Co-authored-by: Omer Raviv <[email protected]>
Co-authored-by: Alexander S <[email protected]>
Co-authored-by: Andrew Lock <[email protected]>
Co-authored-by: Zach Montoya <[email protected]>
Co-authored-by: Anna <[email protected]>
Co-authored-by: Pierre Bonet <[email protected]>
Co-authored-by: Kevin Gosse <[email protected]>
Co-authored-by: Matan Green <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: andrewlock <[email protected]>
Co-authored-by: Omer Raviv <[email protected]>
Co-authored-by: Piotr Kiełkowicz <[email protected]>

Author: 15 people

.vsconfig

@@ -1,8 +1,6 @@
 {
   "version": "1.0",
   "components": [
-    "Microsoft.VisualStudio.Component.ClassDesigner",
-    "Microsoft.VisualStudio.Component.GraphDocument",
     "Microsoft.VisualStudio.Component.CoreEditor",
     "Microsoft.VisualStudio.Workload.CoreEditor",
     "Microsoft.Net.Component.4.8.SDK",
@@ -39,48 +37,25 @@
     "Microsoft.VisualStudio.Component.FSharp.WebTemplates",
     "Microsoft.VisualStudio.Component.DockerTools",
     "Microsoft.NetCore.Component.Web",
-    "Microsoft.VisualStudio.Component.WebDeploy",
-    "Microsoft.VisualStudio.Component.AppInsights.Tools",
     "Microsoft.VisualStudio.Component.Web",
     "Microsoft.Net.Component.4.8.TargetingPack",
     "Microsoft.Net.ComponentGroup.4.8.DeveloperTools",
-    "Microsoft.VisualStudio.Component.AspNet45",
     "Microsoft.VisualStudio.Component.AspNet",
-    "Component.Microsoft.VisualStudio.Web.AzureFunctions",
-    "Microsoft.VisualStudio.ComponentGroup.AzureFunctions",
-    "Microsoft.VisualStudio.ComponentGroup.Web.CloudTools",
-    "Microsoft.VisualStudio.Component.DiagnosticTools",
-    "Microsoft.VisualStudio.Component.EntityFramework",
     "Microsoft.VisualStudio.Component.Debugger.JustInTime",
-    "Component.Microsoft.VisualStudio.LiveShare",
-    "Microsoft.VisualStudio.Component.WslDebugging",
-    "Microsoft.VisualStudio.Component.IntelliCode",
-    "Microsoft.Net.Component.4.6.2.TargetingPack",
-    "Microsoft.Net.Component.4.7.TargetingPack",
     "Microsoft.VisualStudio.Workload.NetWeb",
     "Microsoft.VisualStudio.Component.VC.CoreIde",
     "Microsoft.VisualStudio.Component.VC.Tools.x86.x64",
     "Microsoft.VisualStudio.Component.Windows10SDK.19041",
     "Microsoft.VisualStudio.Component.ManagedDesktop.Prerequisites",
     "Microsoft.ComponentGroup.Blend",
-    "Microsoft.VisualStudio.Component.FSharp.Desktop",
-    "Microsoft.VisualStudio.ComponentGroup.MSIX.Packaging",
     "Microsoft.VisualStudio.Workload.ManagedDesktop",
     "Microsoft.VisualStudio.Component.VC.Redist.14.Latest",
     "Microsoft.VisualStudio.ComponentGroup.NativeDesktop.Core",
-    "Microsoft.VisualStudio.Component.VC.Tools.ARM64",
-    "Microsoft.VisualStudio.Component.VC.CLI.Support",
-    "Microsoft.VisualStudio.Component.Windows10SDK.18362",
     "Microsoft.VisualStudio.Workload.NativeDesktop",
-    "Microsoft.VisualStudio.Component.VC.Tools.ARM",
-    "Microsoft.Net.ComponentGroup.TargetingPacks.Common",
-    "Microsoft.Net.Component.4.6.TargetingPack",
     "Microsoft.VisualStudio.Component.Git",
-    "Microsoft.VisualStudio.Component.LinqToSql",
     "Microsoft.NetCore.Component.Runtime.3.1",
     "Microsoft.NetCore.Component.Runtime.5.0",
-    "Microsoft.Net.Component.4.6.1.TargetingPack",
-    "Microsoft.Net.Component.4.6.2.SDK",
-    "Microsoft.Net.Component.4.7.SDK"
+    "Microsoft.VisualStudio.Component.VC.14.29.16.11.x86.x64",
+    "Microsoft.Net.Component.4.6.1.TargetingPack"
   ]
 }

Datadog.Trace.sln

@@ -468,6 +468,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Datadog.Trace.Tools.Runner.
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Samples.Telemetry", "tracer\test\test-applications\integrations\Samples.Telemetry\Samples.Telemetry.csproj", "{2D1FF937-3237-4A1B-9C6C-82FA5E22CAD7}"
 EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Sandbox.AutomaticInstrumentation", "tracer\test\test-applications\regression\Sandbox.AutomaticInstrumentation\Sandbox.AutomaticInstrumentation.csproj", "{10619BA2-AED1-482A-8570-BB7C7B83DDDC}"
+EndProject
 Global
 	GlobalSection(SharedMSBuildProjectFiles) = preSolution
 		tracer\test\test-applications\Samples.Shared\Samples.Shared.projitems*{24d3e547-8897-4111-baa2-b92f50cc13d4}*SharedItemsImports = 5
@@ -2438,6 +2440,7 @@ Global
 		{887AC8BA-35A6-4646-BF9A-59357155805E} = {BAF8F246-3645-42AD-B1D0-0F7EAFBAB34A}
 		{595BB494-CD26-4A51-8E5B-009219E3F2AE} = {8CEC2042-F11C-49F5-A674-2355793B600A}
 		{2D1FF937-3237-4A1B-9C6C-82FA5E22CAD7} = {BAF8F246-3645-42AD-B1D0-0F7EAFBAB34A}
+		{10619BA2-AED1-482A-8570-BB7C7B83DDDC} = {498A300E-D036-49B7-A43D-821D1CAF11A5}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {160A1D00-1F5B-40F8-A155-621B4459D78F}

docker-compose.yml

@@ -568,7 +568,7 @@ services:
       - framework=${framework:-netcoreapp3.1}
       - baseImage=${baseImage:-debian}
       - TestAllPackageVersions=${TestAllPackageVersions:-true}
-      - IncludeMinorPackageVersions=${IncludeMinorPackageVersions:-false}0
+      - IncludeMinorPackageVersions=${IncludeMinorPackageVersions:-false}
       - Verify_DisableClipboard=true
       - DiffEngine_Disabled=true
       - SIGNALFX_CLR_ENABLE_NGEN=${SIGNALFX_CLR_ENABLE_NGEN:-1}