Pull upstream changes (#276)

* Upstream sync 7fa9a93..25cf4fa

* [2.0] Use distributed tracing for compatibility across different versions of the tracer (#2020)

Introduce communication between different versions of the tracer inside of the same process. This should prevent having disconnected traces.

- `DistributedTracer` is the entry point for the communication between tracers. `Instance` gets the singleton `IDistributedTracer` for this version of the tracer, and `GetDistributedTracer` gets rewritten by the profiler to enable the communication (the manual tracer calls the automatic tracer)
- `IAutomaticTracer` is the interface used by the manual tracer to communicate with the automatic. Notably, it has a `Register` method that is called by the manual trace to enable duplex communication
- `ICommonTracer` is the interface used for duplex communication
- Only the `IDistributedTracer` interface should be used by the tracer/instrumentations. `IAutomaticTracer` and `ICommonTracer` should only be by the `AutomaticTracer` and `ManualTracer` to communicate between themselves
- On the other hand, no method of `IDistributedTracer` should ever be called from inside `AutomaticTracer`/`ManualTracer`. This is enforced by using explicit interface implementation for `IDistributedTracer` (so those methods can't be called accidentally, as the instance needs to be casted first)
- The goal of segregating `IDistributedTracer` and `ICommonTracer`/`IAutomaticTracer` is to distinguish the calls that come from the same version of the tracer, from the calls that come from another version of the tracer

After version 2.0 has shipped, the interfaces `ICommonTracer` and `IAutomaticTracer` can't ever change. If we need new methods, they will be added to new `ICommonTracer2`/`IAutomaticTracer2` interfaces.

* [CIApp] - MsTest 2.2.8 integration. (#2107)

* changes.

* Changes

* Changes.

* Fix integration for MsTest 2.2.8

* Update MSTest package version.

* Revert dependabot changes.

* fix tests

* Changes from review.

* Update nuget package versions definitions.

* Address comment from review.

* Fix flaky test in unit tests (#2106)

Due to the way we calculate service name, if service name tests run at the same time as AzureAppServicesMetadataTests, then AzureAppServices.IsRelevant returns true, and we may unexpectedly pull the service name from the AAS env vars. This change handles that scenario.

* Change the public API to only use ISpan / IScope (#2090)

Goal:
Modify the public API to only work with interfaces `IScope` and `ISpan` instead of the implementation types `Scope` and `Span`. This is a required step for our current approach to solve the mismatching version issue.

Summary of changes:
- Introduced new public interfaces: `IScope`, `ISpan`, `ITracer`, `IHasTags`
- In public API's, replace implementation types with the new interfaces
- Renamed the internal `StartActiveWithTags` methods to `StartActiveInternal`
- Refactored internal usages of `Tracer.ActiveScope` (now returns `IScope`) to `Tracer.InternalActiveScope`
- In testing code, add lots of typecasts from `ISpan` to `Span` so we can access internal members like Duration, TraceContext, etc.

Internal API changes:
- `Span`: The Span object is always initialized with a `SpanContext` object, so the property `internal SpanContext Context` can always be accessed by our code to get internal properties like TraceContext and Origin.
- `Tracer`: The Tracer object always creates Scope objects from this assembly, so the property `internal Scope InternalActiveScope` can always be accessed by our code to get internal properties like Parent and Root.

Additional Changes:
- Add a public interface `ITracer` that the `Tracer` implements. This will allow for users to register the Tracer via DI, though we may need to make some more changes to the `ITracer` API before 2.0
- Add a public API test to `Datadog.Trace.OpenTracing.Tests`
- Add an internal interface called `IDatadogOpenTracingTracer` which has the absolute minimum API that we will maintain for communication from Datadog.Trace.OpenTracing => Datadog.Trace
- Add `"ReSharperTestRunner"` to the list of known TestRunners

* Update dependabot/Nuke targets  (#2117)

* Rename UpdateIntegrationsJson -> UpdateIntegrationDefinitions

We don't have an integrations.json any more

* Don't update the dependabot file when updating integration definitions

Ideally, we should update this file whenever we update the versions we're _testing_ OR when we add a new integration. In that respect, updating the dependabot file when we run `UpdateIntegrationDefinitions` makes sense, but the downside is that this may suggest that we are _testing_ more than we really are.

> This also resulting in the dependabot file being changed in version-bump commits despite the fact we were neither changing integrations or testing more. This was essentially incorrect.

Instead, updating the dependabot as part of the `GeneratePackageVersions` target which updates what we test makes more sense to me.

* Update Dependabot honeypot to list maximum version we _do_ support, not the maximum we _should_ support

* Update supported package versions to match currently supported versions

* Delete dependabot update

* Fix benchmarks comparison (#2130)

Due to a copy-pasta problem, we weren't actually comparing allocations properly

* Use the non truncated UserAgent property in .netfx webapi (#2128)

* Fix Typo

* Use the non truncated UserAgent property in .netfx webapi

- Change the code to pass the useragent when needed
- Impact webapi integration tests

* Update tracer/src/Datadog.Trace/SpanContextPropagator.cs

Removing useless log line

Co-authored-by: Andrew Lock <[email protected]>

* Update tracer/src/Datadog.Trace/SpanContextPropagator.cs

Co-authored-by: Andrew Lock <[email protected]>

* Implement one comment

* Fix Public Api Tests and add a UTest

Co-authored-by: Andrew Lock <[email protected]>

* Add space before exception message (#2133)

* [Test Package Versions Bump] (#2072)

Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Update ITracer interface (#2131)

* Update ITracer interface

Reduces the number of overloads, and leaves the interface open for extension if required

* Rename CreateSpanSettings -> SpanCreationSettings

* Update Public API

* Centralize Mock Trace Agent Initialization (#2114)

* Bump Integrationsjson (#2118)

* Improve performance of AutomaticTracer (#2135)

* Improve performance of AutomaticTracer

- Remove an asynclocal read in GetSpanContext
- Remove the asynclocal write in IDistributedTracer.SetSpanContext when there is no manual tracer

* [AppSec] Fix tag prefix bug (#2123)

Fix tag prefix bug that meant request headers where incorrectly labeled response headers.

* Reinstate Integration.AdoNet integration (#2137)

* Reinstate IntegrationId.AdoNet

Users can use this to disable _all_ ADO.NET integrations

* minor: record IntegrationName in instrumentation attribute

Only informational, but currently required for telemetry etc

* Make Span internal (#2134)

* Make Span internal

Now we expose ISpan instead, we don't need Span to be public

* Delete TraceContext.InvalidOperationException project and smoke test

This project is broken after the InvalidOperationException changes, and has never actually been run in CI, so doesn't seem to provide any value

* Make scope internal

* Log additional information about the AssemblyLoadContext (#2136)

* DBScopeFactory improvements. (#2138)

* Change DBScopeFactory implementation to reduce allocations and use SqlTags directly.

* Changes from review.

* Changes based on the review

* Refactor cache and adds TagsCacheItem cache layer.

* Add a big fat warning in NativeCallTargetDefinition file (#2126)

* Add a big fat warning in NativeCallTargetDefinition file

* add message from review

* Fix asp snapshots file names (#2148)

* Fix asp snapshots file names

* And Owin tests as well

* Mock the active span when using multiple versions of the tracer (#2095)

* Share the active scope across tracers

Co-authored-by: Zach Montoya <[email protected]>

* Use PathBase.ToUriComponent (#2141)

* Implementation of Exploration Tests (#2089)

* Implementation of Exploration Tests

* add memory dump + clean

* remove redundant 'git'

* Update tracer/build/_build/Build.ExplorationTests.cs

Co-authored-by: Andrew Lock <[email protected]>

* Update tracer/build/_build/Build.ExplorationTests.cs

Co-authored-by: Andrew Lock <[email protected]>

* Update Build.ExplorationTests.cs

* Update DotNetSettingsExtensions.cs

* Replace targets with method functions

* Replace 'ExplorationTestName' with enum + rename parameters

* Refactor environment variables

* Refactor BuildVariables with extensions + fix ilspy git repo

* readable if

* Rename Profiler -> ContinuousProfiler

* Rename monitoringType -> explorationTestUseCase

* Renamings

* fix display name

* build tracer home during exploration test + fix ilspy exploration test

* fix property

* rename env variables + add MoveLogsToBuildData method

* rename PrepareExplorationTestEnvironment ->  SetUpExplorationTest

* add skip BuildTracerHome

* fix CI

* add x86 .net sdk

* more time to wait

* clone quietly

* -c advice.detachedHead=false

* no-warnings

* customize build process for different cases

* try..finally

* move helper classes

* add "DD_SERVICE" and "DD_VERSION"

Co-authored-by: Andrew Lock <[email protected]>

* remove "DD_INTEGRATIONS"

Co-authored-by: Andrew Lock <[email protected]>

* disable ilspy

* remove publish test results

* Revert "remove publish test results"

This reverts commit cb2d0530bf02d495124abe0c217852d940247f40.

* remover publish test results

Co-authored-by: Andrew Lock <[email protected]>

* Disable version conflict fix (#2155)

* Disable version conflict fix
* Remove GetDistributedTracer method

* [Version Bump] 2.0.0-prerelease (#2159)

Co-authored-by: pierotibou <[email protected]>

* Reset the AsyncLocal DistributedValue when starting an automatic instrumentation continuation (#2156)

This behavior is similar to resetting the AsyncLocal Active scope. Without this fix, when an async automatic instrumentation method is run, it will update the AsyncLocal DistributedValue to the newly created span and subsequent spans will have the incorrect parent.

* Revert "Disable version conflict fix (#2155)" (#2162)

This reverts commit 60075b6.

* Re-implement NLog logs injection (#2096)

Similar to #1710, this change modifies NLog's logs-injection code to stop using the LibLog based subscriber (which is not safe to use across AppDomain boundaries for some versions of NLog) and to use the MDLC/MDC directly using automatic instrumentation.

This change has a big limitation: **NLog logs injection now requires automatic instrumentation**.

Additionally, the CustomNLogLogProvider type has been renamed to NoOpNLogProvider and modified so that the LibLog interface does not affect the Log4Net state.

* Re-implement Serilog logs injection (#2152)

Similar to #1710 and #2096, this change modifies Serilog's logs-injection code to stop using the LibLog based subscriber to add properties directly to the `LogEvent` using automatic instrumentation.

This change has a big limitation: **Serilog logs injection now requires automatic instrumentation**.

Additionally, the CustomSerilogLogProvider type has been removed

* Remove LockSamplingPriority (#2150)

* Remove LockSamplingPriority
* Add distributed GetSamplingPriority method

* Fix parser null ref and upgrade rule-set (#2167)

* Isolate transport settings (#2166)

* Synchronize runtime id across versions of the tracer (#2172)

* Use DistributedTracer to synchronize the runtime id

Also, remove the ability to enable/disable runtime metrics from code

* Fix warnings in Nuke (#2174)

* Properly disable parallelization in AzureAppServicesMetadataTests (#2173)

* [Redo] Fix logs injection to work correctly when there's a version mismatch (#2161)

- Enables logs injection to work with the new DistributedTracer when version conflicts occur by using the `IReadOnlyDictionary<string, string> IDistributedTracer.GetSpanContext()` API to get the active TraceId / SpanId instead of the local active scope. We only need to fetch the TraceId and SpanId, so we access the dictionary with the appropriate keys.
- Lowers the minimum assembly version for NLog from 2.1.0 to 1.0.0.505, which passes both the DistributedTrace case and the cross-AppDomain scenarios

* Minor improvements (#2177)

* Include missing properties on Exporter Settings (#2179)

* [2.0] Simplify Tracer.StartActive overloads to match ITracer.StartActive overloads (#2176)

### Breaking changes
- Add StartActive(string) and StartActive(string, SpanCreationSettings) to Tracer public API
- When creating a span, set the ServiceName to the DefaultServiceName instead of the parent's ServiceName
  - This is a breaking change and is being considered for 2.x feature development so now is the appropriate time to take the change, if approved.
  - If a user still prefers the new span to have the parent span's ServiceName, this can be fixed locally by updating the `ISpan.ServiceName` property or this can be fixed globally by updating the `TracerSettings.ServiceName` and running `Tracer.Configure(settings)`

### Other changes
- Separate ITracer implementation and Tracer implementation of public methods (we're doing this for ISpan and IScope so I followed convention)
- Modify `Tracer.StartActive(string operationName, ISpanContext parent = null, string serviceName = null, DateTimeOffset? startTime = null, bool ignoreActiveScope = false, bool finishOnClose = true)` API to be internal
- Fix samples that were using the service name parameter previously exposed by the `StartActive` API

* Changes types visibility (#2185)

* Changes types visibility.

* Fix test build issue.

* Rename profiler projects (#2187)

* Rename profiler projects

* Remove obsolete comments

* [appsec] throughput tests (#2160)

We systematically drop traces when running perf tests, as the backend processing of them is quite heavy.

* [AppSec] Handle null keys when reading headers, cookies and query strings (#2171)

When url  is localhost?test&[$slice] IIS gives query string as keys null and values test and [$slice]. In the opposite, net core gives them as keys.

* Update rules file 1.2.4 (#2190)

* Modify tests to use Tracer.StartActive(string, SpanCreationSettings) (#2191)

* Fix TraceId type

* Fix errors after pull

* Remove HttpHeaderNames and fix the code using it

* Revert SpanContext changes

* Move Propagator property from IDatadogTracer to IDatadogOpenTracingTracer

* Fix unit tests build

* Fix DbScopeFactory

* Fix integration tests build

* Fix api snapshot

* Fix api snapshot

* Fix tests

* Fix server timing header setting

* Fix KeyNames in SpanContext

* Fix managed unit tests

* Make TracesTransport readonly

* Fix api snapshot

* Replace env variables with SIGNALFX ones

* Replace env variables with SIGNALFX ones

* remove DD_ vars from code

* remove dd changelog

* update pull from upstream documentation

* fix env variable

* fix env variables

* fix adonet integration tests

* restire modified by Splunk message

* DD_APPSEC_KEEP_TRACES -> SIGNALFX_APPSEC_KEEP_TRACES

* add missiing Modified by Splunk Inc. header

* move exporter settings to exporter file

* add missing Modified by Splunk Inc.

* User-Agent is under CommonHeaderNames

* fix assembly name

* cleanup logs

* DD_INTERNAL_TRACE_VERSION_COMPATIBILITY -> SIGNALFX_INTERNAL_TRACE_VERSION_COMPATIBILITY

* fix file format

* document new env. variables

* nuke UpdateVersion

* update public api verified file

* update publicKey for SignalFx.Tracing

Executed, and manually formated.
using System.Reflection;

var signalFxTracingAssembly = Assembly.LoadFrom(@"C:\GitHub\signalfx-dotnet-tracing\tracer\bin\tracer-home\netstandard2.0\SignalFx.Tracing.dll");
var sfxAssemblyName = signalFxTracingAssembly.GetName();

var publicKeyBytes = sfxAssemblyName.GetPublicKey();

var publicKeyBytesAsString = string.Join(", ", publicKeyBytes);

Console.WriteLine(publicKeyBytesAsString);

* verified files for Datadog.Trace.ClrProfiler.IntegrationTests.OwinWebApi2TestsCallTarget

* verification files for Datadog.Trace.ClrProfiler.IntegrationTests.AspNet
NoFF and WithFF files changed places. Creation process was fixed in the same PR

* fix typos

* adjust elasticsearch operation name after rebase

* adjust to preferred properties format

* fix log message

* skip VersionConfict Tests for now

* fix AspNetCore5.TestSecurity.
Current results can be accepted as we do not support enabled security version

* remove duplicated assertions

Co-authored-by: Kevin Gosse <[email protected]>
Co-authored-by: Tony Redondo <[email protected]>
Co-authored-by: Andrew Lock <[email protected]>
Co-authored-by: Pierre Bonet <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Colin Higgins <[email protected]>
Co-authored-by: Robert Pickering <[email protected]>
Co-authored-by: Zach Montoya <[email protected]>
Co-authored-by: Alexander S <[email protected]>
Co-authored-by: pierotibou <[email protected]>
Co-authored-by: Anna <[email protected]>
Co-authored-by: Piotr Kiełkowicz <[email protected]>
Co-authored-by: Mateusz Lach <[email protected]>

Author: 14 people

.azure-pipelines/ultimate-pipeline.yml

@@ -28,7 +28,7 @@ trigger:
 #     - Managed utility APIs (may be used transitively):
       - shared/src/managed-src/Datadog.Collections/*
       - shared/src/managed-src/Datadog.Util/*
-#     - Managed Logging APIs (may be used transitively):      
+#     - Managed Logging APIs (may be used transitively):
       - shared/src/managed-src/Datadog.Logging.Emission/*
       - shared/src/managed-src/Datadog.Logging.Composition/*
       - shared/src/managed-src/Datadog.Logging/*
@@ -509,7 +509,7 @@ stages:
         Start-CosmosDbEmulator -Timeout 300
       displayName: 'Start CosmosDB Emulator'
       workingDirectory: $(Pipeline.Workspace)
-      
+
     - powershell: |
         Write-Host "Initializing LocalDB"
         sqllocaldb.exe start
@@ -781,7 +781,7 @@ stages:
             build: true
             baseImage: $(baseImage)
             command: "CleanObjFiles"
-      
+
         - template: steps/restore-working-directory.yml
           parameters:
             artifact: build-linux-arm64-working-directory
@@ -841,6 +841,64 @@ stages:
             baseImage: $(baseImage)
             command: "CheckBuildLogsForErrors"
 
+- stage: exploration_tests_windows
+  condition: and(succeeded(), eq(variables['isScheduledBuild'], 'False'), eq(variables['runExplorationTests'], 'True'))
+  dependsOn: build_windows
+
+  jobs:
+  - template: steps/update-github-status-jobs.yml
+    parameters:
+      jobs: [ExplorationTest]
+
+  - job: generator
+    pool:
+      vmImage: ubuntu-18.04
+    steps:
+      - bash: |
+          matrix="{"
+          for explorationTestUseCase in "debugger" "continuousProfiler"; do
+            for explorationTestName in "eshoponweb" "protobuf" "cake" "swashbuckle" "paket" ; do
+              name="${explorationTestUseCase//./_}_${explorationTestName}"
+              matrix="${matrix} '${name}':"
+              matrix="${matrix} { 'explorationTestUseCase': '${explorationTestUseCase}', "
+              matrix="${matrix} 'explorationTestName': '${explorationTestName}' },"
+            done
+          done
+          matrix="${matrix::-1} }"
+          echo "##vso[task.setVariable variable=values;isOutput=true]${matrix}"
+        name: mtrx
+
+  - job: ExplorationTest
+    timeoutInMinutes: 100
+    dependsOn: generator
+    strategy:
+      matrix: $[ dependencies.generator.outputs['mtrx.values'] ]
+    pool:
+      vmImage: windows-2019
+
+    # Enable the Datadog Agent service for this job
+    services:
+      dd_agent: dd_agent
+
+    steps:
+    - template: steps/install-dotnet-sdks.yml
+      parameters:
+        includeX86: true
+    - template: steps/restore-working-directory.yml
+
+    - script: tracer\build.cmd RunExplorationTests_$(explorationTestUseCase) -ExplorationTestName $(explorationTestName)
+      displayName: RunExplorationTest
+
+    - publish: tracer/build_data
+      displayName: Uploading exploration_tests_windows tracer logs
+      artifact: exploration_tests_windows_tracer_logs_$(explorationTestUseCase)_$(explorationTestName)
+      condition: succeededOrFailed()
+      continueOnError: true
+
+    - script: tracer\build.cmd CheckBuildLogsForErrors
+      displayName: Check logs for errors
+
+
 - stage: benchmarks
   condition: and(succeeded(), ne(variables['isNgenTestBuild'], 'True'))
   dependsOn: build_windows
@@ -1283,6 +1341,37 @@ stages:
         SIGNALFX_SERVICE_NAME: dd-trace-dotnet
         SIGNALFX_ENV: CI
 
+- stage: throughput_appsec
+  condition: and(succeeded(), ne(variables['isNgenTestBuild'], 'True'))
+  dependsOn: [build_linux]
+  pool: Throughput-AppSec
+  jobs:
+  - template: steps/update-github-status-jobs.yml
+    parameters:
+      jobs: [Linux64]
+  #### Throughput-AppSec Linux 64
+
+  - job: Linux64
+    timeoutInMinutes: 60
+
+    steps:
+    - task: DownloadPipelineArtifact@2
+      displayName: Download linux native binary
+      inputs:
+        artifact: linux-tracer-home-debian
+        path: $(System.DefaultWorkingDirectory)/tracer/tracer-home-linux
+
+    - script: |
+        test ! -s "tracer/tracer-home-linux/Datadog.Trace.ClrProfiler.Native.so" && echo "tracer/tracer-home-linux/Datadog.Trace.ClrProfiler.Native.so does not exist" && exit 1
+        test ! -s "tracer/tracer-home-linux/libddwaf.so" && echo "tracer/tracer-home-linux/libddwaf.so does not exist" && exit 1
+        cd $(System.DefaultWorkingDirectory)/tracer/build/crank
+        chmod +x ./run-appsec.sh
+        ./run-appsec.sh "linux"
+      displayName: Crank
+      env:
+        DD_SERVICE: dd-trace-dotnet-appsec
+
+
 - stage: coverage
   condition: and(succeeded(), eq(variables['isScheduledBuild'], 'False'), ne(variables['isNgenTestBuild'], 'True'))
   dependsOn: [integration_tests_windows, integration_tests_windows_iis, integration_tests_linux, unit_tests_linux, unit_tests_macos, unit_tests_windows]

.github/workflows/auto_bump_test_package_versions.yml

@@ -26,7 +26,7 @@ jobs:
           dotnet-version: '6.0.x'
 
       - name: "Regenerating package versions"
-        run: .\tracer\build.ps1 UpdateIntegrationsJson GeneratePackageVersions
+        run: .\tracer\build.ps1 UpdateIntegrationDefinitions GeneratePackageVersions
 
       - name: Create Pull Request
         id: pr

.github/workflows/create_version_bump_pr.yml

@@ -34,7 +34,7 @@ jobs:
         run: .\tracer\build.ps1 UpdateChangeLog
 
       - name: "Bump Version"
-        run: .\tracer\build.ps1 UpdateIntegrationsJson UpdateVersion UpdateMsiContents
+        run: .\tracer\build.ps1 UpdateIntegrationDefinitions UpdateVersion UpdateMsiContents
 
       - name: "Verify Changes"
         id: changes

.gitignore

@@ -334,3 +334,5 @@ tracer/src/Datadog.Trace.ClrProfiler.Native/Makefile
 tracer/src/Datadog.Trace.ClrProfiler.Native/CMakeCache.txt
 tracer/src/Datadog.Trace.ClrProfiler.Native/cmake_install.cmake
 !tracer/src/Datadog.Trace.ClrProfiler.Native/lib/**
+
+.ionide/