Activity.SetTag not processed as expected

[nefarius]

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

In my ASP.NET Core (.NET 8) endpoint handler I used

Activity.Current?.SetTag("myApp.userId", request.UserId);

which worked finde before moving to Sentry. I added OTEL integration as per documentation. The tags do not show up in the Sentry Dashboard.

I change it to

SentrySdk.ConfigureScope(scope =>
{
    scope.SetTag("myApp.userId", request.UserId);
});

and it works as expected.

Expected Result

Ideally Activity.Current?.SetTag would be routed to Sentry as expected or adjust the documentation on why this isn't possible/working.

Actual Result

If I open a sample trace in Trace View or search for the tag value, I can not find it if using Activity.SetTag. I repeat the test with SentrySdk.ConfigureScope... and it works.

Product Area

Traces

Link

No response

DSN

No response

Version

No response

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[jamescrosswell]

@nefarius OpenTelemetry doesn't (as far as I'm aware) have any equivalent of :

SentrySdk.ConfigureScope(scope =>
{
    scope.SetTag("myApp.userId", request.UserId);
});

Scope is concept specific to Sentry. It's basically a place where you can store ambient contextual data that will get attached to all/any events that get sent to Sentry (including transactions/spans for traces).

In the .NET implementation of OpenTelemetry, an Activity represents an individual span on a trace. So this:

Activity.Current?.SetTag("myApp.userId", request.UserId);

... would set a "tag" on the specific span represented by the current activity. I put "tag" in quotes, since the concept of a tag also doesn't really exist in OpenTelemetry. This is the terminology Microsoft used in their implementation of OpenTelemetry (which they built before OTel was even a thing). Activity.Tags get mapped to OpenTelemetry Attributes... and we map OpenTelemetry Attributes to SentrySpan.Data.

In any case, the "tag" won't be applied to any other spans and whether Activity.Current gets captured and sent to Sentry in a trace or not depends on what sources you have configured to be captured in your OpenTelemetry configuration (and whether Activity.Current is an activity from one of those sources).

So you might not be seeing the Tag as you're expecting for various reasons:

1. It could be that you're expecting it to come through as a Sentry.Tag... but OpenTelemetry attributes show up as Extra data (not as tags) in Sentry
2. It could be that you've applied it to Activity.Current and you're expecting it to be applied to all Activities in the current "scope" (but OTel doesn't really have any concept of a scope)
3. It could be that you applied it to Activity.Current but Activity.Current is a span that wasn't created by one of the sources you're listening to in your OTel configuration

You can see a demonstration of setting the tag on a specific span, in our samples:

sentry-dotnet/samples/Sentry.Samples.OpenTelemetry.Console/Program.cs

Lines 37 to 44 in cdbb4bc

	// This creates a span called "Task 1" within the transaction
	using (var task = activitySource.StartActivity("Task 1"))
	{
	task?.SetTag("Answer", 42);
	Thread.Sleep(100); // simulate some work
	Console.WriteLine("Task 1 completed");
	task?.SetStatus(Status.Ok);
	}

Which looks like this when it shows up in Sentry:

[nefarius]

Thanks a lot for this in-depth response. I will try to check everything you mentioned and see what works and what doesn't 👍

[nefarius]

It could be that you're expecting it to come through as a Sentry.Tag... but OpenTelemetry attributes show up as Extra data (not as tags) in Sentry

Yes, correct, I assumed this was the case, wasn't aware of this "tags vs extra data" distinction.

[nefarius]

Alright I try to summarize what I think I found to be the case. Sadly I have to redact a lot from the production environment so I can't just post JSON and unedited screenshot examples here.

So in my HTTP endpoint handler, if I dump Activity.Current to the log, it appears like it corresponds with a Transaction in Sentry. I can find the key-values of the Tags property of the Activity in the Sentry Transaction under OpenTelemetry / attributes:

Activity.Current.Tags

Sentry Trace View / Trace - Transaction

So my custom tags did in fact get sent to Sentry, but under a section that is not indexed and searchable. I verified this with a historic transaction:

This is how it behaves on the transaction. On an individual Sentry Span it behaves like your example (ends up in Additional Data):

So ultimately only

SentrySdk.ConfigureScope(scope =>
{
    scope.SetTag("myApp.userId", request.UserId);
});

leads to "tags" being recognized as such that get indexed and are searchable properly within Sentry.

I can live with that, however wouldn't it be nice to ultimately have Activity "tags" end up as Sentry tags maybe via an extension method or other customizing? The ConfigureScope syntax is a bit clunky in comparison to myActivity.SetTag("name", "value"), what do you think?

Cheers

[jamescrosswell]

Thanks @nefarius. It would be nice if these ended up as Tags in Sentry but I'm not sure how we would do it...

What we see, when capturing an OpenTelemetry span, is a bunch of Activity.TagObjects (of type IEnumerable<KeyValuePair<string, object?>>).

The searchable Tags in Sentry are of type IReadOnlyDictionary<string, string> so we can't store all of the TagObjects as Sentry Tags.

We attribute special meaning to some TagObjects based on the Key in the KVP. For example, we know the URL is important in ASP.NET Core apps:

sentry-dotnet/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs

Line 195 in 9b29d1f

var url = attributes.UrlFullAttribute();

The key is really the only thing that provide some hint... but we can't use this for custom tags that you've defined.

If you wanted to make some specific TagObjects searchable, you could potentially write some custom logic in the SetBeforeSendTransaction callback to copy these across.

In dotnet, in particular, I could think of one other way that we could do this to effectively add a custom property to the Activity class. It would be a bit weird though, since we'd be adding a Sentry specific extension to the Activity class - a property that would only be recognised by the Sentry open telemetry implementation... at which point I'd ask the question, "Why use OpenTelemetry at all?" If you're OK using Sentry specific instrumentation, Sentry has it's own Transactions and Spans (both of which support Tags out of the box)...

[nefarius]

at which point I'd ask the question, "Why use OpenTelemetry at all?" If you're OK using Sentry specific instrumentation, Sentry has it's own Transactions and Spans (both of which support Tags out of the box)...

You partially answered that already; it has to do with the history of when which tech has even seen the light of day. Before Sentry, I added OTEL integration into my servers; the .NET Activity class being involved and "just working" was more of an implementation accident (copypasta from tutorial code 😉 ). Now that we "officially" switched to Sentry, knowing there exists an "OTEL-to-Sentry-Bridge" I quickly slapped that into my existing code-base since it was easier and faster to see results without having to do major rework.

So what I learned from this; when relying on Sentry-only instrumentation and concepts, OTEL is just in the way and adds no benefits. .NET's own Tracing types like Activity are not tightly coupled to Sentry and more of "it's own thing" as well so I shouldn't really keep using them either (except if I create an Activity it results in a Span in Sentry which is convenient so, aaaaah....).

TL;DR: it's a bit messy thanks to history and there is no problem if Sentry-only concepts and patterns get used 👍

[jamescrosswell]

Yeah, it's unfortunately not as clean as we'd like. There's not always a 1 to 1 mapping between OTel and Sentry.

I'll close this issue for the time. If there's anything you can think that we can do to improve the OTel integration, we're all ears though.