[FEATURE] Customize or standardize OpenTelemetry span names and attributes

[tvardero]

Problem

I want to integrate FusionCache with Sentry caching insights. Here are the docs for manual instrumentation.

Currently I'm using OpenTelemetry instrumentation of FusionCache, which then exported to Sentry SDK. Sentry does not have its own native integration with FusionCache, so doesn't FusionCache with Sentry.

Currently in FusionCache span.op names are like get or set from cache and others, which are not recognized by Sentry as caching spans and are treated as just regular trace spans.

Solution

• A way to customize span operation names via options/factories
• A way to override span factory of fusion cache with dependency injection
• Or update fusion cache default span.op names, so they line up with what looks like OTel semantic conventions (here and here). There are no conventions for caching in OTel docs, only for databases, however Sentry has their own conventions here and here.

Alternatives

• Manually instrumenting caching layer
• Investigating source code of FusionCache OTel integration and finding what service to override and register in DI.

Additional context

I just like cool graphs.

[tvardero]

Some cool pics for context:

[EikeSchwass]

I would like to piggyback of this and would propose that more control is also given if a span is recorded at all.
For example I don't care so much about "get from cache" (in fact the number of these spans is actively hurting observability), but would still like to see "execute factory".

[jodydonetti]

Hi @tvardero

I want to integrate FusionCache with Sentry caching insights
[...]
I just like cool graphs.

Yes! I'll get back to you tomorrow with more time, but for now thanks for pointing out the Sentry OTEL semantic conventions.
I'll give you more background tomorrow, but back then I think there was nothing like that.

Anyway, for now: this is something that I'm really interested in!

PS: the graphs look awesome 😲

[jodydonetti]

Hi @EikeSchwass

For example I don't care so much about "get from cache" (in fact the number of these spans is actively hurting observability), but would still like to see "execute factory".

When you enable the instrumentation you can set IncludeMemoryLevel, IncludeDistributedLevel, etc for both traces and metrics.

Hope this helps.

[EikeSchwass]

Hi @EikeSchwass

For example I don't care so much about "get from cache" (in fact the number of these spans is actively hurting observability), but would still like to see "execute factory".

When you enable the instrumentation you can set IncludeMemoryLevel, IncludeDistributedLevel, etc for both traces and metrics.

Hope this helps.

Thanks for the quick reply!

Even if I set all of these settings to false, I still get these spans:

The only difference is that the highlighted span is indeed no longer collected, but the "get from cache" span always is.
Am I missing something?

This is my OTel setup:

otelBuilder.WithTracing(tracing =>
{
    var tracerProviderBuilder = tracing
        .AddAspNetCoreInstrumentation()
        .AddSqlClientInstrumentation()
        .AddHttpClientInstrumentation()
        .SetErrorStatusOnException()
        .AddFusionCacheInstrumentation(c =>
        {
            c.IncludeDistributedLevel = false;
            c.IncludeMemoryLevel = false;
            c.IncludeBackplane = false;
        });
    // ...
}

[jodydonetti]

The only difference is that the highlighted span is indeed no longer collected, but the "get from cache" span always is.

If you mean the first "get or set from cache" then yes, that is the top-level span that represents the GetOrSet<T>() method call, and that cannot be disabled (I think it's also a best practice in OTEL world, but i may be wrong).

[EikeSchwass]

The only difference is that the highlighted span is indeed no longer collected, but the "get from cache" span always is.

If you mean the first "get or set from cache" then yes, that is the top-level span that represents the GetOrSet<T>() method call, and that cannot be disabled (I think it's also a best practice in OTEL world, but i may be wrong).

The problem is that we make exzessive use of caching due to legacy reasons (same values are requested a huge number of times per web request) which means we hit FusionCache very often (with a cache hit-rate of >99% in these cases).

Before we integrated FusionCache we had ~40 spans and sub-spans per request trace, now we sometimes have up to 5000, and basically all of them are "get from cache" or "get or set from cache".
We would still like to know when cache entry factories are executed and some other FusionCache spans are also valuable to us, but we can't enable tracing for FusionCache as is, because it's almost impossible to view some traces.

We will go with a custom OTel Processor and filter these spans ourselves, but I would image that many people would like to disable these spans (also for performance reasons).

This is nitpicking of course, because FusionCache is absolutely amazing otherwise, so thank you!

[jodydonetti]

Hi @EikeSchwass

The problem is that we make exzessive use of caching due to legacy reasons (same values are requested a huge number of times per web request) which means we hit FusionCache very often (with a cache hit-rate of >99% in these cases).

Before we integrated FusionCache we had ~40 spans and sub-spans per request trace, now we sometimes have up to 5000, and basically all of them are "get from cache" or "get or set from cache". We would still like to know when cache entry factories are executed and some other FusionCache spans are also valuable to us, but we can't enable tracing for FusionCache as is, because it's almost impossible to view some traces.

Ok, got it, thanks for the extra details.

About the factory trace: one way may be to emit an activity inside of the factory yourself, and disable traces for FusionCache altogether.

But if you still care about some other traces this will not be the right way, imho.

We will go with a custom OTel Processor and filter these spans ourselves, but I would image that many people would like to disable these spans (also for performance reasons).

Mind you, I'm not an OTEL expert, so I'm contacting one right now:we'll see what they say.

Having said that, I was thinking about an OTEL processor too since this is what has been suggested to me in the past for other tangential stuff about configuration: basically I was asking how much I should've let people configure, and a couple of OTEL people told me the correct way was via processors & similar stuff.

Overall I'd like to take the "golden path" approach here, to avoid deviating from the expectations & best practices that comes from the OTEL world.

Let me investigate this.

This is nitpicking of course, because FusionCache is absolutely amazing otherwise, so thank you!

Thanks, aprpeciate that 🙂

[jodydonetti]

I hereby invoke the immense OTEL knowledge of @martinjt 🧙‍♂️

Short recap of the pending questions.

Caching Semantic Convention

As asked by @tvardero , what is the current state of affair about a semantic convention for caching?

Mainly:

• is one being defined by the OTEL committee?
• is the one by Sentry (here) a kind of de facto standard being embraced by the community?
• should we start a roundtable with Sentry (and others?) to define one?

Also: should a library author like me allow users to change/customize the semantic convention being used, to allow them to specify a custom one, or is this not the best approach in OTEL world?

Span Inclusion Configuration

As asked by @EikeSchwass , what would it be the best practice to pick which spans to include from a library author perspective?

Mainly:

• should I allow users to include/exclude every "level/part" of the library?
• should users use an OTEL processor?
• maybe something else?

Thanks!

Whatever input you may give us, it would be more than welcome 🙏