Null-spans, or disabled spans? Null tracer?

[kristjanvalur]

In our stack, I have been using this project for a particular system. Some colleagues have been adding tracing to Rust projects. We have various components here and there.

One feature they talk about is to check if an incoming request has a trace, and only in this case propagate the tracing through the system.

I don't see how this is possible transparently with dd-trace-cpp. The library relies heavily on auto allocated Span objects and it would be cumbersome to conditionally create and propagate spans.

Wouldn't it be cleaner to be able to set a flag on the captured or root span which then propagates to child spans, declaring the whole trace as disabled, and making trace injection a no-op?

Similarly, a report_traces member might control trace reporting on a trace-by-trace basis.

This would also tie in with a NullTracer a tracer you could create if you don't want any tracing to happen, or if tracer configuration somehow failed. Such a Tracer would create normal spans, but could optionally set the default to not inject any traces. ( I have mentioned this elsewhere, such a Null tracer would allow application code to be unchanged even if the library failed to configure correctly, or if it was decided to override all environment variables and not do any tracing at all)

In short, two flags:

class Span {
   ...
   void  set_report_trace(bool);  // call this to set whether this trace should be reported or not.
   void set_inject_trace(bool);  // call this to set whether this trace should be injected into outgoing dicts or not
};

This would affect the entire trace, would probably be booleans on the trace.
A Null tracer would have both false, although it might want to create trace_ids and inject them for outgoing traces
A tracer might also have a flag as to whether to extract would be a no-op or not.

Having such extra flags might allow us to:

• customize trace handling for incoming traces, disabling trace reporting and trace propagation if there are no incoming traces.
• globally disable trace extraction from the Tracer
• customize the behaviour reporting of the library in null mode, e.g. does it create new trace ids and propagate them, or not?

It is possible to implement the above features by wrapping the library, and I have done so for our implementation where custom classes contain optional std::unique_ptr members to a dd::Span, but having it supported by the core library might be useful for other integrators.

Any thoughts?

[dmehala]

Hi @kristjanvalur

Thanks for taking the time to describe the issue you're encountering.

Currently, the library doesn't have built-in support for no-op mode, and I understand how this syntactic sugar could be useful in your case. It's a reasonable request, and I'll prioritize exploring enhancements to improve the usability of the library early in January 2025.

In the meantime, you may need to implement a custom solution. One approach is to create a Span interface with two implementations:

• NoopSpan: A no-op implementation that does nothing (useful when no span is extracted from an incoming request).
• DatadogSpan: A wrapper around datadog::tracing::Span that forwards method calls to the actual span object.

Then depending of extract_span instanciate either a NoopSpan or DatadogSpan.

To illustrate:

namespace mainframe {
   class ISpan {
      public:
         virtual std::unique_ptr<ISpan> create_child(const SpanConfig& config) const = 0;
         ...
   };

   class NoopSpan : ISpan {
   public:
      std::unique_ptr<ISpan> create_child(const SpanConfig&) const { return std::make_unique<NoopSpan>(); }
   };

   class DatadogSpan : ISpan {
      datadog::tracing::Span span_;
      
      public:
         std::unique_ptr<ISpan> create_child(const SpanConfig& config) const override {
            return std::make_unique<DatadogSpan>(span_.create_child(config));
        }
   };

   std::unique_ptr<ISpan> handle_request(Request& req) {
      ...
      auto maybe_span = tracer.extract_span(req);
      if (!maybe_span) {
         return std::make_unique<NoopSpan>();
      }

      return std::make_unique<DatadogSpan>(*maybe_span);
   }
}

The handle_request function demonstrates how you could use these objects conditionally.

Although this solution involves writing some boilerplate code, it provides flexibility and allows you to decouple your application logic from the specific behaviour if tracing context exists or not.

I know it's cumbersome, you probably don't want to write it, unfortunately at the moment, I can't offer an out of the box solution to handle this scenario directly.

Let me know if you need further clarification.

[kristjanvalur]

Thanks for your reply.
In fact, I have already written a bunch of boilerplate to work around the auto semantics of the dd::Span object. Since I'm already doing that, I can in fact add the functionality that I describe. However, my musings here are intended to make it simpler for future integrators to enjoy more flexibility.
Also, the documentation and examples do not show much in the way of async and optional handling.

I am working on an integration for UnrealEngine. Mostly this is a set of C++ modules and interfaces, but also some Blueprint classes. There is a thread local span stack and provision for storing spans in lambdas and other callbacks. Also a bunch of niceties such as unicode transforms. Once this is ready I'll open up the repo and it could be used as an example reference.

In our game, the server performs a lot of http requests, and gprc requests and we need more visibility, hence the integration. We are also looking at integrating this with Unreal's own RPC mechanism, but it remains to be seen how widely we can deploy this. Needless to say, we don't want to send traces from a client, (end user game), even if the user accidentally sets a DD_AGENT_HOST variable in his environment. But we may want to inject trace ids into outgoing requests to be able to group them on the server, even if the traces from the client are not sent themselves.

All work in progress. Cheers!