[Bug] opentelemetry traceState is handled properly in makeWorkflowExporter

[safareli]

What are you really trying to do?

When using otel if workflow has tracestate header like this:

exporting span from worker fails

Describe the bug

getting this errors:

TypeError: _a.serialize is not a function
    at sdkSpanToOtlpSpan (/app/node_modules/.pnpm/@opentelemetry+otlp-transformer@0.57.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-transformer/src/trace/internal.ts:44:33)
    at <anonymous> (/app/node_modules/.pnpm/@opentelemetry+otlp-transformer@0.57.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-transformer/src/trace/internal.ts:159:11)
    at Array.map (<anonymous>)
    at spanRecordsToResourceSpans (/app/node_modules/.pnpm/@opentelemetry+otlp-transformer@0.57.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-transformer/src/trace/internal.ts:158:34)
    at createExportTraceServiceRequest (/app/node_modules/.pnpm/@opentelemetry+otlp-transformer@0.57.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-transformer/src/trace/internal.ts:110:20)
    at Object.serializeRequest (/app/node_modules/.pnpm/@opentelemetry+otlp-transformer@0.57.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-transformer/src/trace/json/trace.ts:26:52)
    at OTLPExportDelegate.export (/app/node_modules/.pnpm/@opentelemetry+otlp-exporter-base@0.57.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-exporter-base/src/otlp-export-delegate.ts:69:48)
    at OTLPTraceExporter.export (/app/node_modules/.pnpm/@opentelemetry+otlp-exporter-base@0.57.2_@opentelemetry+api@1.9.0/node_modules/@opentelemetry/otlp-exporter-base/src/OTLPExporterBase.ts:32:26)
    at Object.fn (/app/node_modules/.pnpm/@temporalio+interceptors-opentelemetry@1.11.7_@temporalio+activity@1.11.7_@temporalio+c_49fd2775686fed0552daf86155185cd7/node_modules/@temporalio/interceptors-opentelemetry/src/worker/index.ts:86:24)
    at <anonymous> (/app/node_modules/.pnpm/@temporalio+worker@1.11.7/node_modules/@temporalio/worker/src/worker.ts:1364:23)
    at Array.map (<anonymous>)
    at Worker.processSinkCalls (/app/node_modules/.pnpm/@temporalio+worker@1.11.7/node_modules/@temporalio/worker/src/worker.ts:1362:21)
    at Worker.handleActivation (/app/node_modules/.pnpm/@temporalio+worker@1.11.7/node_modules/@temporalio/worker/src/worker.ts:1208:22)

Minimal Reproduction

reproducing this locally was bit hacky.
had to add this so tracestate was passed to temporal when invoking a workflow via js client.

then once the trace state is passed to workflow in the makeWorkflowExporter that's passed to Worker.create we get traceState that is instance of Object and not an instance of expected TraceState. via debugger you can see this:

because of this then at some point when span is converted to an request to otel collector as there is traceState .serialize is called on it but this object has no methods only that _internalState which is private prop of the TraceState see:

I suppose the trace state is moved across different JS contexts (from workflow code to host) and information that this object is instance of TraceState is lost and on the other end we just get Object.

Environment/Versions

• OS and processor: M1 Mac
• Temporal SDK Version: 1.11.7
• Are you using Docker or Kubernetes or building Temporal from source? no

hacky fix on my end was to do this:

const fixTemporalBug = (traceState: TraceState): TraceState => {
  const res = new TraceState();

  (res as any)._internalState = (traceState as any)._internalState;

  return res;
};

// inlined version of makeWorkflowExporter that fixes traceState bug in Temporal
const makeWorkflowExporterFixed = (
  exporter: SpanExporter,
  resource: Resource,
): InjectedSink<OpenTelemetryWorkflowExporter> => {
  return {
    export: {
      fn: (info: WorkflowInfo, spanData: SerializableSpan[]): void => {
        const spans = spanData.map((serialized) => {
          Object.assign(serialized.attributes, info);
          // Spans are copied over from the isolate and are converted to ReadableSpan instances
          const {
            spanContext: { traceState, ...spanContext },
            ...rest
          } = serialized;

          return {
            spanContext: (): SpanContext => ({
              ...spanContext,
              traceState: traceState == null ? traceState : fixTemporalBug(traceState as any),
            }),
            resource,
            ...rest,
          };
        });

        // Ignore the export result for simplicity
        exporter.export(spans, () => undefined);
      },
    },
  };
};

[mjameswh]

Thanks for the detailed report, that's helpful.

This is actually the second time I here about this issue (see previous discussion on our Slack workspace), though we never got far enough in investigating this the other time.

Interestingly, however, you are only the second user reporting this, which makes me think that this may only show up in some specific environment. Do you happen to have a dependency on the dd-trace package? Or any other particular trace/otel-related dependency?

Both your trace and the other user's traces show the @opentelemetry/otlp-transformer@0.57.2 package, whereas our SDK is built and tested against v0.46.0 without problem. But otlp-transformer introduced that specific ctx.traceState?.serializer() line in v0.35.1. So I suppose traceState would "normally" be undefined unless something else is present that forces those to be generated...

In any case, this is indeed an issue from our side, and it needs to be fixed. As you correctly figured out, the trace data are sent from the Workflow Worker thread to the main thread, which goes through V8's serialization protocol. Functions and classes do not survive that serialization, and there's really nothing we can do about that. That means we'll probably need to adopt a solution kind of similar to the hack you figured out. Or maybe we can/should just drop the traceState entirely, WDYT?

[safareli]

Instances of TraceState have .serialize() method which converts it to string. long term solution in workflow land will perform that method call to get serialized string version of trace state. and then makeWorkflowExporter will addopt my changes but use new TraceState(traceStateSerialized) instead of my hacky way.

[JCMais]

We are having the same issue after adding trace propagation, all temporal packages are on 1.11.7, and here are the other related packages:

        "dd-trace": "5.22.0",
        "@sentry/node": "9.12.0",
        "@sentry/opentelemetry": "9.12.0",
        "@opentelemetry/api": "1.9.0",
        "@opentelemetry/auto-instrumentations-node": "0.56.0",
        "@opentelemetry/core": "1.30.1",
        "@opentelemetry/exporter-metrics-otlp-grpc": "0.57.2",
        "@opentelemetry/exporter-metrics-otlp-http": "0.57.2",
        "@opentelemetry/exporter-trace-otlp-grpc": "0.57.2",
        "@opentelemetry/exporter-trace-otlp-http": "0.57.2",
        "@opentelemetry/host-metrics": "0.35.5",
        "@opentelemetry/instrumentation": "0.57.2",
        "@opentelemetry/resource-detector-gcp": "0.33.0",
        "@opentelemetry/resources": "1.30.1",
        "@opentelemetry/sdk-metrics": "1.30.1",
        "@opentelemetry/sdk-node": "0.57.2",
        "@opentelemetry/sdk-trace-base": "1.30.1",
        "@opentelemetry/sdk-trace-node": "1.30.1",
        "@opentelemetry/semantic-conventions": "1.30.0",

Adding Sentry as related just because the trace propagation is done through Sentry-related headers:

Stack trace:

TypeError: _a.serialize is not a function
at sdkSpanToOtlpSpan (trace/internal.js:14:83)
at trace/internal.js:110:62
at Array.map (<anonymous>)
at spanRecordsToResourceSpans (trace/internal.js:110:42)
at createExportTraceServiceRequest (trace/internal.js:74:24)
at Object.serializeRequest (protobuf/trace.js:27:72)
at OTLPExportDelegate.export (src/otlp-export-delegate.js:44:52)
at OTLPTraceExporter.export (src/OTLPExporterBase.js:29:30)
at Object.fn (lib/worker/index.js:88:32)
at worker/lib/worker.js:899:29
at Array.map (<anonymous>)
at Worker.processSinkCalls (worker/lib/worker.js:897:41)
at Worker.handleActivation (worker/lib/worker.js:778:32)