traceparent or X-Opaque-Id headers to query tags

[steve-chavez]

Query tags are comments on the SQL query(SELECT ... FROM tbl /*key=value/*) , they're useful for getting context on the database logs.

References:

• https://cloud.google.com/blog/topics/developers-practitioners/enable-query-tagging-sqlcommenter-understand-application-impact-database-performance
• https://docs.vividcortex.com/general-reference/query-tags/
• https://hasura.io/docs/latest/observability/query-tags/

We could include

• A request_id(request_id=...). It could be generated by us or configured to be obtained from a header since a proxy already adds a request id: nginx has request_id, cloudflare has cf-ray, etc. We could also include this in the response header(configurable), to make it easier for users to search the generated query in the logs.
• The HTTP method(request_method=GET)
• The path (request_path=tbl)

It should be configurable

db-query-tags = true
log-request-id = generated|arbitrary-header-name # could be `cf-ray`, `x-request-id`, etc

[steve-chavez]

Just found out that there is a standard HTTP header for doing the above: traceparent.

References:

• https://www.w3.org/TR/trace-context/#traceparent-header
• https://uptrace.dev/opentelemetry/opentelemetry-traceparent.html

I found it on the Elasticsearch docs, which uses an X-Opaque-Id for the same purpose.

Also, the tracestate header allows key value pairs for extra vendor information.

We could include this information on the query tags.

---

One problem with traceparent is that it has its own format

traceparent: 00-80e1afed08e019fc1110464cfa66635c-7a085853722dc6d2-01

The traceparent header uses the version-trace_id-parent_id-trace_flags format where:

version is always 00.
trace_id is a hex-encoded trace id.
span_id is a hex-encoded span id.
trace_flags is a hex-encoded 8-bit field that contains tracing flags such as sampling, trace level, etc.

Though trace_id is 16 bytes and matches the length of nginx's request_id and cloudflare cf-ray, those systems would still have to send a modified header for us to consume.

It also requires parsing and validating the header value as mentioned on the docs.

On contrast the X-Opaque-Id convention can have an arbitrary value inside. It might be simpler to adopt this one for starters.

[steve-chavez]

There's also the x-request-id header.

With these variety of options think it'd be easier to just make it configurable.

So server-trace-id-header as config, empty by default.

[steve-chavez]

A server-trace-header config was added on #2692.

[wolfgangwalther]

Should we somehow add the value passed through this header to the request logs, too?

[steve-chavez]

@wolfgangwalther What would the format look like?

Also, how would that help with observability? Right now I'm mostly thinking of parsing the db logs.

[wolfgangwalther]

What would the format look like?

No idea, yet. It probably doesn't fit CLF, we'd might need Extended Log Format.

Also, how would that help with observability? Right now I'm mostly thinking of parsing the db logs.

I had some cases where I wasn't sure which component (nginx, postgrest, postgresql) was actually throwing an error, when I got some failed requests on the client. It would be nice to just put that trace header on the request on the client and then grep all logs to see where it's logged and where it's not. I also had some cases where I wanted to track 2xx responses, while setting up nginx to cache responses. In those cases it would have been easier to see whether nginx was responding directly from cache or was hitting PostgREST.

[elimisteve]

So server-trace-id-header as config, empty by default.

To be clear, what was implemented as new a new config option is called server-trace-header, not server-trace-id-header.

[olee]

Should we somehow add the value passed through this header to the request logs, too?

That would actually be really nice. Right now there doesn't seem to be any way to create full trace logs from frontend to backend because there's no way to identify the request log in postgrest with the requests from frontend / elsewhere.

[steve-chavez]

Related to #3415 (comment)