diff --git a/src/current/_includes/v25.3/sidebar-data/troubleshooting.json b/src/current/_includes/v25.3/sidebar-data/troubleshooting.json index 3ef93c9de06..86e82211a09 100644 --- a/src/current/_includes/v25.3/sidebar-data/troubleshooting.json +++ b/src/current/_includes/v25.3/sidebar-data/troubleshooting.json @@ -112,6 +112,12 @@ "urls": [ "/${VERSION}/automatic-cpu-profiler.html" ] + }, + { + "title": "Automatic Go Execution Tracer", + "urls": [ + "/${VERSION}/automatic-go-execution-tracer.html" + ] } ] } diff --git a/src/current/v25.3/automatic-cpu-profiler.md b/src/current/v25.3/automatic-cpu-profiler.md index 1bdcde8af51..50f9ab36dd7 100644 --- a/src/current/v25.3/automatic-cpu-profiler.md +++ b/src/current/v25.3/automatic-cpu-profiler.md @@ -45,3 +45,7 @@ Enabling the automatic CPU profile capture on a cluster will add overhead to the - [P99 latency]({% link {{ page.version.version }}/ui-sql-dashboard.md %}#service-latency-sql-99th-percentile) - P50 latency by creating a [custom chart]({% link {{ page.version.version }}/ui-custom-chart-debug-page.md %}) for the `sql.exec.latency-p50` metric - [CPU usage]({% link {{ page.version.version }}/ui-hardware-dashboard.md %}#cpu-percent) + +## See also + +- [Automatic Go Execution Tracer]({% link {{ page.version.version }}/automatic-go-execution-tracer.md %}) diff --git a/src/current/v25.3/automatic-go-execution-tracer.md b/src/current/v25.3/automatic-go-execution-tracer.md new file mode 100644 index 00000000000..24b3e9bfc69 --- /dev/null +++ b/src/current/v25.3/automatic-go-execution-tracer.md @@ -0,0 +1,57 @@ +--- +title: Automatic Go Execution Tracer +summary: The automatic Go execution tracer can be used to gain fine-grained insight into goroutine scheduling, blocking events, garbage collection, and runtime activity. +keywords: go execution trace, go execution tracing, execution tracer, automatic execution tracer +toc: true +--- + +The automatic Go execution tracer allows operators and support engineers to investigate internal system behavior over time. Use this feature to gain fine-grained insight into goroutine scheduling, blocking events, garbage collection, and runtime activity. + +{{site.data.alerts.callout_info}} +Execution tracing introduces a performance overhead. This feature is primarily intended for use under the guidance of [Cockroach Labs Support](https://support.cockroachlabs.com/). +{{site.data.alerts.end}} + +When enabled, the automatic Go execution tracer captures runtime execution traces on a recurring schedule and stores them in the node's log directory. These traces are collected in the background and can be analyzed using Go’s standard tooling (e.g., [`go tool trace`](https://pkg.go.dev/runtime/trace)). + +To prevent disk overuse, older traces are automatically garbage collected based on a configurable storage limit. + +## Configuration + +You can configure automatic Go execution trace capture with the following [cluster settings]({% link {{ page.version.version }}/cluster-settings.md %}): + +Cluster Setting | Description | Default Value +----------------|-------------|--------------- +`obs.execution_tracer.interval` | Enables periodic execution tracing on each node and defines how often traces are collected. Set to a positive duration (e.g., `10m`, `1h`) to enable. Set to `0` to disable (default). | `0` +`obs.execution_tracer.duration` | Sets how long each execution trace runs. Use short durations (e.g., 5-15 seconds) to produce traces that are easier to analyze. | `10s` +`obs.execution_tracer.total_dump_size_limit` | Sets a soft limit on the total disk space used by execution trace files. When the limit is exceeded, the oldest files are deleted. | `4GiB` + +## Accessing Go execution traces + +- The automatic Go execution tracer saves the captured traces to disk on each node's file system in the [logging directory]({% link {{ page.version.version }}/configure-logs.md %}#logging-directory). The default path is `cockroach-data/logs/executiontrace_dump`. +- To access these Go execution traces, you must use the node file system. +- Each file represents a single trace session, includes the `executiontrace.` prefix and `.out` extension, and is timestamped to support correlation with other logs and profiles. For example, the trace file could be named `executiontrace.2025-07-14T18_41_22.216.out`. +- Enabling the automatic Go execution tracer does add Go execution traces to [debug zips]({% link {{ page.version.version }}/cockroach-debug-zip.md %}) in the `nodes/*/executiontrace` paths. + +## Best Practices + +- Trace duration should be short (e.g., 5s–15s). Shorter trace files are easier to parse and load into external tools. +- Storage limit should be generous to allow sufficient trace history. Avoid overly tight disk caps unless necessary. +- Enable tracing temporarily when troubleshooting runtime behavior or under support guidance, as it may slightly impact performance. + +## Performance Impact + +Enabling execution tracing incurs a low overhead, estimated at 1–2% based on upstream Go benchmarks ([Go blog, 2024](https://go.dev/blog/execution-traces-2024#low-overhead-tracing)). No additional CockroachDB-specific benchmarks are available at this time. + +## Example + +To enable 10-second traces every 15 minutes with a storage limit of `8GiB`, configure the following settings: + +```sql +SET CLUSTER SETTING obs.execution_tracer.interval = '15m'; +SET CLUSTER SETTING obs.execution_tracer.duration = '10s'; +SET CLUSTER SETTING obs.execution_tracer.total_dump_size_limit = '8GiB'; +``` + +## See also + +- [Automatic CPU Profiler]({% link {{ page.version.version }}/automatic-cpu-profiler.md %}) diff --git a/src/current/v25.3/troubleshooting-overview.md b/src/current/v25.3/troubleshooting-overview.md index 1ada8bdc479..e4152c431ad 100644 --- a/src/current/v25.3/troubleshooting-overview.md +++ b/src/current/v25.3/troubleshooting-overview.md @@ -33,3 +33,4 @@ If you experience an issue when using CockroachDB, try these steps to resolve th - [`cockroach debug zip`]({% link {{ page.version.version }}/cockroach-debug-zip.md %}) - [`cockroach debug tsdump`]({% link {{ page.version.version }}/cockroach-debug-tsdump.md %}) - [Automatic CPU Profiler]({% link {{ page.version.version }}/automatic-cpu-profiler.md %}) + - [Automatic Go Execution Tracer]({% link {{ page.version.version }}/automatic-go-execution-tracer.md %}) \ No newline at end of file