Describe the enhancement:
The Go toolchain has support for Profile-Guided Optimization (PGO), available since Go 1.20 (and enabled by default via -pgo=auto since Go 1.21). PGO feeds a CPU pprof profile from representative production runs back into the compiler, which then uses that data to make more informed optimization decisions.
Benchmarks from the Go team indicate PGO builds show roughly a 2-14% performance improvement across a representative set of Go programs, and this is expected to improve further as more compiler optimizations become PGO-aware. Given Elastic Agent's performance sensitivity (CPU/memory footprint is a frequent concern for users running it alongside their workloads), it's worth experimenting with PGO to see whether we get a meaningful, low-risk win.
A quick test of running the Beats BenchmarkFilebeatOTelThroughputMockES bench with PGO yields a 4-8% increase in EPS for all presets on a 4 core machine. This is by no means production grade data of a OpenTelemetry Collector like Elastic Agent, but it showcases there are gains to be had with this approach.
The biggest downside of pgo is the burden around collecting meaningful profiles from a production environment. I believe using benchbuilder to instrument Elastic Agent benchmarks to produce pprof files would be a efficient way to have high grade profiles available from multiple production-like sources. We already have something like this in the benchmark summary. We could then concatenate those to produce a unified PGO profile that could be used for releases.
At a high level, PGO would involve an "AutoFDO" style approach:
- Collecting CPU pprof profiles from a representative Elastic Agent deployment (e.g., via
net/http/pprof or an internal continuous profiling pipeline), ideally sampled across multiple instances/times to average out noise.
- Merging profiles with
go tool pprof -proto if collected from multiple sources.
- Committing the resulting profile as
default.pgo in the relevant main package directory(ies), so go build picks it up automatically.
- Establishing a recurring process to refresh the profile as the codebase evolves, since stale profiles gracefully degrade in benefit but shouldn't actively regress performance.
What is the definition of done?
Describe the enhancement:
The Go toolchain has support for Profile-Guided Optimization (PGO), available since Go 1.20 (and enabled by default via
-pgo=autosince Go 1.21). PGO feeds a CPU pprof profile from representative production runs back into the compiler, which then uses that data to make more informed optimization decisions.Benchmarks from the Go team indicate PGO builds show roughly a 2-14% performance improvement across a representative set of Go programs, and this is expected to improve further as more compiler optimizations become PGO-aware. Given Elastic Agent's performance sensitivity (CPU/memory footprint is a frequent concern for users running it alongside their workloads), it's worth experimenting with PGO to see whether we get a meaningful, low-risk win.
A quick test of running the Beats
BenchmarkFilebeatOTelThroughputMockESbench with PGO yields a 4-8% increase in EPS for all presets on a 4 core machine. This is by no means production grade data of a OpenTelemetry Collector like Elastic Agent, but it showcases there are gains to be had with this approach.The biggest downside of pgo is the burden around collecting meaningful profiles from a production environment. I believe using benchbuilder to instrument Elastic Agent benchmarks to produce pprof files would be a efficient way to have high grade profiles available from multiple production-like sources. We already have something like this in the benchmark summary. We could then concatenate those to produce a unified PGO profile that could be used for releases.
At a high level, PGO would involve an "AutoFDO" style approach:
net/http/pprofor an internal continuous profiling pipeline), ideally sampled across multiple instances/times to average out noise.go tool pprof -protoif collected from multiple sources.default.pgoin the relevant main package directory(ies), sogo buildpicks it up automatically.What is the definition of done?
default.pgoor explicit-pgoflag) using the collected profile.