diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 61fd7293b9..603cea9a8e 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -17,9 +17,83 @@ We run two types of automated benchmark tests for Phase 2: Benchmarks](https://open-telemetry.github.io/otel-arrow/benchmarks/nightly/)** - Comprehensive test suites run nightly -Both provide performance metrics for the OTAP dataflow engine, for various -scenarios. - +Both provide performance metrics for the OTAP dataflow engine for various +scenarios. Unless otherwise specified, all tests run on a single CPU core. + +#### 1. Filter + OTel Collector Comparison + +**URL:** + +Tests a filter scenario where a filter processor drops 95% of logs. Processes +approximately 100k logs/sec input with ~5k logs/sec output. The benchmark page +includes a direct comparison with the equivalent OTel Collector performing the +same filtering operation. + +#### 2. Backpressure + +**URL:** + +Measures backpressure impact with `wait_for_result` set to true on the dataflow +engine receivers. Processes approximately 100k logs/sec input and output. The +pipeline includes an attribute processor configured to rename an attribute, +which forces the dataflow engine to perform in-memory representation and +conversion rather than operating in pass-through mode. + +#### 3. Syslog + +**URL:** + +Tests syslog ingestion via UDP with two variations: + +- Basic syslog message format +- CEF (Common Event Format) formatted messages + +Processes approximately 5k logs/sec input and output. + +#### 4. Normal Load + +**URL:** + +Standard load test processing 100k records/sec input and output on a single CPU +core. This test runs with each commit to main. + +#### 5. Saturation and Scaling + +**URL:** + +Tests performance at saturation across different CPU configurations: 1, 2, 4, 8, +and 16 cores. + +*TODO: Update test output to include scalability ratios in addition to raw +throughput numbers.* + +#### 6. Idle State + +**URL:** + +Measures resource consumption in idle state with two variations: + +- Single core configuration +- All-cores configuration (128 cores) + +#### 7. Binary Size + +**URL:** + +Tracks the binary size of the dataflow engine for Linux ARM64 and AMD64 +architectures over time. + +### Metrics Collected + +All benchmark tests measure the following metrics: + +- **Logs/sec input** - Input throughput +- **Logs/sec output** - Output throughput +- **RAM** - Average and maximum memory usage +- **Normalized CPU** - Average and maximum CPU usage, normalized to 0-100% where + 100% represents full utilization of all available cores. For example, in a + 4-core test, 80% means 3.2 cores are being used (0.8 X 4 cores) +- **Network bytes/sec** - Input and output network bandwidth ### Learn More About Phase 2