Skip to content

perf: add comprehensive benchmark suite with CI regression detection#376

Open
dlevy-msft-sql wants to merge 2 commits into
microsoft:mainfrom
dlevy-msft-sql:perf/add-benchmark-suite
Open

perf: add comprehensive benchmark suite with CI regression detection#376
dlevy-msft-sql wants to merge 2 commits into
microsoft:mainfrom
dlevy-msft-sql:perf/add-benchmark-suite

Conversation

@dlevy-msft-sql
Copy link
Copy Markdown

@dlevy-msft-sql dlevy-msft-sql commented May 2, 2026

Problem

The driver has no systematic benchmark coverage and no CI guardrails against performance regressions. Changes that degrade hot-path performance can merge undetected.

Solution

Add 96 benchmarks covering all core driver hot paths, with automated CI regression detection using benchstat.

Benchmarks Added

Category Count File
Token parsing 10 token_benchmark_test.go
TDS buffer I/O 10 buf_benchmark_test.go
String encoding 10 tds_benchmark_test.go
Type conversion 13 convert_benchmark_test.go
Wire protocol types 17 types_benchmark_test.go
RPC encoding 7 rpc_benchmark_test.go
Bulk copy params 9 bulkcopy_benchmark_test.go
Integration round-trips 14 integration_benchmark_test.go
Connection string parsing 5 msdsn/conn_str_benchmark_test.go

CI Integration

  • Dedicated benchmarks job in GitHub Actions with SQL Server 2025 in Docker
  • Warmup run (-benchtime=100ms -count=1) stabilizes CPU caches and turbo boost before measurement; results are discarded
  • Runs baseline benchmarks on main via git worktree, then PR benchmarks, both with -count=10 -benchtime=1s
  • Compares results using benchstat -alpha=0.01 (p < 0.01 significance threshold)
  • Fails CI on statistically significant regressions exceeding 15% magnitude
  • Reports improvements via ::notice annotations
  • Explicit benchmark pattern avoids pre-existing slow/problematic benchmarks
  • Copies benchmark test files to main worktree for fair baseline comparison
  • Excludes TdsBuffer_Write_Large from regression gate (cache-sensitive ~120ns operation with 30-46% natural variance)

Bug Fix

Fixes a pre-existing deadlock in BenchmarkSelectWithTypeMismatch where defer rows.Close() inside a loop with MaxOpenConns=1 caused connection starvation. Moved rows.Close() to end of loop body.

Testing

  • All 96 benchmarks pass locally (go test -run='^$' -bench=. -benchtime=1s . ./msdsn)
  • Unit tests unaffected (go test ./msdsn ./internal/... ./integratedauth ./azuread)
  • CI workflow validates main-vs-PR comparison end-to-end

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.60%. Comparing base (f92e6f0) to head (0e6b55a).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##             main     #376       +/-   ##
===========================================
+ Coverage   80.70%   96.60%   +15.89%     
===========================================
  Files          35       92       +57     
  Lines        6910    74351    +67441     
===========================================
+ Hits         5577    71826    +66249     
- Misses       1064     2190     +1126     
- Partials      269      335       +66     
Flag Coverage Δ
unittests 96.53% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 59 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dlevy-msft-sql dlevy-msft-sql force-pushed the perf/add-benchmark-suite branch 6 times, most recently from de4feca to 34fa0f6 Compare May 3, 2026 20:17
Add 96 benchmarks covering core driver hot paths:
- Token parsing (10): ParseInfo, ParseDone, ParseReturnStatus, etc.
- TDS buffer I/O (10): Read/Write in small/medium/large sizes
- String encoding (10): Str2ucs2, Ucs22str, ManglePassword
- Type conversion (13): ConvertAssign for all common type pairs
- Wire protocol types (17): ReadFixedType, ReadByteLenType, etc.
- RPC encoding (7): SendRpc with various parameter types
- Bulk copy params (9): BulkMakeParam for common column types
- Integration round-trips (14): Full E2E through SQL Server
- Connection string parsing (5): URL and ADO format variants

CI integration (.github/workflows/pr-validation.yml):
- Runs benchmarks on every PR with SQL Server in Docker
- Compares PR branch against main using benchstat
- Uses -count=10 and -alpha=0.01 for statistical rigor
- Fails CI on statistically significant regressions (p<0.01)
- Reports improvements via GitHub Actions notices
- Explicit benchmark pattern avoids slow pre-existing benchmarks
- Copies benchmark files to main worktree for fair comparison

Also fixes a pre-existing deadlock in BenchmarkSelectWithTypeMismatch
where defer rows.Close() inside a loop with MaxOpenConns=1 caused
connection starvation.
@dlevy-msft-sql dlevy-msft-sql force-pushed the perf/add-benchmark-suite branch from 34fa0f6 to a216232 Compare May 3, 2026 21:07
Comment thread msdsn/conn_str_benchmark_test.go Outdated
Comment thread integration_benchmark_test.go
- Extract common Parse benchmark loop into benchmarkParse helper

- Add BenchmarkRoundTrip_MessageQuery covering the sqlexp message-based query loop
@dlevy-msft-sql dlevy-msft-sql self-assigned this May 11, 2026
@dlevy-msft-sql dlevy-msft-sql marked this pull request as ready for review May 11, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants