Skip to content

axiom: pool compression writers to avoid per-call allocations#413

Merged
lukasmalkmus merged 2 commits into
mainfrom
lukasmalkmus/ywpnpuvpvqsy
Mar 2, 2026
Merged

axiom: pool compression writers to avoid per-call allocations#413
lukasmalkmus merged 2 commits into
mainfrom
lukasmalkmus/ywpnpuvpvqsy

Conversation

@lukasmalkmus

@lukasmalkmus lukasmalkmus commented Mar 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

Pool zstd.Encoder and gzip.Writer instances in encoder.go to eliminate the ~4 MB allocation per zstd.NewWriter call (and ~800 KB per gzip.NewWriterLevel). All code paths benefit: ZstdEncoder(), GzipEncoder(), GzipEncoderWithLevel(), and IngestEvents in datasets.go. Adapters (zap, zerolog, slog, etc.) need zero changes since they call the public encoder functions.

Adds a public WriteResetter interface and generic NewPooledEncoder[T]() helper so users can pool custom compressors the same way.

Supersedes #412 with a consolidated approach that covers all encoder call sites, not just datasets.go.

Benchmarks

                        │     old      │              new               │
                        │     B/op     │     B/op      vs base         │
ZstdEncoder_Allocs      19,138 Ki       473 Ki        -97.53%
GzipEncoder_Allocs         796 Ki        51 Ki        -93.59%
ZstdEncoder_Parallel    19,394 Ki       426 Ki        -97.80%
GzipEncoder_Parallel     1,134 Ki       406 Ki        -64.18%
Encoder/zstd            19,138 Ki       463 Ki        -97.58%
Encoder/gzip               796 Ki        48 Ki        -93.96%

Compression ratio and output size are unchanged (all samples equal).

@lukasmalkmus lukasmalkmus self-assigned this Mar 2, 2026
@lukasmalkmus lukasmalkmus requested a review from bahlo March 2, 2026 15:24
@lukasmalkmus lukasmalkmus force-pushed the lukasmalkmus/ywpnpuvpvqsy branch from 35dcde5 to caa62a5 Compare March 2, 2026 15:30
@lukasmalkmus lukasmalkmus marked this pull request as ready for review March 2, 2026 18:00
@tsenart

tsenart commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

Can you share the benchmark results for time and allocations too?

@lukasmalkmus

Copy link
Copy Markdown
Collaborator Author

@tsenart Here's the full benchstat output (count=20, Apple M4 Pro, wall power):

goos: darwin
goarch: arm64
pkg: github.com/axiomhq/axiom-go/axiom
cpu: Apple M4 Pro
                        │ old                │              new                   │
                        │       sec/op       │   sec/op     vs base               │
ZstdEncoder_Allocs-14            24.49m ± 1%   23.92m ± 0%  -2.32% (p=0.000 n=20)
GzipEncoder_Allocs-14            61.54m ± 0%   60.91m ± 0%  -1.02% (p=0.000 n=20)
ZstdEncoder_Parallel-14          4.836m ± 3%   4.569m ± 2%  -5.53% (p=0.000 n=20)
GzipEncoder_Parallel-14          6.445m ± 1%   6.634m ± 5%       ~ (p=0.068 n=20)
Encoder/encoder=gzip-14          61.22m ± 1%   62.14m ± 1%  +1.50% (p=0.040 n=20)
Encoder/encoder=zstd-14          24.46m ± 0%   23.84m ± 0%  -2.52% (p=0.000 n=20)
geomean                          20.32m        20.07m       -1.20%

                        │ old                │              new                      │
                        │        B/op        │     B/op       vs base                │
ZstdEncoder_Allocs-14         19138.3Ki ± 0%   427.0Ki ±  4%  -97.77% (p=0.000 n=20)
GzipEncoder_Allocs-14          795.83Ki ± 0%   43.02Ki ± 97%  -94.59% (p=0.000 n=20)
ZstdEncoder_Parallel-14       19356.5Ki ± 0%   326.6Ki ± 18%  -98.31% (p=0.000 n=20)
GzipEncoder_Parallel-14        1091.4Ki ± 0%   368.4Ki ±  5%  -66.24% (p=0.000 n=20)
Encoder/encoder=gzip-14        795.86Ki ± 0%   43.00Ki ±  6%  -94.60% (p=0.000 n=20)
Encoder/encoder=zstd-14       19138.3Ki ± 0%   418.9Ki ±  2%  -97.81% (p=0.000 n=20)
geomean                         4.025Mi        184.8Ki        -95.52%

                        │ old                │              new                     │
                        │     allocs/op      │  allocs/op   vs base                │
ZstdEncoder_Allocs-14             671.0 ± 0%   609.0 ±  0%   -9.24% (p=0.000 n=20)
GzipEncoder_Allocs-14            20.000 ± 0%   7.000 ±  0%  -65.00% (p=0.000 n=20)
ZstdEncoder_Parallel-14           672.0 ± 0%   609.0 ±  0%   -9.38% (p=0.000 n=20)
GzipEncoder_Parallel-14          21.000 ± 0%   8.000 ± 12%  -61.90% (p=0.000 n=20)
Encoder/encoder=gzip-14          21.000 ± 0%   7.000 ±  0%  -66.67% (p=0.000 n=20)
Encoder/encoder=zstd-14           671.0 ± 0%   609.0 ±  0%   -9.24% (p=0.000 n=20)
geomean                           117.8        66.76        -43.31%

                        │ old                │              new                                │
                        │ compression_ratio/op │ compression_ratio/op  vs base                 │
Encoder/encoder=gzip-14             5.847 ± 0%             5.847 ± 0%       ~ (p=1.000 n=20) ¹
Encoder/encoder=zstd-14             9.253 ± 0%             9.253 ± 0%       ~ (p=1.000 n=20) ¹
geomean                             7.355                  7.355       +0.00%
¹ all samples are equal

                        │ old                │              new                                │
                        │ size_compressed/op │ size_compressed/op  vs base                     │
Encoder/encoder=gzip-14          4.471M ± 0%          4.471M ± 0%       ~ (p=1.000 n=20) ¹
Encoder/encoder=zstd-14          2.825M ± 0%          2.825M ± 0%       ~ (p=1.000 n=20) ¹
geomean                          3.554M               3.554M       +0.00%
¹ all samples are equal

_Allocs and _Parallel benchmarks create the encoder fresh each iteration (simulating per-flush behavior in adapters). Encoder/encoder=* is the existing benchmark that creates the encoder once and reuses the returned ContentEncoder closure.

Compression ratio and output size are identical across old/new (all samples equal).

Comment thread axiom/encoder.go
@lukasmalkmus lukasmalkmus merged commit d3ce148 into main Mar 2, 2026
25 checks passed
@lukasmalkmus lukasmalkmus deleted the lukasmalkmus/ywpnpuvpvqsy branch March 2, 2026 20:48
lukasmalkmus added a commit that referenced this pull request Mar 2, 2026
Add ZstdEncoderWithLevel to allow configuring zstd compression levels,
mirroring the existing GzipEncoderWithLevel API. Uses per-level pools
since zstd.WithEncoderLevel cannot be changed on Reset.

Follows up on PR #413 review feedback from @tsenart.
lukasmalkmus added a commit that referenced this pull request Mar 2, 2026
Add ZstdEncoderWithLevel to allow configuring zstd compression levels,
mirroring the existing GzipEncoderWithLevel API. Uses per-level pools
since zstd.WithEncoderLevel cannot be changed on Reset.

Follows up on PR #413 review feedback from @tsenart.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants