klauspost compression

[jhawk28]

klauspost compress has better performance than the standard library. add zstd support

[AlexHentschel]

I appreciate your contribution. It looks like you provided some benchmarks. What are the performance improvements? Reason why I am asking is because as part of our security and resilience strategy, every library dependency needs to be initially vetted to confirm it is trustworthy and monitored on an ongoing basis for the risk of supply chain attacks.

The libraries you propose to replace are widely-used and subject to rigorous security monitoring by the libraries source. This is generally not the case for smaller-scale specialized libraries. I am not against adopting klauspost compression. Though, trustworthiness at the time of introduction as well as estimated effort to continuously monitor the library needs to be part of the commit process. Hence, we are hoping you might have some information since you are using klauspost compress already (?)

[jhawk28]

Here are the benchmark results (running locally)

Klauspost:
BenchmarkWithGzipCompression
BenchmarkWithGzipCompression-14 6417 159204 ns/op
BenchmarkWithSnappyCompression
BenchmarkWithSnappyCompression-14 8049 134486 ns/op
BenchmarkWithDeflateCompression
BenchmarkWithDeflateCompression-14 10000 114062 ns/op
BenchmarkWithZSTDCompression
BenchmarkWithZSTDCompression-14 9820 118565 ns/op

Std library:
BenchmarkWithGzipCompression
BenchmarkWithGzipCompression-14 7490 155665 ns/op
BenchmarkWithSnappyCompression
BenchmarkWithSnappyCompression-14 8186 130101 ns/op
BenchmarkWithDeflateCompression
BenchmarkWithDeflateCompression-14 7405 158477 ns/op

[jhawk28]

We have been using the klauspost library at QN for a few years now and have found it to perform well. The updates have always been good and haven't broken. It is used to implement our gzip and zstd on our edges (hosting your public endpoints).