Slow compress/uncompress performance compared to C and Cgo version lz4

[xin3liang]

Slow performance compared to C and Cgo version lz4:

The File pg1661.txt compress/uncompress Benchmarks:
https://github.com/xin3liang/golz4/blob/master-test/lz4_test.go#L705-L707
https://github.com/xin3liang/lz4/blob/v4-test/bench_test.go#L59-L61
https://github.com/inikep/lzbench master

[root@server5 lzbench]# ./lzbench -elz4 ../pg1661.txt
[root@server5 golz4]# go test  -bench="Long" -run=^$
[root@server5 lz4]# go test  -bench="Long" -run=^$

[pierrec]

Thank you for filing this. I am aware of it but when I looked at it a while back, I could not see how to improve it. A contributor implemented asm versions for uncompress which dramatically improved the decoding, but I would rather have a more performant version without it.
Note that Go will always be slower than C, which may or may not be acceptable for your use case.
Any idea on improving the performance is welcomed though!

[xin3liang]

Thank you for filing this. I am aware of it but when I looked at it a while back, I could not see how to improve it. A contributor implemented asm versions for uncompress which dramatically improved the decoding, but I would rather have a more performant version without it. Note that Go will always be slower than C, which may or may not be acceptable for your use case. Any idea on improving the performance is welcomed though!

@pierrec Thank you for the reply. Although I have no idea how to improve, just raise a bug here to record this issue.

FYI,
Look at the lz4hc test results below, run on the same x86 and arm64 machines

Arm64 CPU: Kunpeng 920-4826  96 cores @ 2.6GHz 4 numa
[root@server5 lzbench]# ./lzbench -elz4hc  ../pg1661.txt
lzbench 2.1 | GCC 10.3.1 | 64-bit Linux |

Compressor name         Compress. Decompress. Compr. size  Ratio Filename
lz4hc 1.10.0 -1           125 MB/s   790 MB/s      314466  52.86 ../pg1661.txt
lz4hc 1.10.0 -2           125 MB/s   790 MB/s      314466  52.86 ../pg1661.txt
lz4hc 1.10.0 -3          47.0 MB/s   800 MB/s      280384  47.13 ../pg1661.txt
lz4hc 1.10.0 -4          35.3 MB/s   795 MB/s      270572  45.48 ../pg1661.txt
lz4hc 1.10.0 -5          27.5 MB/s   793 MB/s      264987  44.54 ../pg1661.txt
lz4hc 1.10.0 -6          22.2 MB/s   792 MB/s      262404  44.11 ../pg1661.txt
lz4hc 1.10.0 -7          18.9 MB/s   796 MB/s      261329  43.93 ../pg1661.txt
lz4hc 1.10.0 -8          16.8 MB/s   795 MB/s      260900  43.85 ../pg1661.txt
...

x86 CPU: Intel(R) Xeon(R) Platinum 8180 111 cores @ 2.50GHz 2 numa
[root@client5 lzbench]# ./lzbench -elz4hc ../pg1661.txt
lzbench 2.1 | GCC 10.3.1 | 64-bit Linux | Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz

Compressor name         Compress. Decompress. Compr. size  Ratio Filename
lz4hc 1.10.0 -1           106 MB/s   629 MB/s      314466  52.86 ../pg1661.txt
lz4hc 1.10.0 -2           106 MB/s   629 MB/s      314466  52.86 ../pg1661.txt
lz4hc 1.10.0 -3          35.6 MB/s   639 MB/s      280384  47.13 ../pg1661.txt
lz4hc 1.10.0 -4          27.2 MB/s   636 MB/s      270572  45.48 ../pg1661.txt
lz4hc 1.10.0 -5          21.4 MB/s   635 MB/s      264987  44.54 ../pg1661.txt
lz4hc 1.10.0 -6          17.7 MB/s   634 MB/s      262404  44.11 ../pg1661.txt
lz4hc 1.10.0 -7          15.1 MB/s   634 MB/s      261329  43.93 ../pg1661.txt
...

I notice that the compress/decompress speed may be related to the compression ratio. The smaller compression ratio, the slower compress/decompress speed. According to this rule, the compress/decompress speed of pure Go lz4(ratio 55.33) should be better than the first line compress/decompress speed(ratio is 52.86) of lz4hc.
Maybe the worst part is that the pure Go lz4 arm64's decompress speed(321.25 MB/s), which is much slower than lz4hc(790 MB/s, ratio 52.86).