Hey,
we've run into a strange issue on some CPUs when using lz4 blockwise decoding from a custom wire format. We did some investigation already, so let me give some context.
Problem description
We noticed the issue when upgrading to Go 1.24 in a test harness with LZ4 HC. The test harness unfortunately isn't public, but it's trivial, so I'll explain it: It uses a C++ encoder to compress some random data of various sizes with our wire format, and then decompresses them with the equivalent Go implementation (and the other way around).
The thing that started failing after 1.24 is decoding of more than 2KiB of data encoded by the C++ encoder, specifically it fails like this:
tazjin@khamovnik /a/y/g/c/lz4_erms_reproducer> ./reproducer tarpit.yt.lz4
Read 323479 bytes from input file
DECOMPRESSION ERROR: lz4: invalid source or destination buffer too short
We bisected this in Go to this commit: golang/go@601ea46
The commit adds vectorised runtime.memmove (through Fast Short REP MOV) for some CPUs, which also matches our experience (the issue occurs sporadically, for example servers with AMD EPYC CPUs are unaffected).
We noticed that your library uses Go's assembler on amd64 for block decoding: https://github.com/pierrec/lz4/blob/v4/internal/lz4block/decode_amd64.s
The problem might be occurring due to some assumption in this assembly being violated around its invocations of runtime.memmove with the new implementation.
Reproducer
This gist produces a binary which reproduces the problem: https://gist.github.com/tazjin/c83381ed2b10192365fe08dfc6b13e56
It contains a minimal implementation of the parts of the wire format, haven't managed to shorten it much yet :(
This reproducer can be called with the file tarpit.gz (attached to this issue), which should decompress to a PDF of the paper "Out of the Tarpit", but fails instead. (Note that the filename ends in .gz only because Github enforces this for attachments).
Note that this can only be reproduced on the Intl CPUs which have both of the necessary features, on all other CPUs this will always work.
Workaround
Setting GODEBUG=cpu.fsrm=off disables the responsible optimisation (and currently nothing else), so it is a sufficient workaround right now.
Hey,
we've run into a strange issue on some CPUs when using lz4 blockwise decoding from a custom wire format. We did some investigation already, so let me give some context.
Problem description
We noticed the issue when upgrading to Go 1.24 in a test harness with LZ4 HC. The test harness unfortunately isn't public, but it's trivial, so I'll explain it: It uses a C++ encoder to compress some random data of various sizes with our wire format, and then decompresses them with the equivalent Go implementation (and the other way around).
The thing that started failing after 1.24 is decoding of more than 2KiB of data encoded by the C++ encoder, specifically it fails like this:
We bisected this in Go to this commit: golang/go@601ea46
The commit adds vectorised
runtime.memmove(through Fast Short REP MOV) for some CPUs, which also matches our experience (the issue occurs sporadically, for example servers with AMD EPYC CPUs are unaffected).We noticed that your library uses Go's assembler on
amd64for block decoding: https://github.com/pierrec/lz4/blob/v4/internal/lz4block/decode_amd64.sThe problem might be occurring due to some assumption in this assembly being violated around its invocations of
runtime.memmovewith the new implementation.Reproducer
This gist produces a binary which reproduces the problem: https://gist.github.com/tazjin/c83381ed2b10192365fe08dfc6b13e56
It contains a minimal implementation of the parts of the wire format, haven't managed to shorten it much yet :(
This reproducer can be called with the file tarpit.gz (attached to this issue), which should decompress to a PDF of the paper "Out of the Tarpit", but fails instead. (Note that the filename ends in
.gzonly because Github enforces this for attachments).Note that this can only be reproduced on the Intl CPUs which have both of the necessary features, on all other CPUs this will always work.
Workaround
Setting
GODEBUG=cpu.fsrm=offdisables the responsible optimisation (and currently nothing else), so it is a sufficient workaround right now.