[QST] CPU compression for decompression with high-level interface

[technillogue]

How can I generate the metadata needed for decompression?

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[technillogue]

Any update on this? I have a bunch of old data that needs to be compressed and it would be unfortunate to spin up GPUs only for compression

[eschmidt-nvidia]

Hi @technillogue, we're looking at producing a binary that would do this for you (i.e. produce an HLIF buffer using the CPU). Which formats are you interested in?

If you have a significant amount of old data, why not do this on GPU?

[technillogue]

Mostly Gdeflate, though Zstd would be good too. Is the HLIF format proprietary? I have some terabytes of data that sometimes needs to be loaded very quickly. Compression is cost-sensitive, not time-sensitive.

…

On Tue, Aug 22, 2023, 8:53 PM eschmidt-nvidia ***@***.***> wrote: Hi @technillogue <https://github.com/technillogue>, we're looking at producing a binary that would do this for you (i.e. produce an HLIF buffer using the CPU). Which formats are you interested in? If you have a significant amount of old data, why not do this on GPU? — Reply to this email directly, view it on GitHub <#86 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHG4G2LAEAW5XTBPXQJNNDXWT52XANCNFSM6AAAAAAZMHPKC4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[eschmidt-nvidia]

I see. If I understand correctly, based on my experiments GPU and CPU will provide similar throughput/$. I've tested this on H100 compared to Genoa / SPR CPUs.

Given that you're not time sensitive, have you investigated the GDeflate high-compression mode? This could provide cost savings if you're storing the data for a long time.

We're looking at adding a similar mode to ZSTD.

The format isn't proprietary but we haven't had time to produce a public document that fully describes it.

[technillogue]

Interesting, I'm not sure what prices you have access to that make that work out but I can give it a shot and do compression a little more efficiently

As of June, GDeflate high-compression mode was broken for HLIF #81 (comment). I'm mostly interested in compressing model finetunes, not datasets, so almost all gains are from entropy coding and not dictionary compression. When trying the LLIF benchmark entropy-only had about the same compression ratio as high compression. I imagine there's ways to tune specifically entropy coding to be higher compression though.

[eschmidt-nvidia]

I responded to the earlier issue. This should be fixed.

Interesting regarding entropy-only being better. Have you tried our ANS, bitcomp, or Cascaded formats?

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.