Maximum data size for compression

[DrivenByNostalgia]

I have integrated nvCOMP (version nvcomp_3.0.5_windows_12.x) into our system for the compression of scientific data (GIS, simulation, …) and have noticed that compression fails for data of 2^32 bytes or more, while the compression of a buffer of size 2^32 - 1 works flawlessly.

Specifically, nvcompManagerBase::compress throws the exception "Encountered Cuda Error: 2: 'out of memory'." after calling nvcompManagerBase::configure_compression with a decomp_buffer_size >= 2^32.

If the chosen compression manager is a DeflateManager or GdeflateManager, the call of compress first outputs a warning to std::cerr:

"WARNING: In nvcompBatchedGdeflateCompressGetTempSize: Might not have enough memory available on this GPU.
Require 22.002 GB but detected only 0 GB available"

Both of the reported numbers in this case seem wrong to me, but this is not my main issue, as we are not planning to use Deflate/Gdeflate anyways.

Is this an undocumented hard limit for the file size that nvCOMP can handle? I don't mind if there is a limit of 4 GB to the buffer size even with much more available memory. However, in this case, it would be nice to know and query this limit beforehand so to not run into an exception.

This has been tested on a system with:

0 - Device name: NVIDIA GeForce RTX 3060
0 - CUDA Driver Version / Runtime Version: 12.3 / 12.3
0 - CUDA Capability Major/Minor version number: 8.6
0 - Total amount of global memory: 12.00 GBytes

[akshaysubr]

@DrivenByNostalgia Thanks for reporting this. You're right to expect that a buffer sizes >= 2^32 should work or a limitation should be documented. Can you share a minimal repro of this issue?

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[DrivenByNostalgia]

No, I'm sorry, it's not within my time to create a reproducer. But as this is an issue that can easily be circumvented in the calling code, it's not a big issue after all.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.