Optimize SBOM Fetching and Processing to Reduce Memory Usage #75
Description
Problem
The current SBOM (Software Bill of Materials) fetching and processing in our tool consumes excessive memory:
- The FetchSBOMs method preloads all SBOMs into memory, which would becomes inefficient with large SBOMs.
- The sbomProcessing function holds both original and converted SBOMs in memory at the same time, doubling memory usage.
- The upload step keeps all SBOMs in memory until the entire process finishes.
- If something went wrong while fetching, it throws an error, and restart fetching from very beginning.
This inefficiency could lead to cause performance degradation or crashes when processing large numbers of repositories or SBOMs due to memory exhaustion.
For example: it fetches 100 sbom at a time, then process all at once, and then upload it all at once.
Suggested Solution
Refactoring the pipeline to handle SBOMs one at a time using lazy loading and on-demand processing. Although it is kind of implemented while uploading an SBOMs, but still deal with accumlated SBOMs in memory.
For example:
sequential: it fetches 1 sbom(out of 100) at a time, then process one at once, and then upload one at once. And contiue to second one.
Parallel: we can run 3-4 process concurrently.
fetch --> process --> upload (1 SBOM) and then repeat for next SBOM and conitnue.
So, our motive to be, Get Metadata of the SBOM(like filename, download URL, size, etc) --> Wile processing download fetch one SBOM --> Then it downloads the SBOM, and SBOM is processed and converted --> and then converted SBOM being passed to upload SBOM.
In detail Implementation:
- Get SBOMs One by One: The tool will only pick up what it needs to find each SBOM (like where it’s located) without grabbing everything right away.
- Handle Each SBOM Separately: It will process one SBOM at a time—doing whatever needs to be done, like converting it—without holding onto the others.
- Send It Off Quickly: As soon as an SBOM is ready, the tool will upload it to where it needs to go and then move on to the next one, keeping memory free.