software_center_download: Investigate interrupted download resulting in incomplete files

[marcelmamula]

Issue

Current implementation of chunked download is not retrying download properly

community.sap_launchpad/plugins/module_utils/sap_launchpad_software_center_download_runner.py

Line 231 in 9378a71

def _download_file(url, filepath, retry=0, **kwargs):

I was downloading lot of files this week and I noticed output like this

changed: [s05hana] => (item=SAPEXE_51-70007807.SAR : Successfully downloaded SAP software: SAPEXE_51-70007807.SAR)
FAILED - RETRYING: [s05hana]: Download - SAP Software Files - Get files with Python venv (1 retries left).
changed: [s05hana] => (item=SAPEXEDB_51-70007806.SAR : Successfully downloaded SAP software: SAPEXEDB_51-70007806.SAR)

This example was not harmful because file was too small, but when it happened with BW4HANA300_INST_EXPORT_4.zip, it resulted in failed SWPM DB Load because file was invalid. Retry action was not able to remediate issue and file size was smaller than should have been.

[sean-freeman]

Using the BW/4HANA example, as the larger file will be more likely to have a failure that causes a corruption error. Implementing a verification against the checksum is probably the way to ensure integrity of the download file.....

URL: https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')

{
    "d": {
        "__metadata": {
            "id": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')",
            "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')",
            "type": "SVT_SWDC_UI_SRV.Object"
        },
        "ObjectKey": "0030000001509622021",
        "Title": "BW4HANA300_INST_EXPORT_4.zip",
        "Status": "AVAILABLE",
        "StatusDescr": "The File is available to download",
        "FileType": "zip",
        "FileSize": "2304379 ",
        "FileName": "BW4HANA300_INST_EXPORT_4.zip",
        "FileVersion": "013",
        "Checksum": "d16a144feed13068fe70e94fb73419688b21ab85e10ceef461fbac5499a7326b",
        "PatchType": "",
        "EpsFileName": "",
        "PackageLevel": "",
        "ComponentRelease": "",
        "MinimalBasisRelease": "",
        "RequiredSpamVersion": "",
        "InfoXml": "",
        "InfoText": "",
        "InfoMimeType": "",
        "Responsible": "",
        "ResponsibleName": "",
        "CreatedBy": "",
        "CreatedByName": "",
        "CreatedOn": "\/Date(1634823334000)\/",
        "ChangedBy": "",
        "ChangedByName": "",
        "ChangedOn": "\/Date(1743957907000)\/",
        "IsAbapObject": false,
        "IsSupportPackage": false,
        "IsFcmsViewAvailable": false,
        "IsContentInfoAvailable": false,
        "ObjectSAPNotesPriorityFilter": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectSAPNotesPriorityFilter"
            }
        },
        "ObjectSAPNotesCountryFilter": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectSAPNotesCountryFilter"
            }
        },
        "ObjectAttributes": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectAttributes"
            }
        },
        "ObjectSAPNotesComponentFilter": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectSAPNotesComponentFilter"
            }
        },
        "ObjectList": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectList"
            }
        },
        "ObjectConditions": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectConditions"
            }
        },
        "ObjectProperties": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectProperties"
            }
        },
        "ObjectSAPNotesCategoryFilter": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectSAPNotesCategoryFilter"
            }
        },
        "ObjectSAPNotes": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectSAPNotes"
            }
        },
        "ObjectSideEffects": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectSideEffects"
            }
        },
        "ObjectSourcePackages": {
            "__deferred": {
                "uri": "https://launchpad.support.sap.com/services/odata/svt/swdcuisrv/ObjectSet('0030000001509622021')/ObjectSourcePackages"
            }
        }
    }
}

[marcelmamula]

I had same issue with S4 file which got interrupted and then SWPM installation failed. So I think this is good idea to think about checksum validation.
S4CORE108_INST_EXPORT_12.zip

[sean-freeman]

The checksum check was something that was ommitted during the open-source of the original code from IBM internal.

If there is again instability in the downloads (rare, but in my experience this occurs a week in every 10+ months, or around the time of a new S/4 release) - then it is worth adding a checksum check after the file has downloaded.

The checksum returned from the API is SHA256 (i.e. check with binary sha256sum or integrated hashlib Python Module), on larger files it will slow down the download so probably should enable checksum check by default but allow it to be disabled?

---

However.... the hashlib might be Linux specific and not run on macOS, in a similar way to:

• Execution of Ansible Collection linux-system-roles Ansible Role ha_cluster in setting hacluster password, with warning message "crypt.crypt not supported on Mac OS X/Darwin, install passlib python module."
• Execution of any Ansible Playbook from a Windows host, as Ansible Core for I/O requires fcntl Python Module and does not have switch-case logic that instead imports portalocker/waitress Python Module when the Python environment detects running on a Windows host. See:
  • https://docs.ansible.com/ansible/latest/os_guide/intro_windows.html#using-windows-as-the-control-node
  • Please switch from fcntl to portalocker ansible/ansible#73641 (comment)
  • fcntl has 13 usages in Ansible Core > https://github.com/search?q=repo%3Aansible%2Fansible%20fcntl&type=code