component detection support regex to match files.

[wangzelin007]

Hi component detection team,
Our public repo: azure-cli is using component detection in Azure DevOps pipelines.
in our repo, we have multiple reuqirment.$PYVERSION.$PLATFORM.txt file.
For example:
https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.Darwin.txt
https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.Linux.txt
https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/requirements.py3.windows.txt

But component detection task only match requirements.txt file
component-detection.log

Can you support regex while checking files.
Currently:
https://github.com/microsoft/component-detection/blob/main/src/Microsoft.ComponentDetection.Detectors/pip/PipComponentDetector.cs#L18

Suggetsion:
public override IList SearchPatterns => new List { "setup*.py", "requirements*.txt" };

Thanks.

[JamieMagee]

@wangzelin007 Can I ask the technical reason why you need separate files for this? I did a quick diff, and the differences are minor:

• requirements.py3.windows.txt is missing wrapt
• requirements.py3.windows.txt has pywin32
• requirements.py3.windows.txt has an older version of applicationinsights (0.11.7 vs 0.11.9)
• requirements.py3.Linux.txt has distro

wrapt appears to be a cross-platform dependency and has no coupling to a specific operating system: https://pypi.org/project/wrapt/

It doesn't appear that pywin32 is used anywhere in the codebase: https://github.com/Azure/azure-cli/search?q=pywin32

Imports of distro are guarded: https://github.com/Azure/azure-cli/blob/103c4e9636657fa307436c36ade1314831d05d62/src/azure-cli/azure/cli/command_modules/feedback/custom.py#L374-L378

I assume the difference in applicationinsights versions is a mistake?

[jiasli]

First of all, we use equirements.py3.*.txt files to guarantee built packages like deb, rpm are stable. That's why indirect dependencies are also included.

• requirements.py3.windows.txt is missing wrapt

wrapt is an indirect dependency:

> pipdeptree --reverse --packages wrapt
wrapt==1.13.3
  - astroid==2.8.6 [requires: wrapt>=1.11,<1.14]
    - pylint==2.11.1 [requires: astroid>=2.8.0,<2.9]
      - azdev==0.1.36 [requires: pylint==2.11.1]
  - Deprecated==1.2.13 [requires: wrapt>=1.10,<2]
    - PyGithub==1.55 [requires: deprecated]
      - azure-cli==2.32.0 [requires: PyGithub~=1.38]
  - vcrpy==4.1.1 [requires: wrapt]
    - azure-cli-testsdk==0.3.0 [requires: vcrpy>=1.10.3]
    - azure-devtools==1.2.0 [requires: vcrpy>=1.11.0]

It indeed should be included in requirements.py3.windows.txt or removed from other requirements.py3.*.txts.

• requirements.py3.windows.txt has an older version of applicationinsights (0.11.7 vs 0.11.9)

I think this should be an error.

• requirements.py3.windows.txt has pywin32

pywin32 is also an indirect dependency:

> pipdeptree --reverse --packages pywin32
pywin32==302
  - portalocker==2.3.2 [requires: pywin32>=226]
    - azure-cli-telemetry==1.0.6 [requires: portalocker>=1.6,<3]
      - azure-cli-core==2.32.0 [requires: azure-cli-telemetry==1.0.6.*]
        - azure-cli==2.32.0 [requires: azure-cli-core==2.32.0]
    - msal-extensions==0.3.1 [requires: portalocker>=1.6,<3]
      - azure-cli-core==2.32.0 [requires: msal-extensions>=0.3.1,<0.4]
        - azure-cli==2.32.0 [requires: azure-cli-core==2.32.0]
      - azure-identity==1.6.1 [requires: msal-extensions~=0.3.0]
        - azure-cli==2.32.0 [requires: azure-identity]

portalocker does install pywin32 only on Windows:

https://github.com/WoLpH/portalocker/blob/00cffcb1831208691d3bf6850df4768989a0bd4c/setup.py#L131-L134

        install_requires=[
            # Due to CVE-2021-32559 updating the pywin32 requirement
            'pywin32>=226; platform_system == "Windows"',
        ],

• requirements.py3.Linux.txt has distro

This is true because we use distro to detect the Linux distribution. On Windows, we don't need it.