Python: Dependencies are looked up from incorrect urls

[jenshnielsen]

When running component-detection on one of my projects I am seeing output like the following.

[INFO] Getting Python data from https://pypi.org/pypi/pywinpty>=1.1.0/json
[WARN] Received 404 Not Found from https://pypi.org/pypi/pywinpty>=1.1.0/json
[WARN] Dependency Package pywinpty>=1.1.0 not found in Pypi. Skipping package

It seems like component-detection is in some situations passing an incorrect url which contains the version specifier and obviously does not resolve. This seems to happen for transitive dependencies that are constrained to a specific version by some other direct or transitive dependency.

[melotic]

This is handled by this regex:

component-detection/src/Microsoft.ComponentDetection.Detectors/pip/PipDependencySpecification.cs

Lines 32 to 35 in 17a1fb2

	// Extracts abcd from a string like abcd==1.*,!=1.3
	private static readonly Regex PipNameExtractionRegex = new Regex(
	@"^.+?((?=<)|(?=>)|(?=>=)|(?=<=)|(?===)|(?=!=)|(?=~=)|(?====))",
	RegexOptions.Compiled);

Testing this regex on pywinpty>=1.1.0 works, but requirements with optional dependencies like PyJWT[crypto]<3,>=1.0.0 will match as PyJWT[crypto] which is incorrect. We probably need to add

|(?=\[)

to this regex and create a separate issue that tracks us not fetching the transitive dependencies of the optional depenedency.

[melotic]

Additionally, there are some Require-Dists in this format:

Requires-Dist: numpy<1.27.0,>=1.19.5

that do not match this Regex:

component-detection/src/Microsoft.ComponentDetection.Detectors/pip/PipDependencySpecification.cs

Lines 13 to 16 in 17a1fb2

	// Extracts name and version from a Requires-Dist string that is found in a metadata file
	public static readonly Regex RequiresDistRegex = new Regex(
	@"Requires-Dist:\s*(?:(.*?)\s*\((.*?)\)|([^\s;]*))",
	RegexOptions.Compiled);

[cobya]

This is also likely a combination of custom packaging index feeds - #1129 should have support for this now.