License incorrectly identified as any-OSI

[AjaiKN]

When I run spdx-license-matcher on the following file, it says that the license is any-OSI, which is clearly incorrect:

This data made available by Google, Inc. under the Creative Commons Attribution 4.0 International license.
https://creativecommons.org/licenses/by/4.0/

The problem seems to be the normalization step:

from spdx_license_matcher.normalize import normalize

text = '''This data made available by Google, Inc. under the Creative Commons Attribution 4.0 International license.
https://creativecommons.org/licenses/by/4.0/
'''

normalize(text) # => 'normalized/url'

license = "Pick your favourite OSI approved license :)\n\nhttp://www.opensource.org/licenses/alphabetical\n"

normalize(license) # => 'normalized/url'

In both cases, the URL is replaced with normalized/url, then the first line is deleted because it contains the word "license":

spdx-license-matcher/spdx_license_matcher/normalize.py

Lines 86 to 88 in 3e19ab5

	# To remove the license name or title present at the beginning of the license text.
	if 'license' in licenseText.split('\n')[0]:
	licenseText = '\n'.join(licenseText.split('\n')[1:])

Because the normalized versions are exactly the same, the score is 1.0, so spdx-license-matcher skips the more thorough matching (which would be done by checkTextStandardLicense) and says there's an exact match:

spdx-license-matcher/spdx_license_matcher/computation.py

Lines 52 to 64 in 3e19ab5

	elif all(score == 1.0 for score in list(matches.values())):
	matchingString = 'The following license ID(s) match: ' + ", ".join(list(matches.keys()))
	return matchingString
	else:
	for licenseID in matches:
	listedLicense = getListedLicense(licenseID)
	isTextStandard = checkTextStandardLicense(listedLicense, inputText)
	if not isTextStandard:
	matchingString = 'The following license ID(s) match: ' + licenseID
	return matchingString
	else:
	return ''

Thanks a lot, by the way! I've been finding the SPDX tools very useful.

[AjaiKN]

Whoops, didn't mean to close this.

[goneall]

Thanks @AjaiKN for reporting this and the analysis.

Wondering if we should just remove the normalization that remove the license / title. Do you think this would resolve the issue?