Default wildcard rule missing from the PSL algorithm implementation

[braedon]

The PSL formal algorithm includes the following step:

• If no rules match, the prevailing rule is "*".

This rule means that if an explicit public suffix can't be found for a domain in the PSL, the (actual, not effective) TLD is treated as a public suffix. This allows new TLDs (or custom internal TLDs) to be handled without requiring an explicit update to the PSL.

(It also means that the only time the PSL algorithm doesn't return a registrable domain is when the domain is itself a public suffix, which helps disambiguate that case.)

The PSL project has tests for this in the standard test suite:

// Unlisted TLD.
checkPublicSuffix('example', null);
checkPublicSuffix('example.example', 'example.example');
checkPublicSuffix('b.example.example', 'example.example');
checkPublicSuffix('a.b.example.example', 'example.example');

Here's the output for the test domains in tldextract 5.1.2:

>>> tldextract.extract('example').registered_domain                                                                                                                                                                                                                          
''
>>> tldextract.extract('example.example').registered_domain
''
>>> tldextract.extract('b.example.example').registered_domain
''
>>> tldextract.extract('a.b.example.example').registered_domain
''

The last three results are turning '', when they should return example.example.

[elliotwutingfeng]

Good catch. There are bunch of issues related to this at the public suffix list repository, such as publicsuffix/list#694

Also relevant is https://wiki.mozilla.org/Public_Suffix_List/platform.sh_Problem#Further_Information (under the Further Information heading) which comments on the implications of enforcing/not enforcing the "*" rule.

@john-kurkowski

[john-kurkowski]

The last three results are turning '', when they should return example.example.

And the ExtractResult objects for all four results should have suffix='example', instead of domain='example'?

I'm worried about adopting this change. It would be breaking for anybody using this library to distinguish recognized suffixes from bogus ones or from internal hostnames like localhost.

Maybe a new property that honors the PSL formal algorithm? And/or a warning in the documentation?

[braedon]

And the ExtractResult objects for all four results should have suffix='example', instead of domain='example'?

Yip, I'd expect domain='example' and suffix='example', rather than the current subdomain='example' and domain='example'.

I'm worried about adopting this change. It would be breaking for anybody using this library to distinguish recognized suffixes from bogus ones or from internal hostnames like localhost.

Maybe a new property that honors the PSL formal algorithm? And/or a warning in the documentation?

Agree it's simultaneously a bug fix, and likely a breaking change for some users.

If you're not comfortable with changing the default behavior, a flag to enable the "correct" behavior would be workable for us.

You could add a new ExtractResult property that indicates whether an explicit public suffix rule was used vs. the default rule? Or alternatively, a property showing which rule was used? That could help with the "distinguish recognized suffixes from bogus ones" use case.