URLs with many distinct segments fail to match

[antonok-edm]

adblock-rust performs a rough matching step against requests by splitting the URL into tokens, hashing them, and then comparing against hashes from tokens in parsed filters.

There is an upper limit to the number of tokens that can be generated from a URL, after which point any additional tokens are ignored. This was likely put in place to limit the performance impact from maliciously crafted URLs.

As it turns out, this limit is insufficient for a small number of real-world requests.

See ghostery/adblocker#2125 (comment) for relevant test cases.

[remusao]

This was likely put in place to limit the performance impact from maliciously crafted URLs.

Just to add to this, I think there were very long URLs seen during automated crawls (i.e. many tokens) which would slow down matching and based on empirical data at the time, there were no matches after 100-200 tokens so it was useful to keep performance reasonable.