Flat buffers

[boocmp]

This pull request marks the first stage of optimizing the ad blocker’s memory usage. It introduces the ability to store network request blocking rules in FlatBuffers. At this stage, memory consumption is reduced by half (a 2x improvement) with the same lookup performance. Future enhancements will focus on optimizations like eliminating HashMaps, aiming to achieve a 3x overall improvement in memory efficiency.

1. The test code has been moved to the test/unit/ folder, replicating the src directory structure. In my opinion, this improves the efficiency of working with the codebase.

• unit tests from network.rs moved to tests/unit/filters/network.rs
• unit tests from cosmetic.rs moved to tests/unit/filters/cosmetic.rs
• Removed unused files from url_parser.
• unit tests from resource_storage.rs moved to tests/unit/filters/resou…
• unit tests from resource_assembler.rs moved to tests/unit/filters/res…
• unit tests from regex_manager.rs moved to tests/unit/regex_manager.rs
• unit tests from content_blocking.rs moved to tests/unit/content_block
• unit tests from cosmetic_filter_cache.rs moved to tests/unit/cosmetic
• unit tests from engine.rs moved to tests/unit/engine.rs
• unit tests from lists.rs moved to tests/unit/lists.rs
• unit tests from request.rs moved to tests/unit/request.rs
• unit tests from optimizer.rs moved to tests/unit/optimizer.rs
• unit tests from utils.rs moved to tests/unit/utils.rs

2. Rust formatter. The formatter has been applied to all files to ensure consistent formatting throughout the codebase.

• Rust formatter.

3. The filter parser code has been moved to a separate file, abstract_network.rs. This reduces the amount of code in the network.rs file.

• AbstractNetworkFilter moved to abstract_network.rs.

4. A separate file has been created for RegexManager. This is preparation for breaking the direct dependency of RegexManager on NetworkFilter.

• AbstractNetworkFilter moved to abstract_network.rs.
• Key type of compiled regexes map changed to u64.

5. The functions for finding a filter based on a network request have been moved to a separate file, network_matchers.rs. Also in this commit, loops in the macthing functions have also been replaced with any and all. Additionally, a trait AnyOrExt was introduced, adding an any_or method to iterators. However, this turned out to be an good solution, and in subsequent commits, it was removed in favor of using early returns.

• Filter matching functions have beed moved to network_matchers.rs. Rem…

6. Intermediate fix for project compilation.

• Unit tests compilation and Rust-fmt. Matchers tests moved from networ…

7. First version of the FlatBuffers storage. Additionally, the AnyOrExt trait has been removed, and early return has been used instead.

• Added flatbuffer network filters implementation.

8. A url_lower_cased field has been added to the struct Request. This reduces the number of memory allocations during filter lookup and speeds up the matching. A struct Tokens has been introduced, and tokens are now calculated once and stored as a field in the struct Request. This simplifies and speeds up the code. A method checkable_tokens_iter has been added to Request that returns the sequence of tokens that need to be checked. The object-pooling feature has been removed.

• Added url_lower_cased member in request to prevent the memory allocat…
• Removed "object-pooling". request_tokens is a part of Request now.
• Added checkable_tokens_iter
• Tests compilation.
• Tests compilation.

9. The struct NetworkFilterList has been moved from blocker.rs to a separate file, network_filter_list.rs. Corresponding tests have been relocated to tests/unit/network_filter_list.rs

• NetworkFilterList moved to network_filter_list.rs

10. Memory usage in FlatBuffers has been optimized. include and exclude domains now stored as u16 indices instead of u64 hashes in a separate sorted array unique_domains_hashes

• Added flatbuffer structure for network filters.
• Fixed fb_network.rs to match flat schema.

11. Preparation for integrating FlatNetworkFilter into Blocker. Added a NetworkFilterMaskHelper trait, into which the boolean getters have been moved.

• Added NetworkFilterListTrait to provide flat impl later.
• Added NetworkFilterMaskHelper trait to provide bool getters for diffe…

12. Removed copy-paste loops from check, check_all, and filter_exists. Now, there is a single loop iterates over sequence from Request::checkable_tokens_iter().

• Simplified NetworkFilterList impl.

13. Blocker has been renamed to GenericBlocker<NetworkFilterListType>, which may depends on the type of list: NetworkFilterList or FlatNetworkFilterList. Added an implementation for FlatNetworkFilterList.

• Added list type generic parameter to the Blocker.
• Added FlatNetworkFilterList. Added FlatNetworkFilter. Implemented mat…

14. Fixed FlatNetworkFilterList to ensure that the results match in terms of speed and filter lookups with the implementation based on NetworkFilterList.

• Temporary using NetworkFilterList for filters optimization.
• Fixed filter's unique key for regex manager.

15. Improving the speed of filter construction for FlatNetworkFilterList. Rules are optimized during creation. Now FlatNetworkFilterList doesn't depend on NetworkFilterList.

• Optimization for FlatNetworkFilterList.

16. Adding the FlatBuffers feature, fixing tests, and modifying CI steps

• Wrong conflicts due the rebase. Fixed.
• Rust fmt after rebase.
• Enable flatbuffers feature in perf CI
• Added Serialize trait for engine.
• Tests fixing and disabling.
• Added build & test steps in GHWF.
• Disabled serialization tests for flatbuffers feature.
• Changed 'Run Brave-specific tests' CI step.

[github-actions]

Rust Benchmark

Benchmark suite	Current: a2a8f5a	Previous: 7919bdd	Ratio
rule-match-browserlike/brave-list	1917794686 ns/iter (± 20902834)	1745226241 ns/iter (± 10688991)	1.10
rule-match-first-request/brave-list	1052414 ns/iter (± 13996)	1003256 ns/iter (± 7610)	1.05
blocker_new/brave-list	159015429 ns/iter (± 3655404)	210108247 ns/iter (± 7007989)	0.76
memory-usage/brave-list-initial	21457739 ns/iter (± 3)	41409969 ns/iter (± 3)	0.52
memory-usage/brave-list-after-1000-requests	24064706 ns/iter (± 3)	44005995 ns/iter (± 3)	0.55

This comment was automatically generated by workflow using github-action-benchmark.

[ambyjkl]

Would've liked to see capnproto instead of flatbuffers, but still this is good, great work 👍

[boocmp]

Would've liked to see capnproto instead of flatbuffers, but still this is good, great work 👍

I want to try it after the flatbuffers.