Flatbuffers impl #446

boocmp · 2025-03-28T08:24:44Z

No description provided.

Added raw_line field to flatbuffers for testing.

github-actions

Rust Benchmark

Benchmark suite	Current: `0238db7`	Previous: `d56be21`	Ratio
`rule-match-browserlike/brave-list`	`2008705996` ns/iter (`± 16871464`)	`1727033355` ns/iter (`± 11920489`)	`1.16`
`rule-match-first-request/brave-list`	`1006441` ns/iter (`± 8517`)	`1005963` ns/iter (`± 12764`)	`1.00`
`blocker_new/brave-list`	`159390982` ns/iter (`± 2576711`)	`220140757` ns/iter (`± 5159081`)	`0.72`
`memory-usage/brave-list-initial`	`21457739` ns/iter (`± 3`)	`41408849` ns/iter (`± 3`)	`0.52`
`memory-usage/brave-list-after-1000-requests`	`24064706` ns/iter (`± 3`)	`44004875` ns/iter (`± 3`)	`0.55`

This comment was automatically generated by workflow using github-action-benchmark.

github-actions · 2025-04-09T00:58:13Z

src/filters/fb_network.rs

+                    bytes.as_ptr() as *const u16,
+                    bytes.len() / std::mem::size_of::<u16>(),
+                )
+            }


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

It turns out that from_raw_parts results in issues if bytes.as_ptr() isn't aligned to 2 bytes We need to assert this.

github-actions · 2025-04-09T00:58:13Z

src/filters/fb_network.rs

+                    bytes.as_ptr() as *const u16,
+                    bytes.len() / std::mem::size_of::<u16>(),
+                )
+            }


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

github-actions · 2025-04-09T00:58:13Z

src/flat_network_filter_list.rs

+        }
+
+        let filters_list =
+            unsafe { fb::root_as_network_filter_list_unchecked(&self.flatbuffer_memory) };


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

@boocmp use root_as_network_filter_list() + .expect() to remove unsafe

github-actions · 2025-04-09T00:58:13Z

src/flatbuffers/fb_network_filter_generated.rs

+            unsafe {
+                self._tab
+                    .get::<flatbuffers::ForwardsUOffset<&str>>(NetworkFilter::VT_RAW_LINE, None)
+            }


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

atuchin-m

LGTM with a few nits

atuchin-m · 2025-04-09T10:12:44Z

src/filters/fb_network.rs

+        }
+    }
+
+    fn get_or_insert(&mut self, h: &Hash) -> u16 {


nit: get_or_insert_unique_domain

atuchin-m · 2025-04-09T10:13:33Z

.github/workflows/perf-ci.yml

@@ -37,6 +37,18 @@ jobs:
      - name: Bench memory usage
        run: cargo bench --bench bench_memory -- --output-format bencher | tee -a output.txt

+      - name: Bench network filter matching (flat)


let's remove this in favor of enabling flatbuffers-storage by default (or in the tests above).

in the next pr

atuchin-m · 2025-04-09T10:29:24Z

src/filters/fb_network.rs

+            None
+        };
+
+        let raw_line = network_filter


do we create this only in debug mode?

yes only debug

atuchin-m · 2025-04-09T10:31:21Z

src/filters/fb_network.rs

+            &mut self.builder,
+            &&fb::NetworkFilterListArgs {
+                network_filters: Some(filters),
+                unique_domains_hashes: Some(unique_domains),


We don't convert string to hashes here, so let's replace unique_domains_hashes to unique_domains or vice versa

atuchin-m · 2025-04-09T10:33:23Z

src/filters/fb_network.rs

+    owner: &'a FlatNetworkFilterList,
+    fb_filter: &'a fb::NetworkFilter<'a>,
+
+    pub mask: NetworkFilterMask,


it's not clear why mask is pub here, but the other fields are not.

because it is used outside, in the FlatNetworkFilterList::check

should it be pub(crate)?

it could. done

atuchin-m · 2025-04-09T10:40:09Z

src/filters/fb_network.rs

+    }
+
+    #[inline(always)]
+    pub fn iter(&self) -> FlatPatternsIterator {


It's not absolutely clear why we implement iterator traits instead using the structure as-is.
If we do it to use a one check_pattern() for both "old" and flat impl, let's make a comment to inline this after "old" impl is gone.

because I removed direct dependency from any storages in the check functions. I'm not planning to inline it

github-actions · 2025-04-10T11:28:44Z

src/filters/fb_network.rs

+                    bytes.as_ptr() as *const u16,
+                    bytes.len() / std::mem::size_of::<u16>(),
+                )
+            }


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

github-actions · 2025-04-10T11:28:44Z

src/filters/fb_network.rs

+                    bytes.as_ptr() as *const u16,
+                    bytes.len() / std::mem::size_of::<u16>(),
+                )
+            }


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

antonok-edm · 2025-04-17T18:27:44Z

src/blocker.rs

-    pub(crate) generic_hide: NetworkFilterList,
+pub struct GenericBlocker<NetworkFilterListType>
+where
+    NetworkFilterList: NetworkFilterListTrait,


this trait bound isn't doing anything useful since you're missing Type:

NetworkFilterListType: NetworkFilterListTrait ^^^^

antonok-edm · 2025-04-17T18:30:17Z

src/blocker.rs

+        let mut disabled_directives: HashSet<String> = HashSet::new();
+        let mut enabled_directives: HashSet<String> = HashSet::new();


why not &str anymore?

antonok-edm · 2025-04-17T18:31:13Z

src/data_format/mod.rs

@@ -5,13 +5,16 @@
 //! serialization/deserialization implementations and can automatically dispatch to the appropriate
 //! one.

+#![allow(dead_code)]


if we're not using these anymore, let's just remove them

antonok-edm · 2025-04-17T18:43:30Z

src/engine_serializer.rs

+}
+
+pub trait EngineSerializer {
+    fn serialize_raw(&self) -> Result<Vec<u8>, SerializeError>;


let's call it just serialize.

Some context
We previously had serialize, which serialized a compressed format. Then we needed to have an uncompressed format; I added serialize_compressed and serialize_raw to maintain backward compatibility. serialize remained as a direct wrapper around serialize_compressed, but with a #[deprecated] marker explaining that it would later be removed and the default behavior will be changed to serialize_raw.
serialize_compressed has been removed for enough time that I think it's safe to shift back to serialize.

antonok-edm · 2025-04-17T19:17:13Z

src/filters/fb_network.rs

+// Implement ExactSizeIterator for FilterPartIterator
+impl<'a> ExactSizeIterator for FlatPatternsIterator<'a> {


this comment isn't adding any useful information and it is already outdated 😅

antonok-edm · 2025-04-17T19:49:30Z

src/filters/fb_network.rs

+                .iter()
+                .map(|x| self.get_or_insert_unique_domain_hash(x))
+                .collect();
+            o.sort_unstable();


We should be able to get rid of the sorting behavior. We used that previously to make differential updates of serialized buffers feasible, but that's no longer useful now that we distribute plaintext lists.

github-actions · 2025-04-28T07:09:43Z

[puLL-Merge] - brave/adblock-rust@446

Description

This PR introduces a significant refactoring and optimization of the adblock-rust crate's serialization and deserialization functionality. The primary change is the implementation of FlatBuffers for storing network filters, which should provide better performance and memory efficiency. Additionally, the PR reorganizes the serialization code by moving it into a dedicated trait and module.

Changes

Changes

Cargo.toml:
- Added "flatbuffers-storage" to the default features
Benchmarks:
- Moved serialization/deserialization benchmarks from bench_matching.rs to a new dedicated file bench_serialization.rs
- Updated benchmark calls to use the new serialize() method instead of serialize_raw()
Engine Serializer:
- Introduced a new EngineSerializer trait in src/engine_serializer.rs
- Moved serialization/deserialization logic from engine.rs to this new module
- Renamed serialize_raw() to serialize() for cleaner API
FlatBuffers Implementation:
- Added a new module src/filters/fb_network.rs that implements FlatBuffers storage for network filters
- Updated the FlatBuffers schema to include raw_line field
- Updated the network filter list implementation to use FlatBuffers
Network Filter Matching:
- Refactored domain matching code to be more modular and efficient
- Split domain checking functions into separate parts (checking included domains, excluded domains)
- Added mapped versions that work with the FlatBuffers representation
Blocker Implementation:
- Removed some unused methods from Blocker like add_filter and filter_exists
- Updated the CSP directive processing to use owned strings instead of references
Examples and Tests:
- Updated all examples and tests to use the new EngineSerializer trait
- Fixed imports and method calls throughout the codebase

sequenceDiagram
    participant Client
    participant Engine
    participant EngineSerializer
    participant Blocker
    participant FlatNetworkFilter
    participant NetworkFilterList

    Client->>Engine: from_rules(rules)
    Engine->>Blocker: new(network_filters)
    Blocker->>NetworkFilterList: new(filters)
    NetworkFilterList->>FlatNetworkFiltersListBuilder: new()
    FlatNetworkFiltersListBuilder->>FlatNetworkFiltersListBuilder: add(filter)
    FlatNetworkFiltersListBuilder->>NetworkFilterList: finish()
    
    Client->>Engine: serialize()
    Engine->>EngineSerializer: serialize()
    EngineSerializer->>Client: serialized_data
    
    Client->>Engine: deserialize(serialized_data)
    Engine->>EngineSerializer: deserialize(serialized_data)
    EngineSerializer->>Engine: updated Engine
    
    Client->>Engine: check_network_request(request)
    Engine->>Blocker: check(request)
    Blocker->>NetworkFilterList: first_match(request)
    NetworkFilterList->>FlatNetworkFilter: matches(request)
    FlatNetworkFilter->>NetworkFilterList: CheckResult
    NetworkFilterList->>Blocker: result
    Blocker->>Engine: result
    Engine->>Client: BlockerResult

Possible Issues

Some public API methods have been removed from the Blocker class, which could cause compatibility issues if consumers were using these methods directly.
The change from references to owned strings in CSP directive processing might introduce additional allocations.
Removing serialize_raw() in favor of serialize() is a breaking change that will require consumers to update their code.

Security Hotspots

The use of unsafe code in the FlatBuffers implementation for casting between slices of different types (in src/filters/fb_network.rs) should be carefully reviewed, although this appears to be standard practice when working with FlatBuffers.

github-actions · 2025-04-28T07:11:37Z

src/network_filter_list.rs

        if self.filter_map.is_empty() {
            return None;
        }

+        let filters_list =
+            unsafe { fb::root_as_network_filter_list_unchecked(&self.flatbuffer_memory) };


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

github-actions · 2025-04-28T07:11:37Z

src/network_filter_list.rs


        if self.filter_map.is_empty() {
            return filters;
        }

+        let filters_list =
+            unsafe { fb::root_as_network_filter_list_unchecked(&self.flatbuffer_memory) };


_{reported by reviewdog 🐶}
[semgrep] Detected 'unsafe' usage, please audit for secure usage

Source: https://semgrep.dev/r/rust.lang.security.unsafe-usage.unsafe-usage

Cc @thypon @kdenhartog

boocmp added 3 commits March 16, 2025 18:03

Added NetworkFilterListTrait to futher provide the flatbuffer impl.

d7ba6ae

Added FlatNetworkFilterList implementation.

7ce02f0

Added raw_line field to flatbuffers for testing.

Blocker type for data_format lib.

cbbcb10

github-actions bot reviewed Mar 28, 2025

View reviewed changes

boocmp force-pushed the flatbuffers_impl branch from 91c5ec7 to cacfcc6 Compare April 9, 2025 00:55

github-actions bot reviewed Apr 9, 2025

View reviewed changes

github-actions bot added the needs-security-review label Apr 9, 2025

github-actions bot assigned kdenhartog and thypon Apr 9, 2025

Added EngineSerializer trait

a3df20d

boocmp force-pushed the flatbuffers_impl branch from cacfcc6 to a3df20d Compare April 9, 2025 01:05

boocmp marked this pull request as ready for review April 9, 2025 01:38

boocmp requested review from atuchin-m and antonok-edm April 9, 2025 01:38

atuchin-m approved these changes Apr 9, 2025

View reviewed changes

boocmp force-pushed the flatbuffers_impl branch from 6a55c00 to a3df20d Compare April 9, 2025 11:30

github-actions bot added the puLL-Merge label Apr 9, 2025

Renaming.

541cece

github-actions bot reviewed Apr 10, 2025

View reviewed changes

Review.

b01147d

antonok-edm reviewed Apr 18, 2025

View reviewed changes

github-actions bot reviewed Apr 28, 2025

View reviewed changes

boocmp force-pushed the flatbuffers_impl branch 2 times, most recently from 6b08928 to 5087b75 Compare April 28, 2025 07:25

NetworkFilterList impl replaced with flatbuffers.

0238db7

boocmp force-pushed the flatbuffers_impl branch from 5087b75 to 0238db7 Compare April 28, 2025 07:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Flatbuffers impl #446

Flatbuffers impl #446

boocmp commented Mar 28, 2025

github-actions bot left a comment •

edited

Loading

github-actions bot Apr 9, 2025

atuchin-m Apr 9, 2025

github-actions bot Apr 9, 2025

github-actions bot Apr 9, 2025

atuchin-m Apr 9, 2025

github-actions bot Apr 9, 2025

atuchin-m left a comment

atuchin-m Apr 9, 2025

atuchin-m Apr 9, 2025

boocmp Apr 10, 2025

atuchin-m Apr 9, 2025

boocmp Apr 9, 2025

atuchin-m Apr 9, 2025

atuchin-m Apr 9, 2025

boocmp Apr 10, 2025

atuchin-m Apr 10, 2025

boocmp Apr 11, 2025

atuchin-m Apr 9, 2025

boocmp Apr 9, 2025

github-actions bot Apr 10, 2025

github-actions bot Apr 10, 2025

antonok-edm Apr 17, 2025

antonok-edm Apr 17, 2025

antonok-edm Apr 17, 2025

antonok-edm Apr 17, 2025

antonok-edm Apr 17, 2025

antonok-edm Apr 17, 2025

github-actions bot commented Apr 28, 2025

Changes

github-actions bot Apr 28, 2025

github-actions bot Apr 28, 2025

		let mut disabled_directives: HashSet<String> = HashSet::new();
		let mut enabled_directives: HashSet<String> = HashSet::new();

		// Implement ExactSizeIterator for FilterPartIterator
		impl<'a> ExactSizeIterator for FlatPatternsIterator<'a> {

Flatbuffers impl #446

Are you sure you want to change the base?

Flatbuffers impl #446

Conversation

boocmp commented Mar 28, 2025

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

Rust Benchmark

github-actions bot Apr 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot Apr 9, 2025

Choose a reason for hiding this comment

github-actions bot Apr 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot Apr 9, 2025

Choose a reason for hiding this comment

atuchin-m left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot Apr 10, 2025

Choose a reason for hiding this comment

github-actions bot Apr 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Apr 28, 2025

Description

Changes

Possible Issues

Security Hotspots

github-actions bot Apr 28, 2025

Choose a reason for hiding this comment

github-actions bot Apr 28, 2025

Choose a reason for hiding this comment

github-actions bot left a comment •

edited

Loading