I wrote a polars plugin with twox-hash

[paddymul]

Hi,

Thank you for writing this library.

I wanted to share a project I built with it - pl_series_hash. It is a plugin for the dataframe library polars to hash series.

I wanted to have a very fast hash function per series so that I can cache summary stats for another project.

This was my first project in Rust. So I'm still learning.

Here is the rust file
https://github.com/paddymul/pl_series_hash/blob/main/src/expressions.rs
and the python test
https://github.com/paddymul/pl_series_hash/blob/main/tests/test_pl_series_hash.py

I have tests that verify this, but I'd still like to get a sanity check that I'm hashing series properly.

I'm worried about hash collisions from a poor implementation. Here is my approach

For each series I first write out a type identifier.

For each element in a series I add the bytes, for strings I also write a STRING_SEPERATOR of 128u16 which isn't a valid UTF8 symbol and shouldn't ever appear.
For NANs/Nulls I write out NAN_SEPERATOR - 129u16 also an invalid unicode character.

Next I write out the array position in bytes (u64)

All of this is then hashed.

[shepmaster]

I'm glad to hear that this crate proved useful to you!

However, I don't have the knowledge to judge if a given hashing scheme is good or not. I mostly ported the existing xxhash logic and tried to structure it as idiomatic Rust. I'm just a regular programmer, not a hashing enthusiast. 😉

The good news is that I don't think you need hasher-specific feedback: the questions you are asking about quality should apply equally to any hash algorithm. To that end, I'd encourage you to ask somewhere more general, such as on the users forum or the community Discord server. If you are comfortable, you could also probably ask in a general programming context, such as one of the Stack Exchange sites.