`hash-roll` vs `rsroll`, benchmarks, gear CDC, and some other questions

[dpc]

Hi, author of rdedup here. I'm currently using rsroll and it looks your crate is quite well polished comparing to rsroll. I wonder if there are any not-immediately obvious benefits of hash-roll.

I am currently on an optimization mission for rdedup, and the rolling checksum is sometimes a bottleneck that is not easily parallelizable.

I have filled some PRs: https://github.com/aidanhs/rsroll/pulls optimizing the performance, and first: I think some of it there could be re-done for hash-roll (especially Gear). Second: are you planning to add benchmarks?

I am planning to implement FastCDC soon, as it seems to be the state of the art https://www.usenix.org/node/196197 . If you are aware of anything better (especially faster) please let me know. If you're interested in helping, eg. by implementing this stuff in hash-roll or rdedup (I am actually not yet sure at which layer some of this stuff will be) etc. that would be awesome, and please let me know. Just asking and trying to be welcoming, no presure though.

[codyps]

it looks your crate is quite well polished

Well, I'd be cautious about that: I'm not at the point where I'm really happy with the splitting interfaces, and right now only Bup implements the Splitter2 trait (which I've been using on a project). If I spend more time on this, it's likely interfaces will evolve a bit.

think some of it there could be re-done for hash-roll

I'll take a look, thanks for the pointer.

are you planning to add benchmarks?

There are some benchmarks already (src/benches.rs). cargo bench --features nightly should run them. I'll probably add the rsroll crate to compare it :) . I'll also look at adjusting my benchmarks so they're a bit more comparable to the ones in rsroll.

If you are aware of anything better (especially faster) please let me know.

I'm not, I have not been looking at rolling hash/content aware splitting algorithms recently.