Migrate to assembly from OpenSSL

[newpavlov]

Initially crates in this repository were a simple proof-of-work experiment and assembly from Project Nayuki was taken without much consideration to performance. But some people use asm feature and expect performance on par with OpenSSL which is obviously not true today.

So I think we should migrate to OpenSSL assembly and maybe rename this repository to openssl-asm or something. This will require some code refactoring as OpenSSL hash assembly processes several blocks at a time and for the best performance we will have to accommodate for it. My initial experiments with MD5 haven't found any difficult problems.

The main question I currently have is: should we pack Perl scripts into crates or generated assembly? First option will result in a lighter crates (especially considering various possible "flavors" which Perl scripts support), but will require Perl to be installed on the system. For Linux it's not a big deal, but I think for Windows it can be a problem.

[tarcieri]

Preserving the Perl will make it easier to track upstream changes. It sure would be nice to get rid of it though.

It just does templating, so if it can be adapted to some other templating system or syntax, possibly as a mechanical translation from the Perl, I'd be a fan of that.

[newpavlov]

I thought about adding openssl repo as a submodule and using a small bash script which will regenerate assembly for all crates. So time to time we will manually update submodule, execute the script, and check if there any changes.

[tarcieri]

@newpavlov as a minimum viable solution to removing the Perl dependency, that sounds fine to me.

[tarcieri]

Here's an alternative idea that sounds fairly difficult and likely not worth it, but kind of fun:

I think it might be possible to convert the Perl templates (through a similar mechanical preprocessing step on the upstream files from the openssl repo with a checked-in result) into some other template, be it just a format! string or what have you, and then include_str! it, replacing the Perl templating with a combination of CPU architecture-dependent conditional compilation and Rust macros.

[newpavlov]

It will be essentially a limited Perl-to-Rust convertor, so indeed it's probably not worth the trouble. :)

[tarcieri]

I think it's a lot easier than that, but it would essentially involve rewriting the library-level parts of the Perl templating system as e.g. Rust macros, in addition to extracting and converting/munging the asm templates.

Still seems like a lot of work.

[PrismaPhonic]

I would like to see some assembly that takes advantage of avx2. Currently, the Golang SHA-1 library does: https://github.com/golang/go/blob/master/src/crypto/sha1/sha1block_amd64.s

This uses the same algorithm as the Linux kernel: https://github.com/torvalds/linux/blob/master/arch/x86/crypto/sha1_avx2_x86_64_asm.S

[newpavlov]

@PrismaPhonic
IIUC OpenSSL assembly (sha1, sha256) does account for AVX2 instruction set, so it's matter of generating appropriate assembly files. Although I haven't compared performance with the files linked by you.

[PrismaPhonic]

For sure. I know I'm just now stepping into this conversation and am not even a contributor (yet), but I personally would prefer avoiding Perl scripts altogether. I would vote for sticking to raw assembly files, and check Rust side about which instruction set should be used.

EDIT: My reasoning behind that is that combining perl & asm makes for extremely unreadable code. I'm frankly a bit shocked that the openssl devs decided to use that approach. I'm not sure there exists a less readable way of generating asm.

[newpavlov]

It looks like at least for hash functions OpenSSL assembly handles dynamic dispatch itself by reading data from OPENSSL_ia32cap_P, which contains CPUID results (on ARM it's OPENSSL_armcap_P). I think it will be a bit tricky to setup this constant correctly, but should be doable.

[joshtriplett]

Quick note: from a licensing point of view, this needs some care to make sure that it's safe to do. OpenSSL has relicensed to Apache 2.0, but the asm modules in OpenSSL appear to be additionally dual-licensed under BSD and GPLv2. So, if you want to integrate that assembly, you'd need to change the license of this crate accordingly (BSD-3-Clause OR Apache-2.0 would work).

[tarcieri]

@joshtriplett yeah. See recent licensing discussion on #32 (review)