Implement Lizard encoding/decoding

[rozbb]

The Lizard encoding/decoding algorithms specify a map from 16-byte strings to the Ristretto group, and back. This is useful for ElGamal encryption, where plaintexts are group elements.

Signal relies on Lizard, and has been vendoring curve25519-dalek for years now (bc we don't export FieldElement). I figured upstreaming would be a valuable addition to dalek, so long as it is extensively documented so as to be maintainable. It also doesn't add any new dependencies.

Some highlights of this PR:

• Implements and exposes the Lizard algorithms, as well as some lower-level map_to_curve algorithms necessary for downstream. Most of this is from the Signal repo
• Includes a detailed writeup of the Lizard algorithm, how it works, and how the pseudocode maps to the Rust code
• Upgrades our Sage reference file to Python3 (wow)

This was prepared jointly with @rolfe-signal and with input from @jrose-signal. Thank you all so much for your work on this!

[JobDoesburg]

Any chance on this being merged/released soon?

[rozbb]

Yes apologies. I reckon we can merge this week. I need to address Jordan’s points above

[tarcieri]

I think the build failures are due to changes between signature v3.0.0-rc.3 and v3.0.0-rc.4.

As a general recommendation, I think we should check in Cargo.lock to keep the build deterministic. Otherwise changes in prerelease crates like that will cause random breakages like this in unrelated PRs.

Otherwise I can prepare a PR to do the upgrade.

[tarcieri]

Oh wait, signature was updated in #828. So this probably just needs a rebase/merge.

[rozbb]

@jrose-signal Before we merge this (and once the build is working again), do you wanna do a cargo patch for libsignal, do the necessary edits, and see that all tests pass? That way we can keep all the fixes in this branch

[jrose-signal]

The good news: adoption was straightforward and ran into no unexpected trouble: signalapp/libsignal#636

The bad news:

% cargo bench -p zkgroup -- decrypt
Benchmarking decrypt_uuid: Collecting 100 samples in estimated 5.5576 s
decrypt_uuid            time:   [157.25 µs 157.59 µs 157.97 µs]
                        change: [+21.553% +22.019% +22.447%] (p = 0.00 < 0.05)
                        Performance has regressed.

Benchmarking decrypt_profile_key: Collecting 100 samples in estimated 5.9268 s
decrypt_profile_key     time:   [599.99 µs 600.79 µs 601.72 µs]
                        change: [+6.9417% +7.1157% +7.2925%] (p = 0.00 < 0.05)
                        Performance has regressed.

I'm going to do a bit of profiling and see if there's an obvious culprit—maybe the negations are noticeable in the profile, maybe it's something else.

[jrose-signal]

Aha, I think that's almost entirely due to the repeated e_inv_positive() calls we already discussed. If you push the fix for that I'll run the benchmarks again.

[rozbb]

Alright @jrose-signal, I fixed the unnecessary recalculation and rebased the branch on main. Lmk if the benchmarks are improved.

[rozbb]

@tarcieri one last organizational thing: what do you think about making ristretto.rs into a module and shoving all the elligator stuff in ristretto/elligator.rs? We can also expose map_to_curve and map_to_curve_inverse under ristretto instead of lizard. Yes they have footguns, but they come with big warnings anyway.

[jrose-signal]

We are down to a 2% regression for decrypt_profile_key and no change for decrypt_uuid, which is a little curious because they both only call map_to_curve_inverse once; the one that gets called more in the profile key case is map_to_curve, and that one seems a lot closer to the implementation we're currently using. I'll poke at the profiles a little more to see if I spot anything, but I don't think that needs to block the changes here—we're into the point of confounding factors like "it's possible sha2 0.11 is optimizing slightly worse than sha2 0.10 on this system, which wouldn't be ideal but such is life sometimes".

[tarcieri]

what do you think about making ristretto.rs into a module and shoving all the elligator stuff in ristretto/elligator.rs?

@rozbb sounds fine to me.

we're into the point of confounding factors like "it's possible sha2 0.11 is optimizing slightly worse than sha2 0.10 on this system, which wouldn't be ideal but such is life sometimes".

@jrose-signal 2% is definitely in the range where very subtle things unrelated to the particular change can be happening, like link order changes, and unless there's an obvious culprit it probably isn't worth worrying about

[jrose-signal]

(copyediting)

[JobDoesburg]

Any plans on when this will also be released?

[JobDoesburg]

@rozbb any plans on when this will be released?