briansmith/crypto-bench

Benchmarks for crypto libraries (in Rust, or with Rust bindings)

crypto-bench's Issues

Add x86_64-pc-windows-msvc and i686-pc-windows-msvc configurations

Unfortunately, crypto-bench doesn't build on pc-windows-msvc now because several of the libraries fail to build. As Cargo.toml only supports adding dependencies per-target, not removing them, that means we'd need to add an exhaustive set of configurations to Cargo.toml in order to get this to work. Hopefully the instead the libraries will soon improve so that they build and work correctly on pc-windows-msvc.

Publish results somewhere

It would be nice to have place to send people from Reddit asking about performance of crypto in Rust.

Add ECDSA verification benchmarks

Most immediately, we need to benchmark ring against OpenSSL for the P-256 and P-384 curves on x86-64 (Haswell and later).

ECDSA verification isn't (necessarily) a constant-time operation; the speed of the verification may depend on the value of the public key and/or the value of the signature (r, s). We know that ring and recent versions of OpenSSL currently use a constant-time implementation for P-256 verification. ring uses a constant-time implementation for P-384 too, but OpenSSL uses a variable-time implementation. Therefore, we need to use multiple public keys and multiple signatures for each public key in the benchmark. This makes the benchmarking more difficult than what we do for other algorithms, where we assume that the values of the inputs don't affect the timing.

Add rust-nettle (nettle) benchmarks

There is a rust-nettle crate, so that should help.

Nettle is a very interesting library that implements a few things in a very interesting way, differently than many crypto libraries (Rust or C or otherwise) do them. It even has assembly-language-optimized P-384, which almost nobody else bothers with! So, it might very well be faster than everything else, at least in some benchmarks.

@johnschug Do you have any tips for using your rust-nettle crate? the goal here is to measure its performance relative to the other libraries being benchmarked in this crate.

Add 32-bit Linux and Mac OS X builds to Travis CI

Blocked on rust-lang/rust#29388. I haven't figured out how to cross-compile to 32-bit on Linux yet.

Adding new primitives/constructions?

I was wondering whether crypto-bench would accept adding implementations of other primitives/constructions than those listed in the README? Things like BLAKE2*, etc.

I was also wondering if the the ChaCha20-Poly1305 benchmark-listing, in the README, cover a XChaCha20-Poly1305 implementation? I'm asking because sodiumoxide is listed to have benchmarks for Salsa20-Poly1305, when in fact it calls a XSalsa20-Poly1305 implementation.

Add iOS Securtiy Framework (Common Crypto, etc.) benchmarks

It would be useful to see how Rust-implemented crypto libraries compare, performance-wise, to the native crypto APis in Mac OS X and especially on iOS. In particular, this would help crypto library authors decide whether to defer some or all algorithms to the OS on OS X and/or iOS. Especially since code size is a significant issue on iOS, there's a lot of incentive to use the native code if it performs well and is correct.

I don't know of any Rust bindings for Common Crypto, so either (a) the initial benchmarks can be in C, or (b) some bindings need to be made.

Stop pulling crates from github

Just an issue I ran into when using this repository is that it references the github repo for a number of crates. Various projects use various methods working with git and its quite likely that at any given time one of the git HEADs will be broken or may induce breaking changes since this crate has been updated. In my case, tests for the ring crate failed but switching to the most recent crates.io release resolved all issues.

In addition to avoiding potentially broken master branches, using crates.io releases is also much more representative of expected performance as most people are not going to be linking against the GitHub repository.

Add RSA verification benchmarks

Most immediately, we need to compare ring and OpenSSL performance. I'll merge #20 and then we can build on top of that.

For now, we only need to measure the case where the public key has e == 67737 and where the length of n is {2048, 3072, 4096} bits.

RSA verification is not (necessarily) a constant-time operation, so there might be some timing variance for different public keys. But the timing variance is mostly in the value of e, so if we fix that, we should be able to get meaningful numbers for ring and OpenSSL, given what we know about how their modular multiplication and exponentiation primitives are implemented.

If we determine that there are further timing variances between public keys, we can improve the tests later in a separate bug/PR.

Use stable Rust instead of nightly Rust

Benchmarks on nightly Rust aren't as useful as benchmarks on stable Rust would be since most users want to use stable Rust (at least, according to my experience).

I would be happy to switch to either https://crates.io/crates/bencher or https://crates.io/crates/criterion; see https://www.reddit.com/r/rust/comments/8tplwr/what_is_the_standard_way_to_run_benchmarks_in/ for more discussion. One problem may be "black box" support, or the lack of it.

Add sodiumoxide (libsodium) benchmarks

The library is at https://github.com/dnaq/sodiumoxide.

I had tried to do this before, but I had trouble getting sodiumoxide to build because it couldn't find libsodium. I think libsodium-sys may be helpful, because it builds and links libsodium itself?

@zonyitoo @dnaq Do I understand the above correctly, that sodiumoxide should now work even on systems that don't have libsodium installed already?

Add Windows Crypto API (CNG and maybe CAPI) benchmarks

It would be useful to see how Rust-implemented crypto libraries compare, performance-wise, to the native crypto APis in Windows. In particular, this would help crypto library authors decide whether to defer some or all algorithms to the OS on Windows.

I don't know of any Rust bindings for these APIs, so either (a) the initial set of benchmarks can be in C, or (b) some bindings need to be made.

Add benchmarks for Andrew Moon's implementations of ChaCha20-Poly1305

The primitives are at https://github.com/floodyberry/poly1305-opt and https://github.com/floodyberry/chacha-opt. We'd need to write a Rust wrapper around them that composes them into the ChaCha20-Poly1305 construct.

Add ease-of-installation benchmark

For implementations that can be installed by just adding a dependency in Cargo.toml (ring, octavo, rust-crypto), this won't change anything.

Other implementations that require manual installation steps should become optional dependencies and should be off by default, enabled with a feature flag. In order to run the benchmarks, the user will have to learn and perform the manual installation steps first. This way, the benchmark runner can measure qualitatively and quantitatively the usability of the library as far as installation is concerned.

This is would be a very useful benchmark because ease-of-use is a key factor.

Reorganize as a Cargo workspace with multiple crates.

See rust-lang/cargo#2122.