trevorr/circe

Circe: A High-Performance Hash Algorithm Framework & Library

Circe: A High-Performance Hash Algorithm Framework & Library

Circe is a hash algorithm framework that supports pluggable hash implementations discovered at runtime. Its primary goal is to enable the usage of high-performance native-code and hardware-accelerated hash algorithms, while also providing pure Java implementations when native libraries or hardware acceleration are not available.

Circe provides hash functions in both stateful and stateless forms. The stateful form accumulates state internally from potentially multiple input segments and provides access to the output as various primitive types and byte arrays. The stateless form is available when the output can be represented by an int or a long and provides the output immediately for a given input. Additionally, for algorithms like CRC where the state and output are essentially interchangeable, an incremental stateless form is provided that allows more data to be hashed using the output from hashing a previous segment. To facilitate access from native code, all hash interfaces accept input from direct (and non-direct) java.nio.ByteBuffer objects and (unsafe) long memory addresses, in addition to Java byte arrays.

Currently Implemented Algorithms

All of Circe's hash algorithms are implemented in optional provider modules:

• circe-crc: Pure Java implementations of any CRC function up to 64 bits, including CRC-32, CRC-32C, and CRC-64. CRC-32 can be provided by java.util.zip.CRC32, which uses the native implementation in zlib.
• circe-crc32c-sse42: Hardware-accelerated CRC-32C using the x86-64 SSE 4.2 crc32 and (if available) pclmulqdq instructions. Processes almost 15 GB/s on a desktop Intel Core i7-2600, compared to 1 GB/s with java.util.zip.CRC32. Currently supports MSVC on Windows and GCC on Linux and Mac OS X, with runtime detection of processor support.
• circe-digest: Wrapper around javax.security.MessageDigest, providing MD5, SHA-1, SHA-256, SHA-384, and SHA-512.
• circe-guava: Wrapper around com.google.common.hash.Hashing, providing MurmurHash3 and SipHash-2-4.

Dependencies

Circe is designed to be very modular and have limited runtime dependencies. Most of its modules have no required external dependencies. The only exception so far is a Google Guava dependency in circe-cache, an optional cache for hash functions using tables generated at runtime, and circe-guava, an optional provider for the hash algorithms implemented within Guava.

For unit testing, Circe uses JUnit 4, JMockit, and Hamcrest.

Building

Circe is built using Maven, with native code built using the Maven NAR Plugin. To build the entire project and run all unit tests, simply run mvn install. To perform a full release build, including source and Javadoc JARs, and local Maven install, run mvn install -P release. To build aggregated Javadoc (placed under target/site/apidocs), run mvn site.

License

Circe is released under the Apache Software License, Version 2.0.

circe's Issues

CRC32C Incremental-Checksum is not giving correct result

Hi,

I have created following test-case in CRCTest to verify incremental CRC32C checksum computation.

   @Test
    public void testCRC32CIncremental() {
        IncrementalIntHash crc32c = PROVIDER.getIncrementalInt(CRC32C);
        String data = "data";
        String combine = data + data;

        int combineChecksum = crc32c.calculate(combine.getBytes());
        int dataChecksum = crc32c.calculate(data.getBytes()); 
        int incrementalChecksum = crc32c.resume(dataChecksum, data.getBytes());
        assertEquals(combineChecksum, incrementalChecksum);
    }

We have two separate ByteBuffers and we want to compute combined-checksum on it so, I am trying to use incremental checksum but it seems it is giving incorrect result. @trevorr Can you please advice if I am missing anything here.

Packaging and Maven-Repo artifact deployment

hi,

Thanks for providing java solution for crc32c-sse4.2 using native library. I can see you have uploaded SNAPSHOT version of this artifact at : Sonatype-Staging_maven-repo

However, we are not able to use this solution due to following issues:

1. Library not present into maven-repo: The available jni library in maven-repo is only available for windows OS and we can't use it for "Mac OS" or "Linux OS".

Solutions:
1. Upload jni library to maven repo for all well-known OS: Right now, Maven repo only contains jni for windows so other system can't use this library. So, can you also please upload library for MacOSX and Linux as well.

2. One Packaging of all OS specific JNI library using vagrant: Checkout RocksDB Build it uses vagrant to build jni for all OS environments and packages them into one artifact which you can deploy to maven repo

1. Can't use artifact as it as:
   When we build circe-crc32c-sse42 using maven build:

• It internally generates "jni" library based on OS where we build and it packages this generated jni library under .nar artifact. e.g: circe-crc32c-sse42.nar
• Nar Plugin goal also generates NarSystem.java which tries to load this jni library at runtime using System.loadLibrary("circe-crc32c-sse42-0.1-SNAPSHOT"); which fails obviously because it asks ClassLoader to load library from java.library.path or sun.boot.library.path. And as this jni library is packaged under .nar so, definitely it is not present in either library.path and hence, initialization of the interface which provides API to compute checksum Sse42Crc32C.java fails.

Solution

• So, I created a pull request : which generates circe-crc32c-sse42.jar instead circe-crc32c-sse42.nar artifact and internally loads generated jni library without messing up with library.path and user can use this artifact without performing any additional magic.
  I have also added a test case which cleanly uses Sse42Crc32C.java to compute checksum by simply adding dependency of artifact com.scurrilous:circe-crc32c-sse42:${project.version} and without worrying about configuring library.path.