Giter Site home page Giter Site logo

t1ha's Introduction

t1ha

Fast Positive Hash, aka "Позитивный Хэш" by Positive Technologies. Included in the Awesome C list of open source C software.

The Future will (be) Positive. Всё будет хорошо.

License: Zlib Build Status Build status CircleCI Coverity Scan Status

Briefly, it is a portable non-cryptographic 64-bit hash function:

  1. Intended for 64-bit little-endian platforms, predominantly for Elbrus and x86_64, but portable and without penalties it can run on any 64-bit CPU.

  2. In most cases up to 15% faster than xxHash, StadtX, MUM and others portable hash-functions (which do not use specific hardware tricks).

    Currently wyhash outperforms t1ha on x86_64. However next version t1ha3_atonce() will be even faster on all platforms, especially on E2K, architectures with SIMD and most RISC-V implementations. In addition, it should be noted that wyhash have a "blinding multiplication" flaw and can lose entropy (similarly as described below). For instance, when data could be correlated with the seed ^ _wypN values or equal to it. Another case is where one of _wymum() multipliers becomes zero. In result of such blinding all previous data will not be influence to the hash value.

  3. Licensed under zlib License.

Also pay attention to Rust, Erlang and Golang implementations.

FAQ: Why t1ha don't follow NH-approach like FARSH, XXH3, HighwayHash and so on?

Okay, just for clarity, we should distinguish functions families: MMH (Multilinear-Modular-Hashing), NMH (Non-linear Modular-Hashing) and the next simplification step UMAC's NH.

Now take a look to NH hash-function family definition: Wikipedia

It is very SIMD-friendly, since SSE2's _mm_add_epi32() and _mm_mul_epu32() is enough for W = 32. On the other hand, the result of the inner multiplication becomes zero when (m[2i] + k[2i]) mod 2^32 == 0 or (m[2i+1] + k[2i+1]) mod 2^32 == 0, in which case the opposite multiplier will not affect the result of hashing, i.e. NH function just ignores part of the input data. I called this an "blinding multiplication". That's all. More useful related information can be googled by "UMAC NH key recovery attack".

The right NMH/NH code without entropy loss should be looking like this:

    uint64_t proper_NH_block(const uint32_t *M /* message data */,
                             const uint64_t *K /* 64-bit primes */,
                             size_t N_even, uint64_t optional_weak_seed) {
      uint64_t H = optional_weak_seed;
      for (size_t i = 0; i < N_even / 2; ++i)
        H += (uint64_t(M[i*2]) + K[i*2]) * (uint64_t(M[i*2+1]) + K[i*2+1]);
      return H;
    }

Usage

The t1ha library provides several terraced hash functions with the dissimilar properties and for a different cases. These functions briefly described below, see t1ha.h for more API details.

To use in your own project you may link with the t1ha-library, or just add to your project corresponding source files from /src directory.

Please, feel free to fill an issue or make pull request.

t1ha0 = 64 bits, "Just Only Faster"

Provides fast-as-possible hashing for current CPU, including 32-bit systems and engaging the available hardware acceleration. You can rest assured that t1ha0 faster than all other fast hashes (with comparable quality) so, otherwise we will extend and refine it time-to-time.

On the other hand, without warranty that the hash result will be same for particular key on another machine or another version. Moreover, is deliberately known that the result will be different for systems with different bitness or endianness. Briefly, such hash-results and their derivatives, should be used only in runtime, but should not be persist or transferred over a network.

Also should be noted, the quality of t1ha0() hashing is a subject for tradeoffs with performance. Therefore the quality and strength of t1ha0() may be lower than t1ha1() and t1ha2(), especially on 32-bit targets, but then much faster. However, guaranteed that it passes all SMHasher tests.

Internally t1ha0() selects most faster implementation for current CPU, for now these are includes:

Implementation Platform/CPU
t1ha0_ia32aes_avx() x86 with AES-NI and AVX extensions
t1ha0_ia32aes_avx2() x86 with AES-NI and AVX2 extensions
t1ha0_ia32aes_noavx() x86 with AES-NI without AVX extensions
t1ha0_32le() 32-bit little-endian
t1h0a_32be() 32-bit big-endian
t1ha1_le() 64-bit little-endian
t1ha1_be() 64-bit big-endian
t1ha2_atonce() 64-bit little-endian

t1ha1 = 64 bits, baseline fast portable hash

The first version of "Fast Positive Hash" with reasonable quality for checksum, hash tables and thin fingerprinting. It is stable, e.g. returns same result on all architectures and CPUs.

  1. Speed with the reasonable quality of hashing.
  2. Efficiency on modern 64-bit CPUs, but not in a hardware.
  3. Strong as possible, until no penalties on performance.

Unfortunatelly, Yves Orton discovered that t1ha1() family fails the strict avalanche criteria in some cases. This flaw is insignificant for the t1ha1() purposes and imperceptible from a practical point of view. However, nowadays this issue has resolved in the next t1ha2() function, that was initially planned to providing a bit more quality.

The basic version of t1ha1() intends for little-endian systems and will run slowly on big-endian. Therefore a dedicated big-endian version is also provided, but returns the different result than the basic version.

t1ha2 = 64 and 128 bits, slightly more attention for quality and strength

The recommended version of "Fast Positive Hash" with good quality for checksum, hash tables and fingerprinting. It is stable, e.g. returns same result on all architectures and CPUs.

  1. Portable and extremely efficiency on modern 64-bit CPUs.
  2. Great quality of hashing and still faster than other non-t1ha hashes.
  3. Provides streaming mode and 128-bit result.

The t1ha2() is intended for little-endian systems and will run slightly slowly on big-endian systems.

t1ha3 = 128 and 256 bits, fast non-cryptographic fingerprinting

The next-step version of "Fast Positive Hash", but not yet finished and therefore not available.

Planned: t1ha4 = 128 and 256 bits, fast insecure fingerprinting

Planned: t1ha5 = 256 bits, fast Cryptographic, but with some limitations

Planned: t1ha6 = 256 and 512 bits, Cryptographic with reasonable resistance to acceleration on GPU and FPGA.

Planned: t1ha7 = 256, 512 and 1024 bits, Cryptographic, Strong Post-Quantum


Requirements and Portability:

  1. t1ha designed for modern 64-bit architectures. But on the other hand, t1ha doesn't require instructions specific to a particular architecture:
    • therefore t1ha could be used on any CPU for which compiler provides support 64-bit arithmetic.
    • but unfortunately t1ha could be dramatically slowly on architectures without native 64-bit operations.
  2. This implementation of t1ha requires modern GNU C compatible compiler, including Clang/LLVM, or Visual Studio 2013/2015/2017. For proper performance please use one of: GNU C 5.5 or later, CLANG 5.0 or later, Microsoft Visual Studio 2017 15.6 or later.

Acknowledgement:

The t1ha was originally developed by Leonid Yuriev (Леонид Юрьев) for The 1Hippeus project - zerocopy messaging in the spirit of Sparta!


Benchmarking and Testing

Current version of t1ha library includes tool for basic testing and benchmarking. Just try make check from t1ha directory.

To comparison benchmark also includes wyhash, xxHash, StadtX and HighwayHash functions. For example actual results for Intel(R) Core(TM) i7-4600U CPU:

$ make all && sudo make check
Build by GNU C/C++ compiler 9.3 (self-check passed)
Testing t1ha2_atonce... Ok
Testing t1ha2_atonce128... Ok
Testing t1ha2_stream... Ok
Testing t1ha2_stream128... Ok
Testing t1ha1_64le... Ok
Testing t1ha1_64be... Ok
Testing t1ha0_32le... Ok
Testing t1ha0_32be... Ok
Testing t1ha0_ia32aes_noavx... Ok
Testing t1ha0_ia32aes_avx... Ok
Testing t1ha0_ia32aes_avx2... Ok
Testing HighwayHash64_pure_c... Ok
Testing HighwayHash64_portable_cxx... Ok
Testing HighwayHash64_sse41... Ok
Testing HighwayHash64_avx2... Ok
Testing StadtX... Ok
Testing wyhash_v7... Ok

Preparing to benchmarking...
 - running on CPU#0
 - use RDPMC_40000001 as clock source for benchmarking
 - assume it cheap and stable
 - measure granularity and overhead: 54 cycles, 0.0185185 iteration/cycle

Bench for tiny keys (7 bytes):
t1ha2_atonce          :     17.250 cycle/hash,  2.464 cycle/byte,  0.406 byte/cycle,  1.217 GiB/s @3GHz
t1ha2_atonce128*      :     33.281 cycle/hash,  4.754 cycle/byte,  0.210 byte/cycle,  0.631 GiB/s @3GHz
t1ha2_stream*         :     77.500 cycle/hash, 11.071 cycle/byte,  0.090 byte/cycle,  0.271 GiB/s @3GHz
t1ha2_stream128*      :     99.125 cycle/hash, 14.161 cycle/byte,  0.071 byte/cycle,  0.212 GiB/s @3GHz
t1ha1_64le            :     18.219 cycle/hash,  2.603 cycle/byte,  0.384 byte/cycle,  1.153 GiB/s @3GHz
t1ha0                 :     15.102 cycle/hash,  2.157 cycle/byte,  0.464 byte/cycle,  1.391 GiB/s @3GHz
xxhash32              :     16.545 cycle/hash,  2.364 cycle/byte,  0.423 byte/cycle,  1.269 GiB/s @3GHz
xxhash64              :     27.203 cycle/hash,  3.886 cycle/byte,  0.257 byte/cycle,  0.772 GiB/s @3GHz
xxh3_64               :     15.102 cycle/hash,  2.157 cycle/byte,  0.464 byte/cycle,  1.391 GiB/s @3GHz
xxh3_128              :     18.219 cycle/hash,  2.603 cycle/byte,  0.384 byte/cycle,  1.153 GiB/s @3GHz
StadtX                :     20.203 cycle/hash,  2.886 cycle/byte,  0.346 byte/cycle,  1.039 GiB/s @3GHz
HighwayHash64_pure_c  :    607.000 cycle/hash, 86.714 cycle/byte,  0.012 byte/cycle,  0.035 GiB/s @3GHz
HighwayHash64_portable:    513.000 cycle/hash, 73.286 cycle/byte,  0.014 byte/cycle,  0.041 GiB/s @3GHz
HighwayHash64_sse41   :     69.438 cycle/hash,  9.920 cycle/byte,  0.101 byte/cycle,  0.302 GiB/s @3GHz
HighwayHash64_avx2    :     54.875 cycle/hash,  7.839 cycle/byte,  0.128 byte/cycle,  0.383 GiB/s @3GHz
wyhash_v7             :     14.102 cycle/hash,  2.015 cycle/byte,  0.496 byte/cycle,  1.489 GiB/s @3GHz

Bench for large keys (16384 bytes):
t1ha2_atonce          :   3493.000 cycle/hash,  0.213 cycle/byte,  4.691 byte/cycle, 14.072 GiB/s @3GHz
t1ha2_atonce128*      :   3664.000 cycle/hash,  0.224 cycle/byte,  4.472 byte/cycle, 13.415 GiB/s @3GHz
t1ha2_stream*         :   3684.000 cycle/hash,  0.225 cycle/byte,  4.447 byte/cycle, 13.342 GiB/s @3GHz
t1ha2_stream128*      :   3709.239 cycle/hash,  0.226 cycle/byte,  4.417 byte/cycle, 13.251 GiB/s @3GHz
t1ha1_64le            :   3644.000 cycle/hash,  0.222 cycle/byte,  4.496 byte/cycle, 13.488 GiB/s @3GHz
t1ha0                 :   1289.000 cycle/hash,  0.079 cycle/byte, 12.711 byte/cycle, 38.132 GiB/s @3GHz
xxhash32              :   8198.000 cycle/hash,  0.500 cycle/byte,  1.999 byte/cycle,  5.996 GiB/s @3GHz
xxhash64              :   4126.750 cycle/hash,  0.252 cycle/byte,  3.970 byte/cycle, 11.911 GiB/s @3GHz
xxh3_64               :   4929.000 cycle/hash,  0.301 cycle/byte,  3.324 byte/cycle,  9.972 GiB/s @3GHz
xxh3_128              :   4887.536 cycle/hash,  0.298 cycle/byte,  3.352 byte/cycle, 10.057 GiB/s @3GHz
StadtX                :   3667.000 cycle/hash,  0.224 cycle/byte,  4.468 byte/cycle, 13.404 GiB/s @3GHz
HighwayHash64_pure_c  :  55294.000 cycle/hash,  3.375 cycle/byte,  0.296 byte/cycle,  0.889 GiB/s @3GHz
HighwayHash64_portable:  44982.321 cycle/hash,  2.746 cycle/byte,  0.364 byte/cycle,  1.093 GiB/s @3GHz
HighwayHash64_sse41   :   7041.000 cycle/hash,  0.430 cycle/byte,  2.327 byte/cycle,  6.981 GiB/s @3GHz
HighwayHash64_avx2    :   4542.000 cycle/hash,  0.277 cycle/byte,  3.607 byte/cycle, 10.822 GiB/s @3GHz
wyhash_v7             :   3383.000 cycle/hash,  0.206 cycle/byte,  4.843 byte/cycle, 14.529 GiB/s @3GHz

The test tool support a set of command line options to selecting functions and size of keys for benchmarking. For more info please run ./test --help.

The --hash-stdin-strings option

One noteable option is --hash-stdin-strings, it intended to estimate hash collisions on your custom data. With this option test tool will hash each line from standard input and print its hash to standard output.

For instance, you could count collisions for lines from some words.list file by bash's command:

  ./t1ha/test --hash-stdin-strings < words.list | sort | uniq -c -d | wc -l

More complex example - count xxhash() collisions for lines from words.list and 0...10000 numbers, with distinction only in 32 bit of hash values:

  (cat words.list && seq 0 10000) | \
     ./t1ha/test --xxhash --hash-stdin-strings | \
     cut --bytes=-8 | sort | uniq -c -d | wc -l

SMHasher

SMHasher is a wellknown test suite designed to test the distribution, collision, and performance properties of non-cryptographic hash functions.

Reini Urban provides extended version/fork of SMHasher which integrates a lot of modern hash functions, including t1ha.

So, the quality and speed of t1ha can be easily checked with the following scenario:

git clone https://github.com/rurban/smhasher
cd smhasher
cmake .
make
./SMHasher City64
./SMHasher metrohash64_1
./SMHasher xxHash64
...
./SMHasher t1ha

For properly performance please use at least GCC 5.5, Clang 6.0 or Visual Studio 2017.

Scores

Please take in account that the results is significantly depend on actual CPU, compiler version and CFLAGS. The results below were obtained in 2016 with:

  • CPU: Intel(R) Core(TM) i7-6700K CPU;
  • Compiler: gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4);
  • CFLAGS: -march=native -O3 -fPIC;

The SMALL KEYS case

Order by average Cycles per Hash for 1..31 bytes (less is better).

Function MiB/Second Cycles/Hash Notes (quality, portability)
donothing 15747227.36 6.00 not a hash (just for reference)
sumhash32 43317.86 16.69 not a hash (just for reference)
FNV1a_YoshimitsuTRIAD 13000.49 24.96 poor (100% bias, collisions, distrib)
crc64_hw 7308.06 28.37 poor (insecure, 100% bias, collisions, distrib), non-portable (SSE4.2)
crc32_hw 5577.64 29.10 poor (insecure, 100% bias, collisions, distrib), non-portable (SSE4.2)
NOP_OAAT_read64 1991.31 30.46 poor (100% bias, 2.17x collisions)
Crap8 2743.80 32.50 poor (2.42% bias, collisions, 2% distrib)
t1ha_aes 34636.42 33.03 non-portable (AES-NI)
t1ha 12228.80 35.55
MUM 10246.20 37.25 non-portable (different result, machine specific)
Murmur2 2789.89 38.37 poor (1.7% bias, 81x coll, 1.7% distrib)
t1ha_32le 5958.54 38.54 alien (designed for 32-bit CPU)
t1ha_64be 9321.23 38.29 alien (designed for big-endian CPU)
lookup3 1817.11 39.30 poor (28% bias, collisions, 30% distrib)
t1ha_32be 5873.45 39.81 alien (designed for 32-bit big-endian CPU)
Murmur2C 3655.60 42.68 poor (91% bias, collisions, distrib)
fasthash64 5578.06 43.42
Murmur2A 2789.85 43.38 poor (12.7% bias)
xxHash32 5513.55 43.72
Murmur2B 5578.21 44.13 weak (1.8% bias, collisions, distrib)
fasthash32 5381.46 45.50
cmetrohash64_1_optshort 11808.92 46.33 seems weak (likely cyclic collisions)
metrohash64_2 12113.12 46.88 seems weak (likely cyclic collisions)
cmetrohash64_1 12081.32 47.28 seems weak (likely cyclic collisions)
metrohash64_1 12024.68 47.21 seems weak (likely cyclic collisions)
Murmur3F 5473.62 47.37
superfast 1860.25 47.45 poor (91% bias, 5273.01x collisions, 37% distrib)
cmetrohash64_2 12052.58 48.66
Murmur3A 2232.00 48.16
City32 5014.33 51.13 far to perfect (2 minor collisions)
City64 11041.72 51.77
metrohash64crc_2 20582.76 51.39 seems weak (likely cyclic collisions), non-portable (SSE4.2)
sumhash 9668.13 51.31 not a hash (just for reference)
metrohash64crc_1 21319.23 52.36 weak (cyclic collisions), non-portable (SSE4.2)
PMurHash32 2232.26 53.18
Murmur3C 3719.22 54.05
bernstein 921.43 55.17 poor (100% bias, collisions, distrib)
xxHash64 11123.15 56.17
Spooky32 11464.20 59.45
City128 12551.54 60.93
FarmHash64 12145.36 60.12 non-portable (SSE4.2)
Spooky128 11735.99 60.45 weak (collisions with 4bit diff)
Spooky64 11820.20 60.39
CityCrc128 14821.82 62.38 non-portable (SSE4.2)
MicroOAAT 826.32 62.06 poor (100% bias, distrib)
metrohash128_1 11063.78 66.58 seems weak (likely cyclic collisions)
metrohash128_2 11465.18 66.72 weak (cyclic collisions)
GoodOAAT 930.18 68.24
metrohash128crc_1 21322.80 70.33 seems weak (likely cyclic collisions), non-portable (SSE4.2)
metrohash128crc_2 20990.70 70.40 seems weak (likely cyclic collisions), non-portable (SSE4.2)
farmhash64_c 12033.13 71.30 non-portable (SSE4.2)
sdbm 695.29 71.76 poor (100% bias, collisions, distrib)
FNV1a 684.17 72.75 poor (zeros, 100% bias, collisions, distrib)
FNV64 697.67 72.70 poor (100% bias, collisions, distrib)
FarmHash128 12515.98 77.43 non-portable (SSE4.2)
hasshe2 2587.39 81.23 poor (insecure, 100% bias, collisions, distrib), non-portable (SSE2)
BadHash 558.14 87.87 not a hash (just for reference)
x17 551.99 89.24 poor (99.98% bias, collisions, distrib)
JenkinsOOAT_perl 558.14 95.26 poor (1.5-11.5% bias, 7.2x collisions)
farmhash128_c 12709.06 96.42 non-portable (SSE4.1)
MurmurOAAT 465.12 107.61 poor (collisions, 99.99% distrib)
JenkinsOOAT 558.13 116.75 poor (53.5% bias, collisions, distrib)
falkhash 8909.54 124.48 non-portable (AES-NI)
crc32 342.27 142.06 poor (insecure, 8589.93x collisions, distrib)
SipHash 962.35 147.36
md5_32a 433.03 508.98
sha1_32a 531.44 1222.44

The LARGE KEYS case

Order by hashing speed in Mi-bytes (2^20 = 1048576) per second for 262144-byte block (more is better).

Function MiB/Second Cycles/Hash Notes (quality, portability)
donothing 15747227.36 6.00 not a hash (just for reference)
sumhash32 43317.86 16.69 not a hash (just for reference)
t1ha_aes 34636.42 33.03 non-portable (AES-NI)
metrohash128crc_1 21322.80 70.33 seems weak (likely cyclic collisions), non-portable (SSE4.2)
metrohash64crc_1 21319.23 52.36 seems weak (cyclic collisions), non-portable (SSE4.2)
metrohash128crc_2 20990.70 70.40 seems weak (likely cyclic collisions), non-portable (SSE4.2)
metrohash64crc_2 20582.76 51.39 seems weak (likely cyclic collisions), non-portable (SSE4.2)
CityCrc128 14821.82 62.38 non-portable (SSE4.2)
FNV1a_YoshimitsuTRIAD 13000.49 24.96 poor (100% bias, collisions, distrib)
farmhash128_c 12709.06 96.42 non-portable (SSE4.1)
City128 12551.54 60.93
FarmHash128 12515.98 77.43 non-portable (SSE4.2)
t1ha 12228.80 35.55
FarmHash64 12145.36 60.12 non-portable (SSE4.2)
metrohash64_2 12113.12 46.88 seems weak (likely cyclic collisions)
cmetrohash64_1 12081.32 47.28 seems weak (likely cyclic collisions)
cmetrohash64_2 12052.58 48.66 seems weak (likely cyclic collisions)
farmhash64_c 12033.13 71.30 non-portable (SSE4.2)
metrohash64_1 12024.68 47.21 seems weak (likely cyclic collisions)
Spooky64 11820.20 60.39
cmetrohash64_1_optshort 11808.92 46.33 seems weak (likely cyclic collisions)
Spooky128 11735.99 60.45 weak (collisions with 4-bit diff)
metrohash128_2 11465.18 66.72 weak (cyclic collisions)
Spooky32 11464.20 59.45
xxHash64 11123.15 56.17
metrohash128_1 11063.78 66.58 seems weak (likely cyclic collisions)
City64 11041.72 51.77
MUM 10246.20 37.25 non-portable (different result, machine specific)
sumhash 9668.13 51.31 not a hash (just for reference)
t1ha_64be 9321.23 38.29 alien (designed for big-endian CPU)
falkhash 8909.54 124.48 non-portable (AES-NI)
crc64_hw 7308.06 28.37 poor (insecure, 100% bias, collisions, distrib), non-portable (SSE4.2)
t1ha_32le 5958.54 38.54 alien (designed for 32-bit CPU)
t1ha_32be 5873.45 39.81 alien (designed for 32-bit big-endian CPU)
fasthash64 5578.06 43.42
Murmur2B 5578.21 44.13 weak (1.8% bias, collisions, distrib)
crc32_hw 5577.64 29.10 poor (insecure, 100% bias, collisions, distrib), non-portable (SSE4.2)
xxHash32 5513.55 43.72
Murmur3F 5473.62 47.37
fasthash32 5381.46 45.50
City32 5014.33 51.13 far to perfect (2 minor collisions)
Murmur3C 3719.22 54.05
Murmur2C 3655.60 42.68 poor (91% bias, collisions, distrib)
Murmur2 2789.89 38.37 poor (1.7% bias, 81x coll, 1.7% distrib)
Murmur2A 2789.85 43.38 poor (12.7% bias)
Crap8 2743.80 32.50 poor (2.42% bias, collisions, 2% distrib)
hasshe2 2587.39 81.23 poor (insecure, 100% bias, collisions, distrib), non-portable (SSE2)
Murmur3A 2232.00 48.16
PMurHash32 2232.26 53.18
NOP_OAAT_read64 1991.31 30.46 poor (100% bias, 2.17x collisions)
superfast 1860.25 47.45 poor (91% bias, 5273.01x collisions, 37% distrib)
lookup3 1817.11 39.30 poor (28% bias, collisions, 30% distrib)
SipHash 962.35 147.36
GoodOAAT 930.18 68.24
bernstein 921.43 55.17 poor (100% bias, collisions, distrib)
MicroOAAT 826.32 62.06 poor (100% bias, distrib)
FNV64 697.67 72.70 poor (100% bias, collisions, distrib)
sdbm 695.29 71.76 poor (100% bias, collisions, distrib)
FNV1a 684.17 72.75 poor (zeros, 100% bias, collisions, distrib)
BadHash 558.14 87.87 not a hash (just for reference)
JenkinsOOAT 558.13 116.75 poor (53.5% bias, collisions, distrib)
JenkinsOOAT_perl 558.14 95.26 poor (1.5-11.5% bias, 7.2x collisions)
x17 551.99 89.24 poor (99.98% bias, collisions, distrib)
sha1_32a 531.44 1222.44
MurmurOAAT 465.12 107.61 poor (collisions, 99.99% distrib)
md5_32a 433.03 508.98
crc32 342.27 142.06 poor (insecure, 8589.93x collisions, distrib)

This is a mirror of the origin repository that was moved to abf.io because of discriminatory restrictions for Russian Crimea.

t1ha's People

Contributors

erthink avatar flier avatar lemenkov avatar mikkelfj avatar rurban avatar timgates42 avatar troosh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

t1ha's Issues

Could you improve t1ha for compile with Visual C++ 9.0?

i use python extension for some my projects flier/pyfasthash#8
your great t1ha hash function included into rurban/smhasher#30 package

and after i change target platoform to x64
t1ha.c can't compile under VC++ 9.0, but he required for Python2.7 compilation ;(

please, could you help me resolve following errors?

 t1ha.c
    It is recommended to use Visual Studio 2015 (MSC 19.0) or newer.
    src\smhasher\t1ha.c(368) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(368) : error C2146: syntax error : missing ';' before identifier 'w0'
    src\smhasher\t1ha.c(368) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(368) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
    src\smhasher\t1ha.c(369) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(369) : error C2146: syntax error : missing ';' before identifier 'w1'
    src\smhasher\t1ha.c(369) : error C2065: 'w1' : undeclared identifier
    src\smhasher\t1ha.c(369) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
    src\smhasher\t1ha.c(370) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(370) : error C2146: syntax error : missing ';' before identifier 'w2'
    src\smhasher\t1ha.c(370) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(370) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
    src\smhasher\t1ha.c(371) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(371) : error C2146: syntax error : missing ';' before identifier 'w3'
    src\smhasher\t1ha.c(371) : error C2065: 'w3' : undeclared identifier
    src\smhasher\t1ha.c(371) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
    src\smhasher\t1ha.c(373) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(373) : error C2146: syntax error : missing ';' before identifier 'd02'
    src\smhasher\t1ha.c(373) : error C2065: 'd02' : undeclared identifier
    src\smhasher\t1ha.c(373) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(373) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(373) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
    src\smhasher\t1ha.c(374) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(374) : error C2146: syntax error : missing ';' before identifier 'c13'
    src\smhasher\t1ha.c(374) : error C2065: 'c13' : undeclared identifier
    src\smhasher\t1ha.c(374) : error C2065: 'w1' : undeclared identifier
    src\smhasher\t1ha.c(374) : error C2065: 'w3' : undeclared identifier
    src\smhasher\t1ha.c(374) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
    src\smhasher\t1ha.c(375) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(376) : error C2065: 'w1' : undeclared identifier
    src\smhasher\t1ha.c(377) : error C2065: 'd02' : undeclared identifier
    src\smhasher\t1ha.c(377) : error C2065: 'w3' : undeclared identifier
    src\smhasher\t1ha.c(378) : error C2065: 'c13' : undeclared identifier
    src\smhasher\t1ha.c(378) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(387) : error C2143: syntax error : missing ';' before 'const'
    src\smhasher\t1ha.c(389) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(389) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(389) : warning C4022: 'memcpy' : pointer mismatch for actual parameter 2
    src\smhasher\t1ha.c(389) : warning C4047: '=' : 'int' differs in levels of indirection from 'const uint64_t *'
    src\smhasher\t1ha.c(393) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(393) : warning C4022: 'fetch64_le' : pointer mismatch for actual parameter 1
    src\smhasher\t1ha.c(402) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(402) : warning C4022: 'fetch64_le' : pointer mismatch for actual parameter 1
    src\smhasher\t1ha.c(411) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(411) : warning C4022: 'fetch64_le' : pointer mismatch for actual parameter 1
    src\smhasher\t1ha.c(420) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(420) : warning C4022: 'tail64_le' : pointer mismatch for actual parameter 1
    src\smhasher\t1ha.c(428) : error C2061: syntax error : identifier 'fetch64_be'
    src\smhasher\t1ha.c(428) : error C2059: syntax error : ';'
    src\smhasher\t1ha.c(428) : error C2059: syntax error : 'type'
    src\smhasher\t1ha.c(522) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(522) : error C2146: syntax error : missing ';' before identifier 'w0'
    src\smhasher\t1ha.c(522) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(522) : warning C4013: 'fetch64_be' undefined; assuming extern returning int
    src\smhasher\t1ha.c(523) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(523) : error C2146: syntax error : missing ';' before identifier 'w1'
    src\smhasher\t1ha.c(523) : error C2065: 'w1' : undeclared identifier
    src\smhasher\t1ha.c(524) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(524) : error C2146: syntax error : missing ';' before identifier 'w2'
    src\smhasher\t1ha.c(524) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(525) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(525) : error C2146: syntax error : missing ';' before identifier 'w3'
    src\smhasher\t1ha.c(525) : error C2065: 'w3' : undeclared identifier
    src\smhasher\t1ha.c(527) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(527) : error C2146: syntax error : missing ';' before identifier 'd02'
    src\smhasher\t1ha.c(527) : error C2065: 'd02' : undeclared identifier
    src\smhasher\t1ha.c(527) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(527) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(527) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
    src\smhasher\t1ha.c(528) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(528) : error C2146: syntax error : missing ';' before identifier 'c13'
    src\smhasher\t1ha.c(528) : error C2065: 'c13' : undeclared identifier
    src\smhasher\t1ha.c(528) : error C2065: 'w1' : undeclared identifier
    src\smhasher\t1ha.c(528) : error C2065: 'w3' : undeclared identifier
    src\smhasher\t1ha.c(528) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
    src\smhasher\t1ha.c(529) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(530) : error C2065: 'w1' : undeclared identifier
    src\smhasher\t1ha.c(531) : error C2065: 'd02' : undeclared identifier
    src\smhasher\t1ha.c(531) : error C2065: 'w3' : undeclared identifier
    src\smhasher\t1ha.c(532) : error C2065: 'c13' : undeclared identifier
    src\smhasher\t1ha.c(532) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(541) : error C2143: syntax error : missing ';' before 'const'
    src\smhasher\t1ha.c(543) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(543) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(543) : warning C4022: 'memcpy' : pointer mismatch for actual parameter 2
    src\smhasher\t1ha.c(543) : warning C4047: '=' : 'int' differs in levels of indirection from 'const uint64_t *'
    src\smhasher\t1ha.c(547) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(556) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(565) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(574) : error C2065: 'v' : undeclared identifier
    src\smhasher\t1ha.c(574) : warning C4022: 'tail64_be' : pointer mismatch for actual parameter 1
    src\smhasher\t1ha.c(582) : error C2061: syntax error : identifier 'tail32_le'
    src\smhasher\t1ha.c(582) : error C2059: syntax error : ';'
    src\smhasher\t1ha.c(582) : error C2059: syntax error : 'type'
    src\smhasher\t1ha.c(649) : error C2275: 'uint64_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
    src\smhasher\t1ha.c(649) : error C2146: syntax error : missing ';' before identifier 'l'
    src\smhasher\t1ha.c(649) : error C2065: 'l' : undeclared identifier
    src\smhasher\t1ha.c(650) : error C2065: 'l' : undeclared identifier
    src\smhasher\t1ha.c(650) : warning C4244: '*=' : conversion from 'const uint64_t' to 'int', possible loss of data
    src\smhasher\t1ha.c(651) : error C2065: 'l' : undeclared identifier
    src\smhasher\t1ha.c(651) : error C2065: 'l' : undeclared identifier
    src\smhasher\t1ha.c(651) : warning C4293: '>>' : shift count negative or too big, undefined behavior
    src\smhasher\t1ha.c(652) : error C2065: 'l' : undeclared identifier
    src\smhasher\t1ha.c(686) : error C2275: 'uint32_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
    src\smhasher\t1ha.c(686) : error C2146: syntax error : missing ';' before identifier 'w0'
    src\smhasher\t1ha.c(686) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(687) : error C2275: 'uint32_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
    src\smhasher\t1ha.c(687) : error C2146: syntax error : missing ';' before identifier 'w1'
    src\smhasher\t1ha.c(687) : error C2065: 'w1' : undeclared identifier
    src\smhasher\t1ha.c(688) : error C2275: 'uint32_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
    src\smhasher\t1ha.c(688) : error C2146: syntax error : missing ';' before identifier 'w2'
    src\smhasher\t1ha.c(688) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(689) : error C2275: 'uint32_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
    src\smhasher\t1ha.c(689) : error C2146: syntax error : missing ';' before identifier 'w3'
    src\smhasher\t1ha.c(689) : error C2065: 'w3' : undeclared identifier
    src\smhasher\t1ha.c(691) : error C2275: 'uint32_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
    src\smhasher\t1ha.c(691) : error C2146: syntax error : missing ';' before identifier 'c02'
    src\smhasher\t1ha.c(691) : error C2065: 'c02' : undeclared identifier
    src\smhasher\t1ha.c(691) : error C2065: 'w0' : undeclared identifier
    src\smhasher\t1ha.c(691) : error C2065: 'w2' : undeclared identifier
    src\smhasher\t1ha.c(692) : error C2275: 'uint32_t' : illegal use of this type as an expression
            D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
    src\smhasher\t1ha.c(692) : fatal error C1003: error count exceeds 100; stopping compilation
    error: command 'C:\\Users\\e.klimov\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe' failed with exit status 2

clang >= 3.9.1 build issues

Tried compiling latest master with clang, just for the hell of it.
Seems like it needs -maes -msse4 switches to function at all.

clang 3.5 (opensuse 13.2) just produces compilation errors

$clang --version
clang version 3.5.0 (tags/RELEASE_350/final 216961)
Target: x86_64-suse-linux
Thread model: posix

$ make CC=clang
CLANG: Version 3.5, ARCHx86: yes; Affected by 'instructions not enabled' bug: yes
clang -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -march=native -o test src/t1ha1.c src/t1ha0.c tests/main.c
In file included from src/t1ha0.c:316:
/usr/bin/../lib64/clang/3.5.0/include/smmintrin.h:28:2: error: "SSE4.1 instruction set not enabled"
#error "SSE4.1 instruction set not enabled"
 ^
In file included from src/t1ha0.c:317:
/usr/bin/../lib64/clang/3.5.0/include/wmmintrin.h:30:3: error: "AES/PCLMUL instructions not enabled"
# error "AES/PCLMUL instructions not enabled"
  ^
src/t1ha0.c:460:17: error: implicit declaration of function '_mm_aesenc_si128' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
    __m128i y = _mm_aesenc_si128(x, _mm_set_epi64x(p0, p1));
                ^
src/t1ha0.c:460:17: note: did you mean '_mm_and_si128'?
/usr/bin/../lib64/clang/3.5.0/include/emmintrin.h:805:1: note: '_mm_and_si128' declared here
_mm_and_si128(__m128i __a, __m128i __b)

adding -maes -msse4 helps

antoxa@antoxa-suse.:~/_Dev/t1ha> make CC="clang -maes -msse4"
CLANG: Version 3.5, ARCHx86: yes; Affected by 'instructions not enabled' bug: yes
clang -maes -msse4 -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -march=native -o test src/t1ha1.c src/t1ha0.c tests/main.c
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -march=native -c -o t1ha0.o src/t1ha0.c
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -march=native -c -o t1ha1.o src/t1ha1.c
ar rs libt1ha.a t1ha0.o t1ha1.o
ar: creating libt1ha.a
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -march=native -fvisibility=hidden -Dt1ha_EXPORTS -shared -s -o libt1ha.so src/t1ha1.c src/t1ha0.c
./test || rm -rf libt1ha.a libt1ha.so
Testing t1ha_64le... Ok
Testing t1ha_64be... Ok
Testing t1ha_32le... Ok
Testing t1ha_32be... Ok
Testing t1ha_ia32aes... Ok

clang 3.9 (fedora 25) just crashes (eh)

$ clang --version
clang version 3.9.1 (tags/RELEASE_391/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix

$ make CC=clang
CLANG: Version 3.9, ARCHx86: yes; Affected by 'instructions not enabled' bug: no
clang -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -o test src/t1ha1.c src/t1ha0.c tests/main.c
fatal error: error in backend: Cannot select: intrinsic %llvm.x86.aesni.aesenc
clang-3.9: error: clang frontend command failed with exit code 70 (use -v to see invocation)
clang version 3.9.1 (tags/RELEASE_391/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
clang-3.9: note: diagnostic msg: PLEASE submit a bug report to  and include the crash backtrace, preprocessed source, and associated run script.
clang-3.9: note: diagnostic msg:
********************

PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-3.9: note: diagnostic msg: /tmp/t1ha1-c2cbe9.c
clang-3.9: note: diagnostic msg: /tmp/t1ha0-b53109.c
clang-3.9: note: diagnostic msg: /tmp/main-1e8446.c
clang-3.9: note: diagnostic msg: /tmp/t1ha1-c2cbe9.sh
clang-3.9: note: diagnostic msg:

********************
Makefile:45: recipe for target 'test' failed
make: *** [test] Error 70

adding switches helps, but generates warnings

$ make CC="clang -maes -msse4"
CLANG: Version 3.9, ARCHx86: yes; Affected by 'instructions not enabled' bug: no
clang -maes -msse4 -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -o test src/t1ha1.c src/t1ha0.c tests/main.c
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -c -o t1ha0.o src/t1ha0.c
src/t1ha0.c:328:19: warning: unused function 't1ha_aes_resolve' [-Wunused-function]
static uint64_t (*t1ha_aes_resolve(void))(const void *, size_t, uint64_t) {
                  ^
1 warning generated.
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -c -o t1ha1.o src/t1ha1.c
ar rs libt1ha.a t1ha0.o t1ha1.o
ar: creating libt1ha.a
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -fvisibility=hidden -Dt1ha_EXPORTS -shared -s -o libt1ha.so src/t1ha1.c src/t1ha0.c
src/t1ha0.c:328:19: warning: unused function 't1ha_aes_resolve' [-Wunused-function]
static uint64_t (*t1ha_aes_resolve(void))(const void *, size_t, uint64_t) {
                  ^
1 warning generated.
./test || rm -rf libt1ha.a libt1ha.so
Testing t1ha_64le... Ok
Testing t1ha_64be... Ok
Testing t1ha_32le... Ok
Testing t1ha_32be... Ok
Testing t1ha_ia32aes... Ok

Whether should the t1ha source code be split into `basic` and `extras` parts?

Initially the t1ha implementation had be about of 100 lines of code.

Now it is about of 1000 lines, with few crutches for x86 code generation (e.g. for SSE4, AES and AVX).

So, should I split the source code, for instance into the parts:
t1ha-64le.c
t1ha-64be.c
t1ha-32le.c
t1ha-32be.c
t1ha-sse42.c
t1ha-aes.c
t1ha-aes-avx.c
t1ha-avx2.c
t1ha-local.c

Please vote.

valgrind 'invalid read of size 8' in t1ha1 and t1ha2

I stumbled across your nice hash function while looking for ways to improve map performance in my '8th' programming language. I have been using fnv1a successfully. When I plugged in your code, I get the following valgrind complaints:

==29732==    at 0x4CAF62: fetch64_le_unaligned (t1ha_bits.h:587)
==29732==    by 0x4CAF62: tail64_le_unaligned (t1ha_bits.h:684)
==29732==    by 0x4CAF62: t1ha1_le (t1ha1.c:134)

Similar complaint with the t1ha2 equivalent. This occurs any time your functions are called, the context in my code doesn't matter.

Critics of t1ha-crc hash

moved from rurban/smhasher#30

i don't recommend to use this crc-based hash (or any automatic selection of fastest hash as it's currently implemented in this package). while basic t1ha hashes (w/o crc/aes) look reasonable, crc-based hash is absolutely non-sense, made only for speed records rather than real work. in short, it computes 32-bit hash of each 64-bit input word, and then produces 64-bit hash based on these stripped numbers

@leo-yuriev, it will be great if you will finally remove this pseudo-hash from your package. Passing SMHasher doesn't guarantee that your hash is strong, you need also to understand how to change smhasher to make your hash fail. Your idea that users should understand CRC math and research each hash they are using yourself, is absolutely laughable - real users are much less educated even than you. And finally, idea to mix weak crc hash with reasonable main t1ha hash with automatic hash selection based solely on speed, deceives these users by pretending them that they are using reasomnnable main hash while automatic selection actually chooses crc hash in most cases. You are started with good idea, but then lost sense trying to setup meaningless records and relying on automatic smhasher testing rather than understanding of your own algorithms.

t1ha-crc hash is weaker than xxhash and so on. essentially, it's a crc32 hash arttifically widened to 64 bits, and you know that even crc32 can't pass half of smhasher tests. t1ha-crc weakness can be detected by the various smhasher tests checking 40-64 bit keys IF these keys will be preceded by fixed 40-byte header

Notes about t1ha_v3 design

  1. Dirty and Fair kinds:
    • t1ha3_dirty = favour speed over quality: non-uniform, all hardware tricks (ie. AES-NI), one injection point and blinding multiplication are allowed;
    • t1ha3_fair = favour quality over speed: two injection points, no known drawbacks.
  2. Goals:
    • t1ha3_dirty: should be faster than competitors (e.g. xxHash_v3, MeowHash and wyhash_v777), while not inferior in quality; Or at least the same speed with better quality (i.e. no several drawbacks).
    • t1ha3_fair: proven excellence in the quality of competitors, with minimal performance degradation.
    • target architectures in order of priority: E2K, AMD64, AARCH64, RISC-V, MIPS64, MIPS32, ARM (no Thumb, but Thumb-2).
    • optional: additional variations of speed/quality/security tradeoffs.
  3. Endianness: No explicit LE/BE versions:
    • t1ha3_dirty: the native byte order is always used;
    • t1ha3_fair: Little Endian byte order is primary, bswap on BE platforms.
  4. Bitness:
    • result: 32, 64, 128.
    • implementations: 64, 32.
  5. Insights:
    • avoid MUM-mixer;
    • single header-only C99 implementation;
    • avoid unaligned read;
    • avoid unsafe user options;
    • SIMD-frendly;
    • non-injective mixers.

Schedule:

  • Most likely, only the final version will be published when there is no doubt about achieving the goals.
  • No specific deadlines are publicly declared. Sponsor the development if you need any guarantees.

Streaming API does not match "atonce" API.

Hi.
Is it known (by design) what streaming API returns different value for the same data compared to "atonce" API?
I am testing it with next code.

void my_test(void) {
    t1ha_context_t ctx;
    uint64_t seed = 42;
    char key[] = "qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq";
    size_t key_len = sizeof(key) - 1;
    t1ha2_init(&ctx, seed, key_len);
    t1ha2_update(&ctx, key, key_len);
    uint64_t v1 = t1ha2_final(&ctx, NULL);
    uint64_t v2 = t1ha2_atonce(key, key_len, seed);
    printf("v1: %" PRIu64 "\n", v1);
    printf("v2: %" PRIu64 "\n", v2);
}

Code execution results in:

v1: 15442592273487182410
v2: 8257225483948642673

SIGBUS due to unaligned access on armv7 (GCC 9)

(Sort of a duplicate of #30, sorry for that, I was unable to re-open that issue).

Basically armv7 unaligned access means that it's possible to read and write 16- and 32-bit values from addresses that are not 16- or 32-bit aligned (more details). However (as far as I understand) t1ha uses unaligned 64-bit access, and that requires 8-byte alignment (or should be replaced with two 32-bit reads/writes).

Also nothing guarantees that unaligned access is efficient.

Outperform wyhash

Currently wyhash outperforms t1ha.
Nevertheless I think the next version of t1ha will be even faster.

PERFORMANCE COMPARISON (this is not a bug or an issue)

Please note: the benchmarking results depend on the processor and compiler model.

...
model		: 158
model name	: Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
stepping	: 9
microcode	: 0x8e

clang version 6.0.1 (tags/RELEASE_601/final)

Preparing to benchmarking...
 - suggest enable rdpmc for usermode (echo 2 | sudo tee /sys/devices/cpu/rdpmc)
 - running on CPU#4
 - use RDPMC_perf as clock source for benchmarking
 - assume it cheap and stable
 - measure granularity and overhead: 53 cycle, 0.0188679 iteration/cycle

Bench for tiny keys (7 bytes):
t1ha2_atonce            :     12.117 cycle/hash,  1.731 cycle/byte,  0.578 byte/cycle,  1.733 Gb/s @3GHz 
t1ha2_atonce128*        :     29.656 cycle/hash,  4.237 cycle/byte,  0.236 byte/cycle,  0.708 Gb/s @3GHz 
t1ha2_stream*           :     77.625 cycle/hash, 11.089 cycle/byte,  0.090 byte/cycle,  0.271 Gb/s @3GHz 
t1ha2_stream128*        :     97.625 cycle/hash, 13.946 cycle/byte,  0.072 byte/cycle,  0.215 Gb/s @3GHz 
t1ha1_64le              :     12.219 cycle/hash,  1.746 cycle/byte,  0.573 byte/cycle,  1.719 Gb/s @3GHz 
t1ha0                   :     14.070 cycle/hash,  2.010 cycle/byte,  0.498 byte/cycle,  1.493 Gb/s @3GHz 
xxhash64                :     21.234 cycle/hash,  3.033 cycle/byte,  0.330 byte/cycle,  0.989 Gb/s @3GHz 
StadtX                  :     20.219 cycle/hash,  2.888 cycle/byte,  0.346 byte/cycle,  1.039 Gb/s @3GHz 
HighwayHash64_portable  :    504.750 cycle/hash, 72.107 cycle/byte,  0.014 byte/cycle,  0.042 Gb/s @3GHz 
HighwayHash64_sse41     :     81.938 cycle/hash, 11.705 cycle/byte,  0.085 byte/cycle,  0.256 Gb/s @3GHz 
HighwayHash64_avx2      :     51.812 cycle/hash,  7.402 cycle/byte,  0.135 byte/cycle,  0.405 Gb/s @3GHz 

gcc version 8.1.1 20180712 (Red Hat 8.1.1-5) (GCC)

Preparing to benchmarking...
 - suggest enable rdpmc for usermode (echo 2 | sudo tee /sys/devices/cpu/rdpmc)
 - running on CPU#2
 - use RDPMC_perf as clock source for benchmarking
 - assume it cheap and stable
 - measure granularity and overhead: 54 cycle, 0.0185185 iteration/cycle

...

Bench for large keys (16384 bytes):
t1ha2_atonce            :   3461.000 cycle/hash,  0.211 cycle/byte,  4.734 byte/cycle, 14.202 Gb/s @3GHz 
t1ha2_atonce128*        :   3438.000 cycle/hash,  0.210 cycle/byte,  4.766 byte/cycle, 14.297 Gb/s @3GHz 
t1ha2_stream*           :   3471.000 cycle/hash,  0.212 cycle/byte,  4.720 byte/cycle, 14.161 Gb/s @3GHz 
t1ha2_stream128*        :   3484.000 cycle/hash,  0.213 cycle/byte,  4.703 byte/cycle, 14.108 Gb/s @3GHz 
t1ha1_64le              :   3431.000 cycle/hash,  0.209 cycle/byte,  4.775 byte/cycle, 14.326 Gb/s @3GHz 
t1ha0                   :   1187.000 cycle/hash,  0.072 cycle/byte, 13.803 byte/cycle, 41.409 Gb/s @3GHz 
xxhash64                :   4120.000 cycle/hash,  0.251 cycle/byte,  3.977 byte/cycle, 11.930 Gb/s @3GHz 
StadtX                  :   3501.000 cycle/hash,  0.214 cycle/byte,  4.680 byte/cycle, 14.039 Gb/s @3GHz 
HighwayHash64_portable  :  41965.000 cycle/hash,  2.561 cycle/byte,  0.390 byte/cycle,  1.171 Gb/s @3GHz 
HighwayHash64_sse41     :   6222.000 cycle/hash,  0.380 cycle/byte,  2.633 byte/cycle,  7.900 Gb/s @3GHz 
HighwayHash64_avx2      :   4601.000 cycle/hash,  0.281 cycle/byte,  3.561 byte/cycle, 10.683 Gb/s @3GHz 

Проверка NIST тестом.

Здравствуйте, хочу сообщить о своих провеках хэш функции t1ha2.

Код:

#include <stdio.h>
#include "t1ha.h"

int main(void)
{
  uint64_t seed = 1234;
  /*
  FILE *fp;
  fp = fopen("/dev/urandom", "r");
  fread((void *)&seed, 1, 8, fp);
  fclose(fp);
  */
  
  for (uint64_t i = 0; i < 100000; i++)
  {
     uint64_t out_hash = t1ha2_atonce((void *)&i, 8, seed);
     fwrite((void *)&out_hash, 8, 1, stdout);
  }
  return 0;
}

Вывод программы сохраняется в файл my_test > randdata. Компилировалось и запускалось на x86-64 архитектуре, если это важно.
Таким образом я делаю ГПСЧ из хеш-функции. Так вот, такой ГПСЧ проваливает тест NIST. Я взял https://github.com/dj-on-github/sp800_22_tests и запустил это таким образом:
/sp800_22_tests-master/sp800_22_tests.py -t random_excursion_variant_test rand_data_test
Результат:

Tests of Distinguishability from Random
TEST: random_excursion_variant_test
J= 217
x = -9	 count=178	p = 0.321056 
x = -8	 count=201	p = 0.140221 
x = -7	 count=217	p = 0.000000  Not Random
x = -6	 count=224	p = 0.071638 
x = -5	 count=196	p = 0.237595 
x = -4	 count=180	p = 0.474671 
x = -3	 count=172	p = 0.683074 
x = -2	 count=184	p = 0.646686 
x = -1	 count=204	p = 0.441249 
x = 1	 count=217	p = 0.000000  Not Random
x = 2	 count=200	p = 0.333141 
x = 3	 count=192	p = 0.379485 
x = 4	 count=186	p = 0.397697 
x = 5	 count=183	p = 0.384678 
x = 6	 count=215	p = 0.020468 
x = 7	 count=247	p = 0.282416 
x = 8	 count=252	p = 0.306734 
x = 9	 count=246	p = 0.238734 
J too small (J=217 < 500) for result to be reliable
FAIL
P=0.321055625331
P=0.14022146188
P=0.0
P=0.0716377331396
P=0.237595481656
P=0.474671155356
P=0.683073833309
P=0.646685986377
P=0.441248751646
P=0.0
P=0.33314126571
P=0.379485462949
P=0.397697454488
P=0.384678398871
P=0.0204679237542
P=0.282416272064
P=0.306734447862
P=0.238733670118

Тест провален. Может быть эти требования не стоит предьявлять к данной хеш функции, но я просто собщаю.

Enable T1HA0_AESNI_AVAILABLE for AES Supported ARM64 Processor

I am checking into building the pyfasthash package for arm64 and found that it needs to enable T1HA0_AESNI_AVAILABLE for arm64 to build successfully. As already stated in this PR we cannot enable this for arm64 as AES-NI functionalities are not available for all arm64 supported processors.

As per this, the ARMV8-A processor supports AES instruction set. So, I think we can enable the T1HA0_AESNI_AVAILABLE flag for ARMV8-A processors.

If agreed, we can reopen this PR and amend the changes and enable the support for ARMV8-A processors.

Please share your opinion.

Bus error on ARMv7 with Clang 8

Clang erroneously produces ldrd instructions with t1ha with the unaligned code, and it causes bus errors.

ARM can do unaligned 64-bit reads with 2 ldr instructions, but ldrd requires an 8-byte aligned address.

More details hopefully to follow, but right now I am having difficulty with gdb ironically segfaulting.

Can't compile on virtualized FreeBSD 12.1

When I try to compile t1ha package in my FreeBSD 12.1 (i386) virtualized by VirtualBox, I got some errors in mera.c:

$ make
cc -Wextra -Werror  -std=c99 -O3 -DNDEBUG -D_DEFAULT_SOURCE -fno-stack-protector -mtune=native -save-temps -c -o mera.o tests/mera.c
tests/mera.c:234:7: error: no member named 'sa_sigaction' in 'struct sigaction'; did you mean
      '__sigaction_u'?
  act.sa_sigaction = sigaction_handler;
      ^~~~~~~~~~~~
      __sigaction_u
/usr/include/sys/signal.h:370:4: note: '__sigaction_u' declared here
 } __sigaction_u;
   ^
tests/mera.c:234:20: error: assigning to 'union (anonymous union at
      /usr/include/sys/signal.h:367:2)' from incompatible type 'void (int, siginfo_t *, void *)'
      (aka 'void (int, struct __siginfo *, void *)')
  act.sa_sigaction = sigaction_handler;
                   ^ ~~~~~~~~~~~~~~~~~
tests/mera.c:444:7: error: implicit declaration of function 'gettimeofday' is invalid in C99
      [-Werror,-Wimplicit-function-declaration]
  if (gettimeofday(&tv, ((void *)0))) {
      ^
3 errors generated.
make: *** [Makefile:117: mera.o] Error 1

Now, I have solved those errors simply by putting in mera.c:

#ifndef _XOPEN_SOURCE
#define _XOPEN_SOURCE 700
#endif

I had also an error, always in mera.c that I have solved removing usleep and using nanosleep instead.
After fixing those 2 issues with the #define and with nanosleep, I get this:

$ make
cc -Wextra -Werror  -std=c99 -O3 -DNDEBUG -D_DEFAULT_SOURCE -fno-stack-protector -mtune=native -save-temps -c -o mera.o tests/mera.c
error: inline assembly requires more registers than available at line 1412165
make: *** [Makefile:117: mera.o] Error 1

It is a very strange error, and probably it i a clang bug, or something like that. A friend of mine who compiled t1ha package with my those 2 fixes, had no problem.

Is this worth investigating more?

__ImageBase in t1ha-static

Hello,

how to disable this __ImageBase variable? This variable creates a program with broken memory. Manual replacement of this variable through GetModuleHandle() does not help either.

LEA RDX, QWORD PTR DS:[<__ImageBase>]
MOVZX EAX, BYTE PTR DS:[RDX + R10 + 0x1388]
MOV ECX, DWORD PTR DS:[RDX + RAX * 4 + 0x1378]
ADD RCX, RDX
JMP RCX

Error A2102: Symbol not defined : __ImageBase

Regards

inline functions don't follow feature restrections enforced by clang

In file included from external/t1ha/src/t1ha0_ia32aes_avx.c:2:
external/t1ha/src/t1ha0_ia32aes_a.h:54:17: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
    __m128i y = _mm_aesenc_si128(x, _mm_set_epi64x(prime_5, prime_6));
                ^
external/t1ha/src/t1ha0_ia32aes_a.h:70:21: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      __m128i v0y = _mm_aesenc_si128(v0, y);
                    ^
external/t1ha/src/t1ha0_ia32aes_a.h:71:22: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      __m128i v2x6 = _mm_aesenc_si128(v2, _mm_xor_si128(x, v6));
                     ^
external/t1ha/src/t1ha0_ia32aes_a.h:73:25: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
          _mm_xor_si128(_mm_aesenc_si128(v4, v5), _mm_add_epi64(v6, v7));
                        ^
external/t1ha/src/t1ha0_ia32aes_a.h:75:24: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      __m128i v0y7_1 = _mm_aesdec_si128(_mm_sub_epi64(v7, v0y), v1);
                       ^
external/t1ha/src/t1ha0_ia32aes_a.h:76:24: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      __m128i v2x6_3 = _mm_aesenc_si128(v2x6, v3);
                       ^
external/t1ha/src/t1ha0_ia32aes_a.h:78:11: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      x = _mm_aesenc_si128(v45_67, _mm_add_epi64(x, y));
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:79:11: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      y = _mm_aesenc_si128(v2x6_3, _mm_xor_si128(v0y7_1, v5));
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:86:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      x = _mm_aesdec_si128(x, v0y);
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:87:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      y = _mm_aesdec_si128(y, v1x);
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:91:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      x = _mm_aesdec_si128(x, v2y);
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:92:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      y = _mm_aesdec_si128(y, v3x);
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:98:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      x = _mm_aesdec_si128(x, v0y);
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:99:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      y = _mm_aesdec_si128(y, v1x);
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:104:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
      x = _mm_aesdec_si128(x, _mm_loadu_si128(v++));
          ^
external/t1ha/src/t1ha0_ia32aes_a.h:107:23: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
    x = _mm_add_epi64(_mm_aesdec_si128(x, _mm_aesenc_si128(y, x)), y);
                      ^
external/t1ha/src/t1ha0_ia32aes_a.h:107:43: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
    x = _mm_add_epi64(_mm_aesdec_si128(x, _mm_aesenc_si128(y, x)), y);
                                          ^
> clang --version
Apple LLVM version 9.1.0 (clang-902.0.39.1)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Работа со строками

Как работать со строками?

Можете привести пример для хеширования строки?

read_unaligned error

Code

#include <iostream>
#include "windows.h"
#include "t1ha.h"
#include <basetsd.h>
#include <concrt.h>

#pragma comment(lib, "t1ha-static.lib")

int main()
{
	UINT64 Hash = 0;
	int regnumber = 1;
	Hash = t1ha0((const void*)regnumber, (size_t)regnumber, 0x1EE7C0DE);
	system("pause");
}

Exception
s

Licensing

@leo-yuriev
I want to port t1ha to another language (Nim), MIT-licensed.
I would like to clarify the compatibility of licenses.

Proxy for eBay payment

For convenience I decided to buy a pinebook (ARM64 CPU which is important for me). Actually eBay (unexpectedly for me) accepts only PayPal for this order, but not Visa/MasterCard.
PayPal still pending its own "verification" procedure without any reasonable information (except "We will get back to you within 30 business days").

So I can't pay for my purchase even I have enough funds (
It's anecdotal, but I don't know what to do for now.

I hope that the seller of the lot will be able to wait for some more reasonable time (until paypal finishes verification) or someone will offer a solution.

Варианты t1ha, переименования для ясности и порядка

Ради внесения ясности и поддержания порядка текущие t1ha-функции будут переименованы по схеме "t1ha#", где # - цифра обозначающая вариант:

0 = 64 бита, "лишь-бы быстрее" (текущая t1ha_local).
1 = 64 бита, текущая t1ha (в том числа для big-endian).

Одновременно в t1ha0() будут спрятаны все текущие суррогаты, включая 32-битные вариации и AES-NI.

Также (постепенно) будут добавлены новые варианты:
2 = 64 бита, для отпечатков (быстрая, "инъективная", не стойкая).
3 = 128 бит, для отпечатков (быстрая, не стойкая).
4 = 128 бит, быстро-стойкая (целевое применение как у SipHash).
5 = 256 бит, для отпечатков (быстрая, не стойкая).
6 = 256 бит, не быстрая, стойкая (примерно как Argon2 и т.п.).
7 = 256 бит, стойкая, не ускоряемая (для паролей, многораундовая, с широкими умножениями и вариативными циклическими сдвигами).

fixed GH En description a little

The Future will be Positive.
Briefly, it is a portable 64-bit hash function:

Intended for 64-bit little-endian platforms, predominantly for x86_64, but portable and without penalties it can run on any 64-bit CPU.
In most cases up to 15% faster than City64, xxHash, mum-hash, metro-hash and all others portable hash-functions (which do not use specific hardware tricks).

Comparison with xxHash

If the primary purpose of this project is speed, why not xxHash, which is significantly faster than CityHash and has the same quality according to SMHasher?

does t1ha support combine, like crc32?

Привет, если не против, напишу на русском:
У CRC32 есть combine, который позволяет сократить объем вычислений, если изменился не весть датасет, а известный его кусок.
xxHash это не поддерживает.
Скажите, поддерживает ли t1ha?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.