erthink / t1ha Goto Github PK
View Code? Open in Web Editor NEWOne of the fastest hash functions
Home Page: https://www.ptsecurity.com
License: Other
t1ha's Issues
Could you improve t1ha for compile with Visual C++ 9.0?
i use python extension for some my projects flier/pyfasthash#8
your great t1ha hash function included into rurban/smhasher#30 package
and after i change target platoform to x64
t1ha.c can't compile under VC++ 9.0, but he required for Python2.7 compilation ;(
please, could you help me resolve following errors?
t1ha.c
It is recommended to use Visual Studio 2015 (MSC 19.0) or newer.
src\smhasher\t1ha.c(368) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(368) : error C2146: syntax error : missing ';' before identifier 'w0'
src\smhasher\t1ha.c(368) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(368) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
src\smhasher\t1ha.c(369) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(369) : error C2146: syntax error : missing ';' before identifier 'w1'
src\smhasher\t1ha.c(369) : error C2065: 'w1' : undeclared identifier
src\smhasher\t1ha.c(369) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
src\smhasher\t1ha.c(370) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(370) : error C2146: syntax error : missing ';' before identifier 'w2'
src\smhasher\t1ha.c(370) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(370) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
src\smhasher\t1ha.c(371) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(371) : error C2146: syntax error : missing ';' before identifier 'w3'
src\smhasher\t1ha.c(371) : error C2065: 'w3' : undeclared identifier
src\smhasher\t1ha.c(371) : warning C4244: '=' : conversion from 'uint64_t' to 'int', possible loss of data
src\smhasher\t1ha.c(373) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(373) : error C2146: syntax error : missing ';' before identifier 'd02'
src\smhasher\t1ha.c(373) : error C2065: 'd02' : undeclared identifier
src\smhasher\t1ha.c(373) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(373) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(373) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
src\smhasher\t1ha.c(374) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(374) : error C2146: syntax error : missing ';' before identifier 'c13'
src\smhasher\t1ha.c(374) : error C2065: 'c13' : undeclared identifier
src\smhasher\t1ha.c(374) : error C2065: 'w1' : undeclared identifier
src\smhasher\t1ha.c(374) : error C2065: 'w3' : undeclared identifier
src\smhasher\t1ha.c(374) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
src\smhasher\t1ha.c(375) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(376) : error C2065: 'w1' : undeclared identifier
src\smhasher\t1ha.c(377) : error C2065: 'd02' : undeclared identifier
src\smhasher\t1ha.c(377) : error C2065: 'w3' : undeclared identifier
src\smhasher\t1ha.c(378) : error C2065: 'c13' : undeclared identifier
src\smhasher\t1ha.c(378) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(387) : error C2143: syntax error : missing ';' before 'const'
src\smhasher\t1ha.c(389) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(389) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(389) : warning C4022: 'memcpy' : pointer mismatch for actual parameter 2
src\smhasher\t1ha.c(389) : warning C4047: '=' : 'int' differs in levels of indirection from 'const uint64_t *'
src\smhasher\t1ha.c(393) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(393) : warning C4022: 'fetch64_le' : pointer mismatch for actual parameter 1
src\smhasher\t1ha.c(402) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(402) : warning C4022: 'fetch64_le' : pointer mismatch for actual parameter 1
src\smhasher\t1ha.c(411) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(411) : warning C4022: 'fetch64_le' : pointer mismatch for actual parameter 1
src\smhasher\t1ha.c(420) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(420) : warning C4022: 'tail64_le' : pointer mismatch for actual parameter 1
src\smhasher\t1ha.c(428) : error C2061: syntax error : identifier 'fetch64_be'
src\smhasher\t1ha.c(428) : error C2059: syntax error : ';'
src\smhasher\t1ha.c(428) : error C2059: syntax error : 'type'
src\smhasher\t1ha.c(522) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(522) : error C2146: syntax error : missing ';' before identifier 'w0'
src\smhasher\t1ha.c(522) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(522) : warning C4013: 'fetch64_be' undefined; assuming extern returning int
src\smhasher\t1ha.c(523) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(523) : error C2146: syntax error : missing ';' before identifier 'w1'
src\smhasher\t1ha.c(523) : error C2065: 'w1' : undeclared identifier
src\smhasher\t1ha.c(524) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(524) : error C2146: syntax error : missing ';' before identifier 'w2'
src\smhasher\t1ha.c(524) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(525) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(525) : error C2146: syntax error : missing ';' before identifier 'w3'
src\smhasher\t1ha.c(525) : error C2065: 'w3' : undeclared identifier
src\smhasher\t1ha.c(527) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(527) : error C2146: syntax error : missing ';' before identifier 'd02'
src\smhasher\t1ha.c(527) : error C2065: 'd02' : undeclared identifier
src\smhasher\t1ha.c(527) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(527) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(527) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
src\smhasher\t1ha.c(528) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(528) : error C2146: syntax error : missing ';' before identifier 'c13'
src\smhasher\t1ha.c(528) : error C2065: 'c13' : undeclared identifier
src\smhasher\t1ha.c(528) : error C2065: 'w1' : undeclared identifier
src\smhasher\t1ha.c(528) : error C2065: 'w3' : undeclared identifier
src\smhasher\t1ha.c(528) : warning C4244: '=' : conversion from 'unsigned __int64' to 'int', possible loss of data
src\smhasher\t1ha.c(529) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(530) : error C2065: 'w1' : undeclared identifier
src\smhasher\t1ha.c(531) : error C2065: 'd02' : undeclared identifier
src\smhasher\t1ha.c(531) : error C2065: 'w3' : undeclared identifier
src\smhasher\t1ha.c(532) : error C2065: 'c13' : undeclared identifier
src\smhasher\t1ha.c(532) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(541) : error C2143: syntax error : missing ';' before 'const'
src\smhasher\t1ha.c(543) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(543) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(543) : warning C4022: 'memcpy' : pointer mismatch for actual parameter 2
src\smhasher\t1ha.c(543) : warning C4047: '=' : 'int' differs in levels of indirection from 'const uint64_t *'
src\smhasher\t1ha.c(547) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(556) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(565) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(574) : error C2065: 'v' : undeclared identifier
src\smhasher\t1ha.c(574) : warning C4022: 'tail64_be' : pointer mismatch for actual parameter 1
src\smhasher\t1ha.c(582) : error C2061: syntax error : identifier 'tail32_le'
src\smhasher\t1ha.c(582) : error C2059: syntax error : ';'
src\smhasher\t1ha.c(582) : error C2059: syntax error : 'type'
src\smhasher\t1ha.c(649) : error C2275: 'uint64_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(15) : see declaration of 'uint64_t'
src\smhasher\t1ha.c(649) : error C2146: syntax error : missing ';' before identifier 'l'
src\smhasher\t1ha.c(649) : error C2065: 'l' : undeclared identifier
src\smhasher\t1ha.c(650) : error C2065: 'l' : undeclared identifier
src\smhasher\t1ha.c(650) : warning C4244: '*=' : conversion from 'const uint64_t' to 'int', possible loss of data
src\smhasher\t1ha.c(651) : error C2065: 'l' : undeclared identifier
src\smhasher\t1ha.c(651) : error C2065: 'l' : undeclared identifier
src\smhasher\t1ha.c(651) : warning C4293: '>>' : shift count negative or too big, undefined behavior
src\smhasher\t1ha.c(652) : error C2065: 'l' : undeclared identifier
src\smhasher\t1ha.c(686) : error C2275: 'uint32_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
src\smhasher\t1ha.c(686) : error C2146: syntax error : missing ';' before identifier 'w0'
src\smhasher\t1ha.c(686) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(687) : error C2275: 'uint32_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
src\smhasher\t1ha.c(687) : error C2146: syntax error : missing ';' before identifier 'w1'
src\smhasher\t1ha.c(687) : error C2065: 'w1' : undeclared identifier
src\smhasher\t1ha.c(688) : error C2275: 'uint32_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
src\smhasher\t1ha.c(688) : error C2146: syntax error : missing ';' before identifier 'w2'
src\smhasher\t1ha.c(688) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(689) : error C2275: 'uint32_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
src\smhasher\t1ha.c(689) : error C2146: syntax error : missing ';' before identifier 'w3'
src\smhasher\t1ha.c(689) : error C2065: 'w3' : undeclared identifier
src\smhasher\t1ha.c(691) : error C2275: 'uint32_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
src\smhasher\t1ha.c(691) : error C2146: syntax error : missing ';' before identifier 'c02'
src\smhasher\t1ha.c(691) : error C2065: 'c02' : undeclared identifier
src\smhasher\t1ha.c(691) : error C2065: 'w0' : undeclared identifier
src\smhasher\t1ha.c(691) : error C2065: 'w2' : undeclared identifier
src\smhasher\t1ha.c(692) : error C2275: 'uint32_t' : illegal use of this type as an expression
D:\Portable\Python27\include\stdint.h(14) : see declaration of 'uint32_t'
src\smhasher\t1ha.c(692) : fatal error C1003: error count exceeds 100; stopping compilation
error: command 'C:\\Users\\e.klimov\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe' failed with exit status 2
[REJECTED] Бинарник для Windows
Леонид, в разделе релизов одни исходники, хорошо бы скомпилировать .exe под Win7x64.
Provide packages for some Linux distros
TBD
Enable T1HA0_AESNI_AVAILABLE for AES Supported ARM64 Processor
I am checking into building the pyfasthash package for arm64 and found that it needs to enable T1HA0_AESNI_AVAILABLE for arm64 to build successfully. As already stated in this PR we cannot enable this for arm64 as AES-NI functionalities are not available for all arm64 supported processors.
As per this, the ARMV8-A processor supports AES instruction set. So, I think we can enable the T1HA0_AESNI_AVAILABLE flag for ARMV8-A processors.
If agreed, we can reopen this PR and amend the changes and enable the support for ARMV8-A processors.
Please share your opinion.
Bus error on ARMv7 with Clang 8
Clang erroneously produces ldrd instructions with t1ha with the unaligned code, and it causes bus errors.
ARM can do unaligned 64-bit reads with 2 ldr instructions, but ldrd requires an 8-byte aligned address.
More details hopefully to follow, but right now I am having difficulty with gdb ironically segfaulting.
Whether should the t1ha source code be split into `basic` and `extras` parts?
Initially the t1ha implementation had be about of 100 lines of code.
Now it is about of 1000 lines, with few crutches for x86 code generation (e.g. for SSE4, AES and AVX).
So, should I split the source code, for instance into the parts:
t1ha-64le.c
t1ha-64be.c
t1ha-32le.c
t1ha-32be.c
t1ha-sse42.c
t1ha-aes.c
t1ha-aes-avx.c
t1ha-avx2.c
t1ha-local.c
Please vote.
Some strange issue with t1ha2_update
Could you please take a look at rspamd/rspamd#2222 ? I cannot reproduce it locally but you might know it better. I am personally a bit worried that you alias restrict pointer in loop function but it might not be the case.
clang >= 3.9.1 build issues
Tried compiling latest master with clang, just for the hell of it.
Seems like it needs -maes -msse4 switches to function at all.
clang 3.5 (opensuse 13.2) just produces compilation errors
$clang --version
clang version 3.5.0 (tags/RELEASE_350/final 216961)
Target: x86_64-suse-linux
Thread model: posix
$ make CC=clang
CLANG: Version 3.5, ARCHx86: yes; Affected by 'instructions not enabled' bug: yes
clang -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -march=native -o test src/t1ha1.c src/t1ha0.c tests/main.c
In file included from src/t1ha0.c:316:
/usr/bin/../lib64/clang/3.5.0/include/smmintrin.h:28:2: error: "SSE4.1 instruction set not enabled"
#error "SSE4.1 instruction set not enabled"
^
In file included from src/t1ha0.c:317:
/usr/bin/../lib64/clang/3.5.0/include/wmmintrin.h:30:3: error: "AES/PCLMUL instructions not enabled"
# error "AES/PCLMUL instructions not enabled"
^
src/t1ha0.c:460:17: error: implicit declaration of function '_mm_aesenc_si128' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
__m128i y = _mm_aesenc_si128(x, _mm_set_epi64x(p0, p1));
^
src/t1ha0.c:460:17: note: did you mean '_mm_and_si128'?
/usr/bin/../lib64/clang/3.5.0/include/emmintrin.h:805:1: note: '_mm_and_si128' declared here
_mm_and_si128(__m128i __a, __m128i __b)adding -maes -msse4 helps
antoxa@antoxa-suse.:~/_Dev/t1ha> make CC="clang -maes -msse4"
CLANG: Version 3.5, ARCHx86: yes; Affected by 'instructions not enabled' bug: yes
clang -maes -msse4 -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -march=native -o test src/t1ha1.c src/t1ha0.c tests/main.c
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -march=native -c -o t1ha0.o src/t1ha0.c
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -march=native -c -o t1ha1.o src/t1ha1.c
ar rs libt1ha.a t1ha0.o t1ha1.o
ar: creating libt1ha.a
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -march=native -fvisibility=hidden -Dt1ha_EXPORTS -shared -s -o libt1ha.so src/t1ha1.c src/t1ha0.c
./test || rm -rf libt1ha.a libt1ha.so
Testing t1ha_64le... Ok
Testing t1ha_64be... Ok
Testing t1ha_32le... Ok
Testing t1ha_32be... Ok
Testing t1ha_ia32aes... Okclang 3.9 (fedora 25) just crashes (eh)
$ clang --version
clang version 3.9.1 (tags/RELEASE_391/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
$ make CC=clang
CLANG: Version 3.9, ARCHx86: yes; Affected by 'instructions not enabled' bug: no
clang -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -o test src/t1ha1.c src/t1ha0.c tests/main.c
fatal error: error in backend: Cannot select: intrinsic %llvm.x86.aesni.aesenc
clang-3.9: error: clang frontend command failed with exit code 70 (use -v to see invocation)
clang version 3.9.1 (tags/RELEASE_391/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
clang-3.9: note: diagnostic msg: PLEASE submit a bug report to and include the crash backtrace, preprocessed source, and associated run script.
clang-3.9: note: diagnostic msg:
********************
PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-3.9: note: diagnostic msg: /tmp/t1ha1-c2cbe9.c
clang-3.9: note: diagnostic msg: /tmp/t1ha0-b53109.c
clang-3.9: note: diagnostic msg: /tmp/main-1e8446.c
clang-3.9: note: diagnostic msg: /tmp/t1ha1-c2cbe9.sh
clang-3.9: note: diagnostic msg:
********************
Makefile:45: recipe for target 'test' failed
make: *** [test] Error 70adding switches helps, but generates warnings
$ make CC="clang -maes -msse4"
CLANG: Version 3.9, ARCHx86: yes; Affected by 'instructions not enabled' bug: no
clang -maes -msse4 -Wextra -Werror -O -g -DT1HA_TESTING -std=c99 -o test src/t1ha1.c src/t1ha0.c tests/main.c
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -c -o t1ha0.o src/t1ha0.c
src/t1ha0.c:328:19: warning: unused function 't1ha_aes_resolve' [-Wunused-function]
static uint64_t (*t1ha_aes_resolve(void))(const void *, size_t, uint64_t) {
^
1 warning generated.
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -c -o t1ha1.o src/t1ha1.c
ar rs libt1ha.a t1ha0.o t1ha1.o
ar: creating libt1ha.a
clang -maes -msse4 -Wall -ffunction-sections -O3 -fPIC -g -std=c99 -fvisibility=hidden -Dt1ha_EXPORTS -shared -s -o libt1ha.so src/t1ha1.c src/t1ha0.c
src/t1ha0.c:328:19: warning: unused function 't1ha_aes_resolve' [-Wunused-function]
static uint64_t (*t1ha_aes_resolve(void))(const void *, size_t, uint64_t) {
^
1 warning generated.
./test || rm -rf libt1ha.a libt1ha.so
Testing t1ha_64le... Ok
Testing t1ha_64be... Ok
Testing t1ha_32le... Ok
Testing t1ha_32be... Ok
Testing t1ha_ia32aes... OkOutperform wyhash
Currently wyhash outperforms t1ha.
Nevertheless I think the next version of t1ha will be even faster.
Can't compile on virtualized FreeBSD 12.1
When I try to compile t1ha package in my FreeBSD 12.1 (i386) virtualized by VirtualBox, I got some errors in mera.c:
$ make
cc -Wextra -Werror -std=c99 -O3 -DNDEBUG -D_DEFAULT_SOURCE -fno-stack-protector -mtune=native -save-temps -c -o mera.o tests/mera.c
tests/mera.c:234:7: error: no member named 'sa_sigaction' in 'struct sigaction'; did you mean
'__sigaction_u'?
act.sa_sigaction = sigaction_handler;
^~~~~~~~~~~~
__sigaction_u
/usr/include/sys/signal.h:370:4: note: '__sigaction_u' declared here
} __sigaction_u;
^
tests/mera.c:234:20: error: assigning to 'union (anonymous union at
/usr/include/sys/signal.h:367:2)' from incompatible type 'void (int, siginfo_t *, void *)'
(aka 'void (int, struct __siginfo *, void *)')
act.sa_sigaction = sigaction_handler;
^ ~~~~~~~~~~~~~~~~~
tests/mera.c:444:7: error: implicit declaration of function 'gettimeofday' is invalid in C99
[-Werror,-Wimplicit-function-declaration]
if (gettimeofday(&tv, ((void *)0))) {
^
3 errors generated.
make: *** [Makefile:117: mera.o] Error 1
Now, I have solved those errors simply by putting in mera.c:
#ifndef _XOPEN_SOURCE
#define _XOPEN_SOURCE 700
#endif
I had also an error, always in mera.c that I have solved removing usleep and using nanosleep instead.
After fixing those 2 issues with the #define and with nanosleep, I get this:
$ make
cc -Wextra -Werror -std=c99 -O3 -DNDEBUG -D_DEFAULT_SOURCE -fno-stack-protector -mtune=native -save-temps -c -o mera.o tests/mera.c
error: inline assembly requires more registers than available at line 1412165
make: *** [Makefile:117: mera.o] Error 1
It is a very strange error, and probably it i a clang bug, or something like that. A friend of mine who compiled t1ha package with my those 2 fixes, had no problem.
Is this worth investigating more?
MOVBE support for modern x86
Streaming API does not match "atonce" API.
Hi.
Is it known (by design) what streaming API returns different value for the same data compared to "atonce" API?
I am testing it with next code.
void my_test(void) {
t1ha_context_t ctx;
uint64_t seed = 42;
char key[] = "qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq";
size_t key_len = sizeof(key) - 1;
t1ha2_init(&ctx, seed, key_len);
t1ha2_update(&ctx, key, key_len);
uint64_t v1 = t1ha2_final(&ctx, NULL);
uint64_t v2 = t1ha2_atonce(key, key_len, seed);
printf("v1: %" PRIu64 "\n", v1);
printf("v2: %" PRIu64 "\n", v2);
}
Code execution results in:
v1: 15442592273487182410
v2: 8257225483948642673
Bindings for Java via "Critical Natives"
JNIEXPORT jlong JNICALL
JavaCritical_com_package_t1ha_t1ha1_le(jint length, jbyte* buf, jlong seed) {
return t1ha1_le(buf, length, seed);
}
Related to benalexau/hash-bench#4
Related to https://bugs.openjdk.java.net/browse/JDK-7013347
Proxy for eBay payment
For convenience I decided to buy a pinebook (ARM64 CPU which is important for me). Actually eBay (unexpectedly for me) accepts only PayPal for this order, but not Visa/MasterCard.
PayPal still pending its own "verification" procedure without any reasonable information (except "We will get back to you within 30 business days").
So I can't pay for my purchase even I have enough funds (
It's anecdotal, but I don't know what to do for now.
I hope that the seller of the lot will be able to wait for some more reasonable time (until paypal finishes verification) or someone will offer a solution.
WebAssembly support
Is webassembly supported?
compatibility testing for implementation
I'm working on a Rust implementation of t1ha, it can pass all the selfcheck in the C code now, are there more compatibility test cases?
Hello, can we contact?
Hello, I accidentally found your github and saw that you are from Moscow, can we talk by email?
Pls, contact me:
[email protected]
Licensing
@leo-yuriev
I want to port t1ha to another language (Nim), MIT-licensed.
I would like to clarify the compatibility of licenses.
valgrind 'invalid read of size 8' in t1ha1 and t1ha2
I stumbled across your nice hash function while looking for ways to improve map performance in my '8th' programming language. I have been using fnv1a successfully. When I plugged in your code, I get the following valgrind complaints:
==29732== at 0x4CAF62: fetch64_le_unaligned (t1ha_bits.h:587)
==29732== by 0x4CAF62: tail64_le_unaligned (t1ha_bits.h:684)
==29732== by 0x4CAF62: t1ha1_le (t1ha1.c:134)
Similar complaint with the t1ha2 equivalent. This occurs any time your functions are called, the context in my code doesn't matter.
Проверка NIST тестом.
Здравствуйте, хочу сообщить о своих провеках хэш функции t1ha2.
Код:
#include <stdio.h>
#include "t1ha.h"
int main(void)
{
uint64_t seed = 1234;
/*
FILE *fp;
fp = fopen("/dev/urandom", "r");
fread((void *)&seed, 1, 8, fp);
fclose(fp);
*/
for (uint64_t i = 0; i < 100000; i++)
{
uint64_t out_hash = t1ha2_atonce((void *)&i, 8, seed);
fwrite((void *)&out_hash, 8, 1, stdout);
}
return 0;
}Вывод программы сохраняется в файл my_test > randdata. Компилировалось и запускалось на x86-64 архитектуре, если это важно.
Таким образом я делаю ГПСЧ из хеш-функции. Так вот, такой ГПСЧ проваливает тест NIST. Я взял https://github.com/dj-on-github/sp800_22_tests и запустил это таким образом:
/sp800_22_tests-master/sp800_22_tests.py -t random_excursion_variant_test rand_data_test
Результат:
Tests of Distinguishability from Random
TEST: random_excursion_variant_test
J= 217
x = -9 count=178 p = 0.321056
x = -8 count=201 p = 0.140221
x = -7 count=217 p = 0.000000 Not Random
x = -6 count=224 p = 0.071638
x = -5 count=196 p = 0.237595
x = -4 count=180 p = 0.474671
x = -3 count=172 p = 0.683074
x = -2 count=184 p = 0.646686
x = -1 count=204 p = 0.441249
x = 1 count=217 p = 0.000000 Not Random
x = 2 count=200 p = 0.333141
x = 3 count=192 p = 0.379485
x = 4 count=186 p = 0.397697
x = 5 count=183 p = 0.384678
x = 6 count=215 p = 0.020468
x = 7 count=247 p = 0.282416
x = 8 count=252 p = 0.306734
x = 9 count=246 p = 0.238734
J too small (J=217 < 500) for result to be reliable
FAIL
P=0.321055625331
P=0.14022146188
P=0.0
P=0.0716377331396
P=0.237595481656
P=0.474671155356
P=0.683073833309
P=0.646685986377
P=0.441248751646
P=0.0
P=0.33314126571
P=0.379485462949
P=0.397697454488
P=0.384678398871
P=0.0204679237542
P=0.282416272064
P=0.306734447862
P=0.238733670118
Тест провален. Может быть эти требования не стоит предьявлять к данной хеш функции, но я просто собщаю.
Critics of t1ha-crc hash
moved from rurban/smhasher#30
i don't recommend to use this crc-based hash (or any automatic selection of fastest hash as it's currently implemented in this package). while basic t1ha hashes (w/o crc/aes) look reasonable, crc-based hash is absolutely non-sense, made only for speed records rather than real work. in short, it computes 32-bit hash of each 64-bit input word, and then produces 64-bit hash based on these stripped numbers
@leo-yuriev, it will be great if you will finally remove this pseudo-hash from your package. Passing SMHasher doesn't guarantee that your hash is strong, you need also to understand how to change smhasher to make your hash fail. Your idea that users should understand CRC math and research each hash they are using yourself, is absolutely laughable - real users are much less educated even than you. And finally, idea to mix weak crc hash with reasonable main t1ha hash with automatic hash selection based solely on speed, deceives these users by pretending them that they are using reasomnnable main hash while automatic selection actually chooses crc hash in most cases. You are started with good idea, but then lost sense trying to setup meaningless records and relying on automatic smhasher testing rather than understanding of your own algorithms.
t1ha-crc hash is weaker than xxhash and so on. essentially, it's a crc32 hash arttifically widened to 64 bits, and you know that even crc32 can't pass half of smhasher tests. t1ha-crc weakness can be detected by the various smhasher tests checking 40-64 bit keys IF these keys will be preceded by fixed 40-byte header
Comparison with Google's HighwayHash
Comparison with https://github.com/google/highwayhash
@jan-wassenberg, this may be interesting for you.
ARM64 AES acceleration?
Do you plan to add support for ARM64 AES instructions, as you did for x86?
PERFORMANCE COMPARISON (this is not a bug or an issue)
Please note: the benchmarking results depend on the processor and compiler model.
...
model : 158
model name : Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz
stepping : 9
microcode : 0x8e
clang version 6.0.1 (tags/RELEASE_601/final)
Preparing to benchmarking...
- suggest enable rdpmc for usermode (echo 2 | sudo tee /sys/devices/cpu/rdpmc)
- running on CPU#4
- use RDPMC_perf as clock source for benchmarking
- assume it cheap and stable
- measure granularity and overhead: 53 cycle, 0.0188679 iteration/cycle
Bench for tiny keys (7 bytes):
t1ha2_atonce : 12.117 cycle/hash, 1.731 cycle/byte, 0.578 byte/cycle, 1.733 Gb/s @3GHz
t1ha2_atonce128* : 29.656 cycle/hash, 4.237 cycle/byte, 0.236 byte/cycle, 0.708 Gb/s @3GHz
t1ha2_stream* : 77.625 cycle/hash, 11.089 cycle/byte, 0.090 byte/cycle, 0.271 Gb/s @3GHz
t1ha2_stream128* : 97.625 cycle/hash, 13.946 cycle/byte, 0.072 byte/cycle, 0.215 Gb/s @3GHz
t1ha1_64le : 12.219 cycle/hash, 1.746 cycle/byte, 0.573 byte/cycle, 1.719 Gb/s @3GHz
t1ha0 : 14.070 cycle/hash, 2.010 cycle/byte, 0.498 byte/cycle, 1.493 Gb/s @3GHz
xxhash64 : 21.234 cycle/hash, 3.033 cycle/byte, 0.330 byte/cycle, 0.989 Gb/s @3GHz
StadtX : 20.219 cycle/hash, 2.888 cycle/byte, 0.346 byte/cycle, 1.039 Gb/s @3GHz
HighwayHash64_portable : 504.750 cycle/hash, 72.107 cycle/byte, 0.014 byte/cycle, 0.042 Gb/s @3GHz
HighwayHash64_sse41 : 81.938 cycle/hash, 11.705 cycle/byte, 0.085 byte/cycle, 0.256 Gb/s @3GHz
HighwayHash64_avx2 : 51.812 cycle/hash, 7.402 cycle/byte, 0.135 byte/cycle, 0.405 Gb/s @3GHz
gcc version 8.1.1 20180712 (Red Hat 8.1.1-5) (GCC)
Preparing to benchmarking...
- suggest enable rdpmc for usermode (echo 2 | sudo tee /sys/devices/cpu/rdpmc)
- running on CPU#2
- use RDPMC_perf as clock source for benchmarking
- assume it cheap and stable
- measure granularity and overhead: 54 cycle, 0.0185185 iteration/cycle
...
Bench for large keys (16384 bytes):
t1ha2_atonce : 3461.000 cycle/hash, 0.211 cycle/byte, 4.734 byte/cycle, 14.202 Gb/s @3GHz
t1ha2_atonce128* : 3438.000 cycle/hash, 0.210 cycle/byte, 4.766 byte/cycle, 14.297 Gb/s @3GHz
t1ha2_stream* : 3471.000 cycle/hash, 0.212 cycle/byte, 4.720 byte/cycle, 14.161 Gb/s @3GHz
t1ha2_stream128* : 3484.000 cycle/hash, 0.213 cycle/byte, 4.703 byte/cycle, 14.108 Gb/s @3GHz
t1ha1_64le : 3431.000 cycle/hash, 0.209 cycle/byte, 4.775 byte/cycle, 14.326 Gb/s @3GHz
t1ha0 : 1187.000 cycle/hash, 0.072 cycle/byte, 13.803 byte/cycle, 41.409 Gb/s @3GHz
xxhash64 : 4120.000 cycle/hash, 0.251 cycle/byte, 3.977 byte/cycle, 11.930 Gb/s @3GHz
StadtX : 3501.000 cycle/hash, 0.214 cycle/byte, 4.680 byte/cycle, 14.039 Gb/s @3GHz
HighwayHash64_portable : 41965.000 cycle/hash, 2.561 cycle/byte, 0.390 byte/cycle, 1.171 Gb/s @3GHz
HighwayHash64_sse41 : 6222.000 cycle/hash, 0.380 cycle/byte, 2.633 byte/cycle, 7.900 Gb/s @3GHz
HighwayHash64_avx2 : 4601.000 cycle/hash, 0.281 cycle/byte, 3.561 byte/cycle, 10.683 Gb/s @3GHz
Add explicit description of T1HA_USE_FAST_ONESHOT_READ at the top of t1ha.h
inline functions don't follow feature restrections enforced by clang
In file included from external/t1ha/src/t1ha0_ia32aes_avx.c:2:
external/t1ha/src/t1ha0_ia32aes_a.h:54:17: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
__m128i y = _mm_aesenc_si128(x, _mm_set_epi64x(prime_5, prime_6));
^
external/t1ha/src/t1ha0_ia32aes_a.h:70:21: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
__m128i v0y = _mm_aesenc_si128(v0, y);
^
external/t1ha/src/t1ha0_ia32aes_a.h:71:22: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
__m128i v2x6 = _mm_aesenc_si128(v2, _mm_xor_si128(x, v6));
^
external/t1ha/src/t1ha0_ia32aes_a.h:73:25: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
_mm_xor_si128(_mm_aesenc_si128(v4, v5), _mm_add_epi64(v6, v7));
^
external/t1ha/src/t1ha0_ia32aes_a.h:75:24: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
__m128i v0y7_1 = _mm_aesdec_si128(_mm_sub_epi64(v7, v0y), v1);
^
external/t1ha/src/t1ha0_ia32aes_a.h:76:24: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
__m128i v2x6_3 = _mm_aesenc_si128(v2x6, v3);
^
external/t1ha/src/t1ha0_ia32aes_a.h:78:11: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
x = _mm_aesenc_si128(v45_67, _mm_add_epi64(x, y));
^
external/t1ha/src/t1ha0_ia32aes_a.h:79:11: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
y = _mm_aesenc_si128(v2x6_3, _mm_xor_si128(v0y7_1, v5));
^
external/t1ha/src/t1ha0_ia32aes_a.h:86:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
x = _mm_aesdec_si128(x, v0y);
^
external/t1ha/src/t1ha0_ia32aes_a.h:87:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
y = _mm_aesdec_si128(y, v1x);
^
external/t1ha/src/t1ha0_ia32aes_a.h:91:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
x = _mm_aesdec_si128(x, v2y);
^
external/t1ha/src/t1ha0_ia32aes_a.h:92:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
y = _mm_aesdec_si128(y, v3x);
^
external/t1ha/src/t1ha0_ia32aes_a.h:98:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
x = _mm_aesdec_si128(x, v0y);
^
external/t1ha/src/t1ha0_ia32aes_a.h:99:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
y = _mm_aesdec_si128(y, v1x);
^
external/t1ha/src/t1ha0_ia32aes_a.h:104:11: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
x = _mm_aesdec_si128(x, _mm_loadu_si128(v++));
^
external/t1ha/src/t1ha0_ia32aes_a.h:107:23: error: always_inline function '_mm_aesdec_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
x = _mm_add_epi64(_mm_aesdec_si128(x, _mm_aesenc_si128(y, x)), y);
^
external/t1ha/src/t1ha0_ia32aes_a.h:107:43: error: always_inline function '_mm_aesenc_si128' requires target feature 'aes', but would be inlined into function 't1ha0_ia32aes_avx' that is compiled without support for 'aes'
x = _mm_add_epi64(_mm_aesdec_si128(x, _mm_aesenc_si128(y, x)), y);
^
> clang --version
Apple LLVM version 9.1.0 (clang-902.0.39.1)
Target: x86_64-apple-darwin17.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Notes about t1ha_v3 design
- Dirty and Fair kinds:
t1ha3_dirty= favour speed over quality: non-uniform, all hardware tricks (ie. AES-NI), one injection point and blinding multiplication are allowed;t1ha3_fair= favour quality over speed: two injection points, no known drawbacks.
- Goals:
t1ha3_dirty: should be faster than competitors (e.g.xxHash_v3,MeowHashandwyhash_v777), while not inferior in quality; Or at least the same speed with better quality (i.e. no several drawbacks).t1ha3_fair: proven excellence in the quality of competitors, with minimal performance degradation.- target architectures in order of priority: E2K, AMD64, AARCH64, RISC-V, MIPS64, MIPS32, ARM (no Thumb, but Thumb-2).
- optional: additional variations of speed/quality/security tradeoffs.
- Endianness: No explicit LE/BE versions:
t1ha3_dirty: the native byte order is always used;t1ha3_fair: Little Endian byte order is primary, bswap on BE platforms.
- Bitness:
- result: 32, 64, 128.
- implementations: 64, 32.
- Insights:
- avoid MUM-mixer;
- single header-only C99 implementation;
- avoid unaligned read;
- avoid unsafe user options;
- SIMD-frendly;
- non-injective mixers.
Schedule:
- Most likely, only the final version will be published when there is no doubt about achieving the goals.
- No specific deadlines are publicly declared. Sponsor the development if you need any guarantees.
Работа со строками
Как работать со строками?
Можете привести пример для хеширования строки?
Comparison with xxHash
If the primary purpose of this project is speed, why not xxHash, which is significantly faster than CityHash and has the same quality according to SMHasher?
read_unaligned error
Code
#include <iostream>
#include "windows.h"
#include "t1ha.h"
#include <basetsd.h>
#include <concrt.h>
#pragma comment(lib, "t1ha-static.lib")
int main()
{
UINT64 Hash = 0;
int regnumber = 1;
Hash = t1ha0((const void*)regnumber, (size_t)regnumber, 0x1EE7C0DE);
system("pause");
}
Exception
does t1ha support combine, like crc32?
Привет, если не против, напишу на русском:
У CRC32 есть combine, который позволяет сократить объем вычислений, если изменился не весть датасет, а известный его кусок.
xxHash это не поддерживает.
Скажите, поддерживает ли t1ha?
Варианты t1ha, переименования для ясности и порядка
Ради внесения ясности и поддержания порядка текущие t1ha-функции будут переименованы по схеме "t1ha#", где # - цифра обозначающая вариант:
0 = 64 бита, "лишь-бы быстрее" (текущая t1ha_local).
1 = 64 бита, текущая t1ha (в том числа для big-endian).
Одновременно в t1ha0() будут спрятаны все текущие суррогаты, включая 32-битные вариации и AES-NI.
Также (постепенно) будут добавлены новые варианты:
2 = 64 бита, для отпечатков (быстрая, "инъективная", не стойкая).
3 = 128 бит, для отпечатков (быстрая, не стойкая).
4 = 128 бит, быстро-стойкая (целевое применение как у SipHash).
5 = 256 бит, для отпечатков (быстрая, не стойкая).
6 = 256 бит, не быстрая, стойкая (примерно как Argon2 и т.п.).
7 = 256 бит, стойкая, не ускоряемая (для паролей, многораундовая, с широкими умножениями и вариативными циклическими сдвигами).
SIGBUS due to unaligned access on armv7 (GCC 9)
(Sort of a duplicate of #30, sorry for that, I was unable to re-open that issue).
Basically armv7 unaligned access means that it's possible to read and write 16- and 32-bit values from addresses that are not 16- or 32-bit aligned (more details). However (as far as I understand) t1ha uses unaligned 64-bit access, and that requires 8-byte alignment (or should be replaced with two 32-bit reads/writes).
Also nothing guarantees that unaligned access is efficient.
Split test and bench application
Another option - add command line option to disable execution of performance tests
fixed GH En description a little
The Future will be Positive.
Briefly, it is a portable 64-bit hash function:
Intended for 64-bit little-endian platforms, predominantly for x86_64, but portable and without penalties it can run on any 64-bit CPU.
In most cases up to 15% faster than City64, xxHash, mum-hash, metro-hash and all others portable hash-functions (which do not use specific hardware tricks).
Wrong x86 code generated by Microsoft Visual Studio 2015
This is not t1ha's bug, but MSVC.
Nevertheless, one more crutch/workaround is required for troubleful Microsoft compiler.
Update scores for t1ha_ia32aes_xyz()
- t1ha_ia32aes_noavx()
- t1ha_ia32aes_avx()
- t1ha_ia32aes_avx2()
__ImageBase in t1ha-static
Hello,
how to disable this __ImageBase variable? This variable creates a program with broken memory. Manual replacement of this variable through GetModuleHandle() does not help either.
LEA RDX, QWORD PTR DS:[<__ImageBase>]
MOVZX EAX, BYTE PTR DS:[RDX + R10 + 0x1388]
MOV ECX, DWORD PTR DS:[RDX + RAX * 4 + 0x1378]
ADD RCX, RDX
JMP RCX
Error A2102: Symbol not defined : __ImageBase
Regards
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.