"zpaqfranz autotest" segfaults on i686 platform

[ppisar]

"zpaqfranz autotest" segfaults on i686 platform

[ppisar]

It crashes in HighwayHashReset() on line:

state->v1[3] = state->mul1[3] ^ ((key[3] >> 32) | (key[3] << 32));

when loading key[3]. But the key[] array looks good.

What does not look good is a disassembled HighwayHashReset() function. A code for handling a line above it is correct:

   0x0804c0e0 <+256>:   mov    0x14(%eax),%esi
   0x0804c0e3 <+259>:   mov    0x10(%eax),%edi
   0x0804c0e6 <+262>:   xor    $0x34e90c6c,%esi
   0x0804c0ec <+268>:   xor    $0xbe5466cf,%edi
   0x0804c0f2 <+274>:   mov    %esi,0x30(%edx)
   0x0804c0f5 <+277>:   mov    %edi,0x34(%edx)

But a code for handling the segfaulting code uses wrong registers:

   0x0804c0f8 <+280>:   mov    0x1c(%eax),%ecx
=> 0x0804c0fb <+283>:   mov    0x18(%ecx),%ebx
   0x0804c0fe <+286>:   xor    $0x38d01377,%ecx
   0x0804c104 <+292>:   xor    $0x452821e6,%ebx
   0x0804c10a <+298>:   mov    %ecx,0x38(%edx)
   0x0804c10d <+301>:   mov    %ebx,0x3c(%edx)

The line with the arrow is the segaulting instruction and it should read:

 mov    0x18(%eax),%ebx

Interesting is that instructions for "state->v0[3] = state->mul0[3] ^ key[3];" line which also loads key[3] is fine:

   0x0804c099 <+185>:   mov    0x18(%eax),%esi
   0x0804c09c <+188>:   mov    0x1c(%eax),%edi
   0x0804c09f <+191>:   xor    $0x85a308d3,%esi
   0x0804c0a5 <+197>:   xor    $0x243f6a88,%edi
   0x0804c0ab <+203>:   mov    %esi,0x18(%edx)
   0x0804c0ae <+206>:   mov    %edi,0x1c(%edx)

I conclude it's a bug GCC (gcc-13.2.1-1.fc39.i686). I will try to minimize the reproducer and report it to GCC.

[fcorbelli]

(1) there are compiler bugs on finding exact region size (yes) on some versions
(2) for example this one zpaqfranz.cpp:53655:46: warning: ‘memset’ writing 8 bytes into a region of size 0 overflows the destination [-Wstringop-overflow=]
53655 | buffer8bit[j]=77;
(3) blake3 is a "strange" thing, very, very strange. But you can trust on them :)
You can see one of them here BLAKE3-team/BLAKE3#189

Short version: on amd64 I am pretty confident everything will run

[fcorbelli]

(2) for 32 bit I am not so sure
No problems at all on very old compilers BUT not build on newer one by some time
I am making a 32 bit VM

[fcorbelli]

If it not a big deal, please use build 58.10g, instead of 58.9.
Just a "drop in" of zpaqfranz.cpp, nothing more is required.

WOW Fedora dropped from version 30, downloading old ISO...

[fcorbelli]

Update
Fedora 25, 32 bit, running on a VMware VM, gcc 6.4.1

g++ -O3 -Dunix zpaqfranz.cpp -o zpaqfranz -pthread

Runs fine (-fcf-protection and -fstack-clash-protection not available on this compiler). No warnings

g++ -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Wno-complain-wrong-lang -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m32 -march=i686 -mtune=generic -msse2 -mfpmath=sse -mstackrealign -fasynchronous-unwind-tables -Dunix zpaqfranz.cpp -Wl,-z,relro -Wl,--as-needed -Wl,-z,now  -Wl,--build-id=sha1  -pthread -o zpaqfranz

Runs fine, no warnings.
I try to update the compiler, but it's not easy

[fcorbelli]

[root@localhost zp]# ./zpaqfranz b -debug
48149: array franz flag size 61
48150: -715                   0  <<Runs just about like zpaq 7.15>>
(...)
41575: Pretend to be Linux
zpaqfranz v58.10g-JIT-L(2023-09-12)
FULL exename <</zp/zpaqfranz>>
42993: The chosen algo 3 SHA-1
franz:-debug -verbose
sysname : Linux
nodename: localhost.localdomain
release : 4.13.16-100.fc25.i686+PAE
version : #1 SMP Mon Nov 27 20:26:05 UTC 2017
machine : i686
uname i686
full exename seems <</zp/zpaqfranz>>
Hello Linux
MemAvailable: |MemAvailable:    1896652 kB
|
Onlynumber |1896652|
Atoi  1.896.652
Free RAM seems 1.942.171.648
Benchmarks: XXHASH64 XXH3 SHA-1 SHA-256 BLAKE3 CRC-32 CRC-32C WYHASH WHIRLPOOL M            D5 SHA-3 NILSIMSA HIGHWAY64
Time limit          5 s  (-n X)
Chunks of     390.62 KB  (-minsize Y)

001 s     XXHASH64: speed (    1.36 GB/s)
002 s     XXHASH64: speed (    1.36 GB/s)
003 s     XXHASH64: speed (    1.36 GB/s)
(...)

On 32-bit Fedora VM (on 64 bit host) everything seems good

Given that Fedora seems to have abandoned 32-bit altogether, is it possible to exclude that platform from the package?
I would like to reduce the time you dedicate to me to the minimum possible, and for which I am grateful

[ppisar]

I will probably disable i686 build on Fedora.
I can reproduce it locally. A trigger is changing an optimization level from -O0 to -O1 (g++ -O1 -Dunix zpaqfranz.cpp -pthread -o zpaqfranz).

[fcorbelli]

Would you prefer that I disable Highway altogether?
I know one of the authors, in the medium term I can report the situation to him to ask for refactoring as a workaroud for the compiler (!)
Or do I leave it as is?

[fcorbelli]

On a Fedora 64bit VM, with g++ and i686 libraries, compiling with

g++ -m32 -O1 -Dunix zpaqfranz.cpp -o zpaqfranz -pthread

goes in fault, for gcc 13.2.1 20230728 (Red hat 13.2.1-1)

Using clang 16.0.6 (Fedora 16.0.6-3.fc38) with

clang++ -m32 -O1 -Dunix zpaqfranz.cpp -o zpaqfranz -pthread

runs fine

[root@fedora zp]# clang++ -m32 -O1  -Dunix zpaqfranz.cpp -o zpaqfranz -pthread
[root@fedora zp]# ./zpaqfranz autotest
zpaqfranz v58.10g-JIT-L(2023-09-12)
Self-test for correct internal functioning
Quick check chunsize 5: ABCDE
This seems a LITTLE ENDIAN CPU (aka:'normal')
----------------------------------------------------------------------------
CALC          BLAKE3: 61274278289E9F6233DF34ABB392AAFE03EE7CE9D77167F3A8D9CDE1AD9861C0
CALC          CRC-32: 72D31AD5
CALC         CRC-32C: B6AFE183
CALC      HIGHWAY128: C4C97D024A5DDDC7609B799BDBC46492
CALC      HIGHWAY256: 13F23C2D5C49147DB7B10C8D8923CB052C2B3E1FF3EED5681E4FD69F05FFA6FE
CALC       HIGHWAY64: 4755E237BAD15B4C
CALC             MD5: 2ECDDE3959051D913F61B14579EA136D
CALC     SHA-256-PUT: F0393FEBE8BAAA55E32F7BE2A7CC180BF34E52137D99E056C817A9C07B8F239A
CALC   SHA-256-WRITE: F0393FEBE8BAAA55E32F7BE2A7CC180BF34E52137D99E056C817A9C07B8F239A
CALC           SHA-3: 034AF02F68F8874B6668CCBEE49143A64BE435610E1282D93BF35FD80ACCE1FB
CALC        SHA1-PUT: 7BE07AAF460D593A323D0DB33DA05B64BFDCB3A5
CALC      SHA1-WRITE: 7BE07AAF460D593A323D0DB33DA05B64BFDCB3A5
CALC       WHIRLPOOL: C73D8F890F181CE6EB9FF431E08828D6BADA50AC2427546BA10A8F8226F527850FB61E638F798CE86028248262DF17D77D9D00FA5FE5E6CE6C94267E1DC2E99C
CALC            XXH3: 1C8288B6013152D97B4A5D7E6C7893D4
CALC        XXHASH64: E47599E7C7CEF609
CALC       XXHASH64Y: E47599E7C7CEF609
----------------------------------------------------------------------------
Time 0.00 seconds for bytes 0

root@fedora zp]# file ./zpaqfranz
./zpaqfranz: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, BuildID[sha1]=d2c7ab751a8fa38f09bbc0e0e67669183a9b1434, for GNU/Linux 3.2.0, not stripped

And the data is good

[root@fedora zp]# ./zpaqfranz autotest -all
zpaqfranz v58.10g-JIT-L(2023-09-12)
franz:-all                                      4
Self-test for correct internal functioning
Iteration 0/9 chunksize    1.000.000                               
Iteration 1/9 chunksize      333.333                               
Iteration 2/9 chunksize      111.111                               
Iteration 3/9 chunksize       37.037                               
Iteration 4/9 chunksize       12.345                               
Iteration 5/9 chunksize        4.115                               
Iteration 6/9 chunksize        1.371                               
Iteration 7/9 chunksize          457                               
Iteration 8/9 chunksize          152                               
Iteration 9/9 chunksize           50                               
This seems a LITTLE ENDIAN CPU (aka:'normal')
         BLAKE3 : OK
         CRC-32 : OK
        CRC-32C : OK
     HIGHWAY128 : OK
     HIGHWAY256 : OK
      HIGHWAY64 : OK
            MD5 : OK
    SHA-256-PUT : OK
  SHA-256-WRITE : OK
          SHA-3 : OK
       SHA1-PUT : OK
     SHA1-WRITE : OK
      WHIRLPOOL : OK
           XXH3 : OK
       XXHASH64 : OK
      XXHASH64Y : OK
Time 54.30 seconds for bytes 1.748.003.691

54.302 seconds (000:00:54) (all OK)
[root@fedora zp]# 

=> this really does not seems my fault 😄

[ppisar]

A bug report for Fedora GCC https://bugzilla.redhat.com/show_bug.cgi?id=2238781.

[fcorbelli]

Can you please test the attached pre-release?
58_10h.zip
I had to work on it a little bit, but I hope I succeeded.

[ppisar]

That works for me.

[fcorbelli]

I had to be, shall we say, "creative" (really "creative")
Obviously it's a workaround, who knows if it will work for future versions, but it's not my fault

To recap, as a courtesy, if you proceed with the "packaging" please use the current zpaqfranz.cpp file
https://github.com/fcorbelli/zpaqfranz/blob/main/zpaqfranz.cpp

This include a double quote disclaimer too, like here #74, and a change to make... easier use of $ substitution on Linux (replaced via %something)

[fcorbelli]

AFAIK my workaroud fixed the gcc bug, closing for now