Unexpected error for huge data

[facebook]

Unexpected error for huge data

[Cyan4973]

Thanks Takayuki

I suspect that, in my hastiness to release zStd, I forgot to re-enable a match offset limit, which is useless in most circumstances, but should prove useful in such a case.

I'll look into it.

[Cyan4973]

There's a new version available in the "dev" branch which is supposed to fix this issue.
I can't test it directly, since my system would not allow me to allocated that much memory.
Would you mind having a look ?

Regards

[t-mat]

Confirmed. New logic works fine both Ubuntu and Windows.

But new code has another problem.

Using defined(_INCLUDED_IMM) for AVX2 detection is really problematic. It detects just a existence of the header. But it does not indicate compiler's code generation capability and target architecture. For example, currently we can't genarete generic x86-64 binary for non-AVX2 environment.

Please remove defined(_INCLUDED_IMM) from AVX2 detection.

[Cyan4973]

OK.
Is __AVX2__ macro detection enough for this, on both Unix & Windows ?

[t-mat]

Yes. Recent gcc, clang, icc and MSVC has predefined __AVX2__ macro for AVX2 target.

$ gcc -march=core-avx-i -dM -E -x c /dev/null | grep "AVX"
#define __AVX__ 1

$ gcc -march=core-avx2 -dM -E -x c /dev/null | grep "AVX"
#define __AVX__ 1
#define __AVX2__ 1

$ clang -march=core-avx-i -dM -E -x c /dev/null | grep "AVX"
#define __AVX__ 1

$ clang -march=core-avx2 -dM -E -x c /dev/null | grep "AVX"
#define __AVX2__ 1
#define __AVX__ 1

I'm not sure we should support older compiler or not, but optional configure macro (eg. ZSTD_ENABLE_AVX2) would be nice for some people.

[Cyan4973]

Excellent suggestion. Thanks Takayuki.

[Cyan4973]

I've changed AVX2 detection macro to __AVX2__
The idea of providing a manual switch to turn on/turn off AVX2 is a good one, and will probably be achieved in a later revision.