madler/zlib

A massively spiffy yet delicately unobtrusive compression library.

zlib's Issues

Might want to drop the local and OF preprocessor defines

Hi,

I just submitted a patch to minizip in GDAL that you might be interested in (aside from the C++ parts):

https://trac.osgeo.org/gdal/changeset/31508

The 2 main things are to drop the local and OF pre-processor #defines. It seems like modern debuggers should all be able to handle static functions and worrying about K&R C support these days is not worth it. If you disagree, I'd love to hear why. I tried looking at the mailing list archives to see if this had been discussed, but they 404 at the moment.

Pardon the C++ noise in the GDAL patch.

cross compiling from 64bit system to 32bit

$ CC="gcc -m32" CXX="g++ -m32" ; ./configure --archs="-arch i386 -arch x86_64"
Checking for gcc...
Compiler error reporting is too harsh for ./configure (perhaps remove -Werror).
** ./configure aborting.

Without options "--archs=..." I have
$ objdump -p libz.so.1.2.8 | grep format
libz.so.1.2.8: file format elf64-x86-64

zlib 1.2.8

ABI compatibility issue with get_crc_table on Linux

We've been seeing a problem with zlib in newer Linux distributiions on the x86_64 architecture.

Because of the type change made to the CRC table, the table returned by get_crc_table is aligned differently. Programs built against an older zlib and dynamically linked will read the table incorrectly when used with a newer zlib, and could potentially read past the buffer.

Would it be possible to provide the current get_crc_table with a new symbol version and provide a work-alike get_crc_table (with the incorrect type) at the old symbol version?

If you're busy, I'm willing to do the work to fix this, but I wanted to get your opinion before starting on it.

This was discovered in testing newer distributions for LSB compliance. Our bug is here:

https://lsbbugs.linuxfoundation.org/show_bug.cgi?id=3441

crash (segfault) in inflate_fast when linked with -Wl,-z,relro

I'm getting a crash (segfault) in "inflate_fast" when extracting a zipped content from a PDF. I'm using zlib version 1.2.3 and compiled/linked it with -Wl,-z,relro using GCC. This linker flags weren't entered by me but by somebody else (probably my distribution vendor). Without these flags I don't get a crash.

As far as I understand this, the flags shall warn (by crashing) that somebody, namely zlib, is writing into a read-only memory region.

Do you know what could cause this problem? Or have you fixed this issue in a newer version of zlib already? I couldn't find this in the ChangeLog.

minizip creates corrupted zip files and miniunz cannot open zip files

I found the reason why this happens:

In the file of zlib-1.2.8\contrib\minizip\iowin32.c
in function of
long ZCALLBACK win32_seek64_file_func (voidpf opaque, voidpf stream, ZPOS64_T offset, int origin)
the line
if (!MySetFilePointerEx(hFile, pos, NULL, FILE_CURRENT))
should be
if (!MySetFilePointerEx(hFile, pos, NULL, dwMoveMethod))

and in function of
static BOOL MySetFilePointerEx(HANDLE hFile, LARGE_INTEGER pos, LARGE_INTEGER *newPos, DWORD dwMoveMethod)
the line
DWORD dwNewPos = SetFilePointer(hFile, pos.LowPart, &lHigh, FILE_CURRENT);
should be
DWORD dwNewPos = SetFilePointer(hFile, pos.LowPart, &lHigh, dwMoveMethod);

These two errors look simple mistyping/copy paste errors. FILE_CURRENT was used instead of the correct move methods. Interesting though that no one has complained about this issue.

To make things work on Win7 with VC14 I had to add
#undef IOWIN32_USING_WINRT_API
after the lines of
#if defined(WINAPI_FAMILY_PARTITION) && (!(defined(IOWIN32_USING_WINRT_API)))
#if WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_APP)
#define IOWIN32_USING_WINRT_API 1
#endif
#endif
There is an open bug for this issue already (zlib 1.2.8 always uses Windows 8 - level API #49)

Enable Sourcegraph

I want to use Sourcegraph for zlib code search, browsing, and usage examples. Can an admin enable Sourcegraph for this repository? Just go to https://sourcegraph.com/github.com/madler/zlib. (It should only take 30 seconds.)

Thank you!

CMake skip building examples/tests/running tests and installing files? (As submodule)

When using this repo as a submodule I'd like to just build the static lib/dll/so. Is there a way to disable:

-Building example binaries
-Adding/running test (the example binaries)
-Installing files

I noticed there seems to be a skip var for each install section?

if(NOT SKIP_INSTALL_HEADERS AND NOT SKIP_INSTALL_ALL )
install(FILES ${ZLIB_PUBLIC_HDRS} DESTINATION "${INSTALL_INC_DIR}")
endif()

Please add deflateGetDictionary function

Please add deflateGetDictionary function. It will be useful in streaming mode (after sync flush) when inflateGetDictionary and deflateGetDictionary can be used to get dictionary on both sides to be used as preset dictionary for another compression algorithm when it is needed to change compression algorithm due to channel speed change.

test/infcover.c does not compile on linux

The following definition in infcover.c fixes the compile-error, but I am unsure whether this is enough?

+/* reallocf is BSD-specific */
+#ifndef reallocf
+# define reallocf realloc
+#endif

Building fails in Cygwin.

$ ./configure
Checking for gcc...
Checking for shared library support...
No shared library support.
Building static library libz.a version 1.2.8 with gcc.
Checking for off64_t... No.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... No.

$ make -f win32/Makefile.gcc
gcc -O3 -Wall -c -o adler32.o adler32.c
gcc -O3 -Wall -c -o compress.o compress.c
gcc -O3 -Wall -c -o crc32.o crc32.c
gcc -O3 -Wall -c -o deflate.o deflate.c
gcc -O3 -Wall -c -o gzclose.o gzclose.c
gcc -O3 -Wall -c -o gzlib.o gzlib.c
gcc -O3 -Wall -c -o gzread.o gzread.c
gcc -O3 -Wall -c -o gzwrite.o gzwrite.c
gcc -O3 -Wall -c -o infback.o infback.c
gcc -O3 -Wall -c -o inffast.o inffast.c
gcc -O3 -Wall -c -o inflate.o inflate.c
gcc -O3 -Wall -c -o inftrees.o inftrees.c
gcc -O3 -Wall -c -o trees.o trees.c
gcc -O3 -Wall -c -o uncompr.o uncompr.c
gcc -O3 -Wall -c -o zutil.o zutil.c
ar rcs libz.a adler32.o compress.o crc32.o deflate.o gzclose.o gzlib.o gzread.o gzwrite.o infback.o inffast.o inflate.o inftrees.o trees.o uncompr.o zutil.o
windres --define GCC_WINDRES -o zlibrc.o win32/zlib1.rc
gcc -shared -Wl,--out-implib,libz.dll.a
-o zlib1.dll win32/zlib.def adler32.o compress.o crc32.o deflate.o gzclose.o gzlib.o gzread.o gzwrite.o infback.o inffast.o inflate.o inftrees.o trees.o uncompr.o zutil.o zlibrc.o
Cannot export gzopen_w: symbol not defined
collect2: ld returned 1 exit status
win32/Makefile.gcc:96: recipe for target `zlib1.dll' failed
make: *** [zlib1.dll] Error 1

‘gzopen_w’ in win32/zlib.def is useful for MinGW, but it is wrong for Cygwin.

Double pointer issue in testzlib.c

I am following these instructions to test the zlibStat library with testzlib. When I try to build testzlib, VS2012 throws the following error in testzlib.c, Line: 167, Char:43:

IntelliSense: argument of type "unsigned char *" is incompatible with parameter of type "void *"

Implicit conversion doesn't work with double indirection. Converting char* to void* is ok, but char** to void** isn't. See here for more details: http://c-faq.com/ptrs/genericpp.html

I replaced the ReadFileMemory signature with ReadFileMemory(const char* filename,long* plFileSize,unsigned char** pFilePtr).

After that, I got an error on line 141, so I changed it to *pFilePtr=(unsigned char*)ptr;.

Hopefully you guys would update the code accordingly.

OS_CODE documentation correction

The documentation for using Zlib to write a GZip header seems to be incorrect on the behavior of how the header fields are set.

The docs: If deflateSetHeader is not used, the default gzip header has text false, the time set to zero, and os set to 255, with no extra, name, or comment fields.

However, in update 1.2.2.2 (30 December 2004) this behavior was changed to use OS_CODE in the deflate() default gzip header.

On my Windows machine this results in zlib setting the gzip header to have the OS field set to 0 rather than 255.

Unsafe casts

Hi, I am running a static scanner against zlib and have a bunch of bugs to report. From a sample:

In gzclose(File f) Line 19 of gzclose.c
state = (gz_statep)file;

file points to struct of type gzFile_s but state points to struct of type gz_state that is much bigger than gzFile_s in size. The cast from file to state is thus unsafe.

Note: The snippet of code is valid if macro #ifndef NO_GZCOMPRESS is valid.

The other warnings are listed below. To be read as cast from Type, cast to type, line number, function name.

Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 323 of gzbuffer
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 347 of gzrewind
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 374 of gzseek64
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 459 of gztell64
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 487 of gzoffset64
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 519 of gzeof
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 537 of gzerror
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 557 of gzclearerr
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 300 of gzread
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 397 of gzgetc
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 432 of gzungetc
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 496 of gzgets
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 555 of gzdirect
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 576 of gzclose_r
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 177 of gzwrite
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 257 of gzputc
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 319 of gzvprintf
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 474 of gzflush
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 508 of gzsetparams
Invalid cast: 
From struct.gzFile_s    to struct.gz_state
Line: 548 of gzclose_w

Build warning in inflate.c with clang 3.7

I get this when building Firefox with clang 3.7:

 4:38.27 /Users/ehsan/moz/src/modules/zlib/src/inflate.c:1507:61: error: shifting a negative signed value is undefined [-Werror,-Wshift-negative-value]
 4:38.27     if (strm == Z_NULL || strm->state == Z_NULL) return -1L << 16;
 4:38.27                                                         ~~~ ^
 4:38.28 1 error generated.

This warning is turned on by -Wall, and we treat warnings as errors, hence the error message.

inftrees incorrectly assumes huffman tree is complete

The huffman tree recreation function assumes that the huffman tree is complete, i.e. that all available codes are used. However, the DEFLATE specification contains no such requirement, as far as I can tell.

Since the huffman tree is itself compressed, there are cases where it is beneficial to use an incomplete tree that compresses better, shaving bits off the block header that more than make up for the reduced coding efficiency of the main data. Sadly, since Zlib is a defacto standard, this compression trick cannot be used in practice, even though it is valid according to the DEFLATE standard.

:(

Problems building zlibvc using Visual C++ Express 2010

------ Erstellen gestartet: Projekt: zlibvc, Konfiguration: Release Win32 ------
gzlib.c
......\gzlib.c(180): warning C4013: 'open' undefiniert; Annahme: extern mit Rückgabetyp int
......\gzlib.c(204): warning C4013: '_lseeki64' undefiniert; Annahme: extern mit Rückgabetyp int

This can be solved by changing in gzlib.c

#  define LSEEK _lseeki64

to

#  define LSEEK _lseeki64
#  include <io.h>

Proposal of performance up (ZSWAP32())

Hi there.

There is a suggestion of performance up to ZSWAP32() of zutil.h.

If the version of gcc seemed 4.3.0 above, proposes to replace the
gcc's builtin function __builtin_bswap32 ().

for example,

diff --git a/zlib/zutil.h b/zlib/zutil.h
index 24ab06b..95080ed 100644
--- a/zlib/zutil.h
+++ b/zlib/zutil.h
@@ -247,7 +247,11 @@ extern z_const char * const z_errmsg[10]; /* indexed by 2-zlib_error */
#define TRY_FREE(s, p) {if (p) ZFREE(s, p);}

/* Reverse the bytes in a 32-bit value */
+#if (GNUC > 3) && (GNUC_MINOR > 2)
+#define ZSWAP32(q) __builtin_bswap32(q)
+#else
#define ZSWAP32(q) ((((q) >> 24) & 0xff) + (((q) >> 8) & 0xff00) +
(((q) & 0xff00) << 8) + (((q) & 0xff) << 24))
+#endif

#endif /* ZUTIL_H */

This modification affects the crc32.c(big endian only) and inflate.c.
The arm architecture, about 5 instructions of the object inflate.o is reduced.

Thank you.

---

よっちい

ASM zlib build on Windows gives erroneous results

The bug happens using an ASM optimized static zlib 1.2.5 or 1.2.7 on Windows x86. Supposed is to have no output, but currently it outs "%Cë". Non ASM versions on Windows do that right, as well both ASM and non ASM builds on Linux. The bad data can be fetched from http://188.40.74.4/corrupted.gz .

Using the snippet and data from

https://gist.github.com/anonymous/2c5a88ca9ac6f4c2a064

the erroneous behavior is reproducible.

It is possible, that the same issue does exist on Darwin, but sadly I've no such OS to test. This bug is originally detected in PHP https://bugs.php.net/bug.php?id=61677 .

Bug in deflate with Z_FULL_FLUSH and a specific output buffer size

When deflate is called with a buffer that is exactly large enough to hold the output, avail_out comes back as zero. As per the ZLib documentation:

If deflate returns Z_OK and with zero avail_out, it must be called again after making room in the output buffer because there might be more output pending.

Note that in this case, the sync trailer has already been emitted (0x00, 0x00, 0xff, 0xff). On the subsequent call to deflate, a second sync trailer is emitted. This results in an unnecessary 4 bytes being added to the output.

This example program illustrates the problem. The result of compress should be the same no matter the value of availOut. But it isn't, the first string is 24 bytes in size and the second string is 19 bytes in size even though both inflate to the same result.

#include <iostream>
#include <zlib.h>

std::string
compress(std::string const& in, std::size_t availOut)
{
    z_stream zs;
    zs.zalloc = Z_NULL;
    zs.zfree = Z_NULL;
    zs.opaque = Z_NULL;
    zs.avail_in = 0;
    zs.next_in = Z_NULL;
    deflateInit2(&zs,
        Z_DEFAULT_COMPRESSION,
        Z_DEFLATED, -15,
        4, // memory level 1-9
        Z_DEFAULT_STRATEGY
    );

    std::string out;
    zs.avail_in = in.size();
    zs.next_in = (Bytef*)in.data();
    do
    {
        out.resize(out.size() + availOut);
        zs.avail_out = availOut;
        zs.next_out = (Bytef*)&out[zs.total_out];
        deflate(&zs, Z_FULL_FLUSH);
    }
    while(zs.avail_out == 0);
    out.resize(zs.total_out);
    deflateEnd(&zs);
    return out;
}

int main()
{
    auto s1 = compress("Hello, world!", 19);
    auto s2 = compress("Hello, world!", 30);

    if(s1 != s2)
        std::cerr << "Strings are different!";
}

Can't build zlib 1.2.7 for arm64

As described here http://stackoverflow.com/questions/22227029/how-to-build-zlib-for-arm64

configure from different directory

I have to compile zlib for a couple different platforms, and it would be nice to run configure from a different directory, but the configure script does not seem to support it and ends with below. This run is from OSX. It would be nice to be able to set a BASEDIR or even better to have configure automatically figure it out (autoconf?).

M01:zlib_temp tim$ zlib/configure
zlib/configure: line 31: zlib.h: No such file or directory
zlib/configure: line 32: zlib.h: No such file or directory
zlib/configure: line 33: zlib.h: No such file or directory
zlib/configure: line 34: zlib.h: No such file or directory
Checking for gcc...
Checking for shared library support...
No shared library support.
Building static library libz.a version with gcc.
Checking for off64_t... No.
Checking for fseeko... Yes.
Checking for strerror... Yes.
cp: zconf.h.in: No such file or directory
zlib/configure: line 441: zconf.h: No such file or directory
mv: rename zconf.temp.h to zconf.h: No such file or directory
Checking for unistd.h... Yes.
zlib/configure: line 456: zconf.h: No such file or directory
mv: rename zconf.temp.h to zconf.h: No such file or directory
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.
zlib/configure: line 720: zconf.h: No such file or directory
mv: rename zconf.temp.h to zconf.h: No such file or directory
Looking for a four-byte integer type... Found.
zlib/configure: line 766: Makefile.in: No such file or directory
zlib/configure: line 796: zlib.pc.in: No such file or directory

gzclose() for empty stream broken

Creating an empty stream with gzopen() followed by gzclose() is broken in 1.2.7. It creates an empty file, with no header (and so all gzip tools complain about unexpected EOF).

I believe this is broken by this commit: 53bfe01, which skips the finishing of the stream when state->size is zero.

Simple test program showing the problem:

#include <zlib.h> 

int main() { 
    gzFile f = gzopen("/tmp/gztest.gz", "w"); 
    gzclose(f); 
} 

The resulting file is 0 in length, whereas it should be 20 bytes (as it is with prior zlib versions).

zlib 1.2.8 always uses Windows 8 - level API

When compiled with VS2012, zlib ends up always using Windows 8 level API, e.g.: CreateFile2().

Looking at the code, I see that this regression has been introduced in the following commit: 5481269

More specifically, the define on line 31 always defines IOWIN32_USING_WINRT_API. That happens because WINAPI_FAMILY_PARTITION is defined, even on Windows 7.

I use Windows 7 and Visual Studio 2012 and experience errors since the library don't work (missing methods in kernel32.dll).

windowBits issue with Inflate for zlib format

Hi
I'm having a issue with window bits for inflate for the zlib format when window bits is set 8. A call to inflate returns a Z_DATA_ERROR with strm->msg set to: invalid window size.

Please note not that it should be in issue but I compress the data with deflateInit2()-->deflate()--> deflateEnd() with the same windowBits I pass to inflateInit2. So I'm trying to decompress data that is compressed with the same windowbits value.

the sequence of function calls is:
deflateInit2()
deflate()
deflateEnd()
inflateInit2()
inflate()
inflateEnd()

with windowBits set to 8 for deflateInit2() and inflateInit2() I get the error Z_DATA_ERROR with strm->msg set to: invalid window size.

with windowBits set to 8 for deflateInit2() and windowBits set to 0 inflateInit2() --> pass

gzseek broken on uncompressed streams on Windows

On Windows, when calling gzopen on a plain text file, reading some bytes (at least 1), then calling gzseek(f, 0L, SEEK_SET), all subsequent gzread calls fail (they return 0).

This does not happen when opening a compressed stream.

In contrast, calling gzrewind(f) (which should be equivalent to the above, according to the documentation in "zlib.h") always works. Also, on Linux and MacOSX there is no such issue.

I'm using zlib 1.2.8, and Windows 7 32bit (but I believe 64bit is also affected).

Here is a simple test file which shows the problem (assuming "tst.tmp.txt" is any plain text file):

#include <stdio.h>
#include <zlib.h>

int main(int argc,char *argv[])
{
    char buf[1000];
    int ret;
    gzFile f = gzopen("tst.tmp.txt", "rb");
    //gzFile f = gzopen("tst.tmp.gz", "rb"); // this works

    while (!gzeof(f)) {
        ret = gzread(f, buf, 1);
        if (!ret) {
            printf("READ FAILED\n");
        } else {
            printf("%c %i\n", buf[0], buf[0]);
        }
    }
    printf("EOF\n");
    ret = gzseek(f, 0, SEEK_SET);
    printf("SEEK RETURNED: %i\n", ret);
    while (!gzeof(f)) {
        ret = gzread(f, buf, 1);
        if (!ret) {
            printf("READ FAILED\n");
        } else {
            printf("%c %i\n", buf[0], buf[0]);
        }
    }
    printf("EOF\n");

    return 0;
}

when running the code, the first while loop succeeds, then gzseek returns 0, and subsequent gzreads fail. If the first while loop is omitted, the rest succeeds, but as long as a single byte is read the bug is observed.

CVE against pigz - need public release.

Howdy, I was hoping to get a statement from you on if you have plans to do a public release of pigz soon.

There are now 2 CVEs against pigz (CVE-2015-1191 and CVE-2015-1192) and we'd like to do an updated release of it, but given the amount of changes fdad1406 is sitting on top of, I'd be more comfortable taking a new release rather than taking on faith commit can safely be applied to your Oct 2013 release.

Thanks for the hard work,
Todd

zconf.h not splintable due to wrong #define constant

line 438 in my 1.2.7 zconf reads:

#if defined(LARGEFILE64_SOURCE) && -_LARGEFILE64_SOURCE - -1 == 1

shouldn't that be _LARGEFILE64_SOURCE in both uses?

line 446 reads:

#  if defined(Z_HAVE_UNISTD_H) || defined(LARGEFILE64_SOURCE)

again: wrong name of preprocessor constant?

Starting an organization to manage the project?

We heavily use this library at my current employer and foresee the possibility of making some enhancements. In the interest of giving back to the community and avoiding a proprietary fork, we'd like to make these changes available when they are ready.

It unfortunately appears as though the project hasn't been updated in ~2 years and there are many outstanding issues and pull requests. I may be speaking from ignorance, but has there been any consideration to starting an organization to oversea the maintenance of the project? Mark might either be too busy or have other things going on and it may be the best path forward for ensuring future changes make it in.

Now being an outsider to this community, I have no idea what group of people should be in charge of the organization. My hope is that asking the question would raise some interest and get the ball rolling.

Fatal error RC1 (VS2012 x64)

Hi,
the actual version from git (0b16609) doesn't compile on VS2012 x64. Most likely also other VS versions should be affected. The problem I get is:

2>win32\../zconf.h(398): error RC2177: constant too big
2>
2>win32\../zconf.h(400): error RC2177: constant too big
2>
2>win32\../zconf.h(402): error RC2177: constant too big
2>
2>c:\Data\Developx64\zlib-0b16609\zlib.dir\Release\RCa02516(41): fatal error RC1
116: RC terminating after preprocessor errors
2>

The compilation was done from git (as stated), using CMake in the source folder, and devenv for compilation. However, the compilation succeeded for the RV b06dee4.

Best,
Daniel

Direct API for SSE-optimized crc32

Hi Mark,

Thanks for a great project. I'm looking for efficient crc32 implementation for Intel 64.
I see that zlib has great implementation of algorithm based on PCLMULQDQ instruction.
However it is only used in deflate (crc_fold_copy). Generic crc32 computation function uses pure C table-based implementation.

Any specific reason why crc32 uses "unoptimised" version? Any way to change this in upstream or should I just branch zlib to implement fast crc32 based on crc_fold_copy?

Thanks,
Vlad

Can't build shared library of current develop

./configure will print "ztest27459.c:1:2: error: #error error" and continue to only build the static library.

Removing that check in the script will have the compilation abort with

/usr/bin/ld: unable to find version dependency `ZLIB_1.2.7'

Building 1.2.7 finishes fine.

Arch Linux x64, gcc 4.7.2, glibc 2.16.0, binutils 2.23.1

zLib 1.2.3.5 and newer do not work with z/OS datasets

I have been using zLib for the longest time as part of a z/OS (mainframe) application. Unfortunately I cannot update zLib beyond version 1.2.3.4 as changes zLib version 1.2.3.5 include use of I/O functions based on file descriptors instead of I/O functions based on streams. Functions like fopen() were replaced with open(). Looks like I have to reimplement all of gzlib.c, based on stream I/O functions.

Any reason why gzlib.c should keep using the file descriptor versions going forward?

iowin32.c, invalid MySetFilePointerEx() in version 1.2.8

Code has been factored from previous versions in a MySetFilePointerEx() method which use a dwMoveMethod argument. The code always use the hard value FILE_CURRENT instead of the argument.

Version 1.2.8:
`static BOOL MySetFilePointerEx(HANDLE hFile, LARGE_INTEGER pos, LARGE_INTEGER *newPos, DWORD dwMoveMethod)
{

ifdef IOWIN32_USING_WINRT_API

return SetFilePointerEx(hFile, pos, newPos, dwMoveMethod);

else

LONG lHigh = pos.HighPart;
DWORD dwNewPos = SetFilePointer(hFile, pos.LowPart, &lHigh, `**FILE_CURRENT**`);`

should be:
`static BOOL MySetFilePointerEx(HANDLE hFile, LARGE_INTEGER pos, LARGE_INTEGER *newPos, DWORD dwMoveMethod)
{

ifdef IOWIN32_USING_WINRT_API

 return SetFilePointerEx(hFile, pos, newPos, dwMoveMethod);

else

LONG lHigh = pos.HighPart;
DWORD dwNewPos = SetFilePointer(hFile, pos.LowPart, &lHigh, `**dwMoveMethod**`);`

There is a similar issue in the win32_seek64_file_func() method when calling MySetFilePointerEx()

version 1.2.8:
if (hFile) { LARGE_INTEGER pos; pos.QuadPart = offset; if (!MySetFilePointerEx(hFile, pos, NULL,FILE_CURRENT))

should be:

if (hFile) { LARGE_INTEGER pos; pos.QuadPart = offset; if (!MySetFilePointerEx(hFile, pos, NULL,dwMoveMethod))

Regards
Julien

Adding support for position independent code when compiling in static

Adding zlib as a static part of a shared library is not supported without editing the Makefile after executing ./configure (with --static), which could be a nice feature.

Misleading comment on deflateReset

Small issue but potentially misleading:

Comment to deflateReset states:

"The stream will keep the same compression level and any other attributes that may have been set by deflateInit2."

should be changed to

"The stream will keep the same compression level and any other attributes that may have been set by deflateInit2 or changed by deflateParams."

because otherwise one might think the compression level gets activly reset to the value set with deflateInit2, even if deflateParams was called in the meantime.

deflateParams() may not fully flush before changing compression

deflateParams() calls deflate(..., FLUSH) to flush the current block before changing compression levels. This may not fully flush the current block if the output buffer is too small. In that case, compression function may be changed in the middle of the current block, which may corrupt the deflated data.

Proposal for a solution:

• Change deflateParams(): if the deflate(..., FLUSH) is not able to fully flush the current block because it needs more output buffer, it shall not change the compression parameters but return a new return code (e.g. "Z_UNFINISHED_BLOCK") which the caller shall handle by repeating the call to deflateParams() with the same compression parameters and more output buffer.
• deflateParams() may also set z_stream->avail_in to 0 on its call to deflate(..., FLUSH) to temporarily defer new input till current block is fully flushed.
• All this would change the behaviour of deflateParams() incompatibly, so maybe a new deflateParams2() would be better instead.

This is related to a bug in the OpenJDK (https://bugs.openjdk.java.net/browse/JDK-8028804)

Access of uninitialized field strm->next_in

Running the following program linked to zlib 1.2.8...

#include <zlib.h>

int main(void)
{
    gzclose(gzopen("test.gz", "w"));
}

... produces the following output in valgrind:

==12682== Conditional jump or move depends on uninitialised value(s)
==12682==    at 0x4E37B1A: deflate (deflate.c:678)
==12682==    by 0x4E4742F: gz_comp (gzwrite.c:115)
==12682==    by 0x4E47E32: gzclose_w (gzwrite.c:562)
==12682==    by 0x4E4537C: gzclose (gzclose.c:21)
==12682==    by 0x400607: main (in /home/e/test/zlib/write_empty)

This occurs because deflate() on line 679 of deflate.c performs the following check:

(strm->next_in == Z_NULL && strm->avail_in != 0)

which accesses the field next_in before avail_in, even though gz_reset() only initializes avail_in, not next_in.

This is unlikely to make a difference, but it is still undefined behavior. This check should be changed to

(strm->avail_in != 0 && strm->next_in == Z_NULL)

windowBits documentation is unclear.

In reading the documentation (and looking at the source) for inflateInit2(), it is not clear to me when it is valid to set windowBits = 0. Is doing so valid only in inflate mode, or is it also valid for unzip and gunzip?
It's clear that windowBits = 0 isn't valid in the raw inflate case because headers are ignored.

I did some testing and it appears that windowBits = 0 only works in inflate mode. i.e., not gunzip, unzip, etc. Is this right?

static_l_desc, static_d_desc, and static_bl_desc in trees.c need const

Symbols static_l_desc, static_d_desc, and static_bl_desc in trees.c need const so they are stored in readonly memory in the final binary which can be shared by the OS between processes instead of readwrite memory which is unique to each process.

Is it possible to recover a lost preset dictionary?

Mark,

Thanks so much for maintaining this excellent piece of software. My question is a generic one. I use zlib delate and inflate with a preset dictionary to help increase compression for short strings. I was curious if it would be possible to recover a lost preset dictionary in some fashion? If a dictionary is lost, are there steps to recover it -- especially if one knows the expected decompressed output?

Thanks again Mark for all your work!

Converting to VC11

It would be lovely to have VS2012 solution file. In addition when converting from the VS2010 solution there is an annoying issue with def file when it fails with "syntax error in 'VERSION' statement". Microsoft says it must be in "VERSION major[.minor]" format.

Add support for building under Visual Studio 2013

The changes to support Visual Studio are bundled in pull request #77. (I tried to make two, independent pull requests, but messed up.)

Building Glib for armv8 Processor

I want to build Glib for ARMv8 (AARCH64) processor. I am getting the following error.Anyone having any idea about how to proceed .please help

Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../gobject/gtype.h:1531:7: note: in expansion of macro 'g_once_init_enter'
if (g_once_init_enter (&g_define_type_id__volatile))
^
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../gobject/gtype.h:1465:61: note: in expansion of macro '_G_DEFINE_INTERFACE_EXTENDED_BEGIN'
#define G_DEFINE_INTERFACE_WITH_CODE(TN, t_n, T_P, C) _G_DEFINE_INTERFACE_EXTENDED_BEGIN(TN, t_n, T_P) {C;} _G_DEFINE_INTERFACE_EXTENDED_END()
^
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../gobject/gtype.h:1446:47: note: in expansion of macro 'G_DEFINE_INTERFACE_WITH_CODE'
#define G_DEFINE_INTERFACE(TN, t_n, T_P) G_DEFINE_INTERFACE_WITH_CODE(TN, t_n, T_P, ;)
^
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../gio/gproxy.c:45:1: note: in expansion of macro 'G_DEFINE_INTERFACE'
G_DEFINE_INTERFACE (GProxy, g_proxy, G_TYPE_OBJECT)
^
In file included from /Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../android/glibconfig.h:9:0,
from /Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../android/config.h:500,
from /Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../gio/gproxy.c:23:
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../glib/gmacros.h:162:53: error: size of array '_GStaticAssertCompileTimeAssertion_1' is negative
#define G_STATIC_ASSERT(expr) typedef char G_PASTE (GStaticAssertCompileTimeAssertion, COUNTER)[(expr) ? 1 : -1] G_GNUC_UNUSED
^
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../glib/gmacros.h:159:47: note: in definition of macro 'G_PASTE_ARGS'
#define G_PASTE_ARGS(identifier1,identifier2) identifier1 ## identifier2
^
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../glib/gmacros.h:162:44: note: in expansion of macro 'G_PASTE'
#define G_STATIC_ASSERT(expr) typedef char G_PASTE (GStaticAssertCompileTimeAssertion, COUNTER)[(expr) ? 1 : -1] G_GNUC_UNUSED
^
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../glib/gthread.h:257:5: note: in expansion of macro 'G_STATIC_ASSERT'
G_STATIC_ASSERT (sizeof *(location) == sizeof (gpointer));
^
/Users/aagman/Desktop/glib/glib-2.35.8-for-android/glib-2.35.8/jni/../gobject/gtype.h:1547:7: note: in expansion of macro 'g_once_init_leave'
g_once_init_leave (&g_define_type_id__volatile, g_define_type_id); \

Missing zconf.h when using zlib as a git submodule (with CMake)

I used add_subdirectory() to integrate zlib into my CMake-controlled project. This works well, but there is the problem that git detects a "false update" and constantly asks me to commit a change that I didn't make.

Apparently, something in the build process renames zconf.h to zconf.h.included, so git detects the former as deleted and the latter as new.

Is there a way to work around that, or better yet, fix it ?

Is it possible to backport new functionality of gzbuffer() to boost file writing performance from zlib-1.2.5 to zlib-1.2.3?

Hi,

Is it possible to backport new functionality of gzbuffer() to boost file writing performance from zlib-1.2.5 to zlib-1.2.3? Updating to zlib-1.2.5 breaks libxml2 because of a known bug [1] of libxml2 relying on undocumented internals. libxml2 can not be updated.

I can not find any notes on this new functionality in changelog so can not say if it is 1.2.5.1 or 1.2.5.2 or other.

[1] http://osdir.com/ml/svn-commits-list/2010-01/msg05723.html

Thanks
Jan

int16 overflow

I've found a possible bug: if MAX_MEM_LEVEL = 9 and into deflateInit2_ (deflate.c:213) passed memLevel = 9 then hash_size (deflate.c:289) = 0 (1 << (memLevel + 7) = 0x00010000). Please check it, comrades.

error on compile to cortex-m4

export TARGETMACH=arm-none-eabi
export CROSS=arm-none-eabi
export CFLAGS="-mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16"
export CC=${CROSS}-gcc
export LD=${CROSS}-ld
export AS=${CROSS}-as

make distclean
rm -f .o *.lo *~
example minigzip examplesh minigzipsh
example64 minigzip64
infcover
libz. foo.gz so_locations
match.s maketree contrib/infback9/.o
rm -rf objs
rm -f *.gcda *.gcno *.gcov
rm -f contrib/infback9/.gcda contrib/infback9/.gcno contrib/infback9/_.gcov
cp -p zconf.h.in zconf.h
rm -f Makefile zlib.pc configure.log
[jmartins@pinguim zlib]$
[jmartins@pinguim zlib]$ ./configure --prefix=/home/jmartins/zlib/joao/
Checking for shared library support...
Building shared library libz.so.1.2.8 with arm-none-eabi-gcc.
Checking for off64_t... No.
Checking for fseeko... No.
Checking for strerror... No.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... No.
WARNING: vsnprintf() not found, falling back to vsprintf(). zlib
can build but will be open to possible buffer-overflow security
vulnerabilities.
Checking for return value of vsprintf()... Yes.
Checking for attribute(visibility) support... Yes.
[jmartins@pinguim zlib]$ make test
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -I. -c -o example.o test/example.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o adler32.o adler32.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o crc32.o crc32.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o deflate.o deflate.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o infback.o infback.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o inffast.o inffast.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o inflate.o inflate.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o inftrees.o inftrees.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o trees.o trees.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o zutil.o zutil.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o compress.o compress.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o uncompr.o uncompr.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o gzclose.o gzclose.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o gzlib.o gzlib.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o gzread.o gzread.c
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -c -o gzwrite.o gzwrite.c
ar rc libz.a adler32.o crc32.o deflate.o infback.o inffast.o inflate.o inftrees.o trees.o zutil.o compress.o uncompr.o gzclose.o gzlib.o gzread.o gzwrite.o
arm-none-eabi-gcc -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -DNO_FSEEKO -DNO_STRERROR -DNO_vsnprintf -DHAVE_HIDDEN -o example example.o -L. libz.a
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-exit.o): In function exit': exit.c:(.text.exit+0x16): undefined reference to_exit'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-sbrkr.o): In function _sbrk_r': sbrkr.c:(.text._sbrk_r+0xc): undefined reference to_sbrk'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-writer.o): In function _write_r': writer.c:(.text._write_r+0x12): undefined reference to_write'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-closer.o): In function _close_r': closer.c:(.text._close_r+0xc): undefined reference to_close'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-lseekr.o): In function _lseek_r': lseekr.c:(.text._lseek_r+0x12): undefined reference to_lseek'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-openr.o): In function _open_r': openr.c:(.text._open_r+0x12): undefined reference to_open'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-readr.o): In function _read_r': readr.c:(.text._read_r+0x12): undefined reference to_read'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-fstatr.o): In function _fstat_r': fstatr.c:(.text._fstat_r+0x10): undefined reference to_fstat'
/usr/lib/gcc/arm-none-eabi/5.3.0/../../../../arm-none-eabi/lib/armv7e-m/fpu/libc.a(lib_a-isattyr.o): In function _isatty_r': isattyr.c:(.text._isatty_r+0xc): undefined reference to_isatty'
collect2: error: ld returned 1 exit status
Makefile:170: recipe for target 'example' failed
make: *** [example] Error 1

deflate.c:887:54: runtime error: left shift of negative value -1

883 /* Make sure there is something to do and avoid duplicate consecutive
884 * flushes. For repeated and useless calls with Z_FINISH, we keep
885 * returning Z_STREAM_END instead of Z_BUF_ERROR.
886 */
887 } else if (strm->avail_in == 0 && RANK(flush) <= RANK(old_flush) &&
888 flush != Z_FINISH) {
889 ERR_RETURN(strm, Z_BUF_ERROR);
890 }

old_flush may be -1.. RANK(-1) produces undefined behaviour.

How to reproduce :

Build with gcc 5.2 and fsanitize=undefined
run make check.

hello world
zlib version 1.2.8 = 0x1280, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
deflate.c:887:54: runtime error: left shift of negative value -1
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
*** zlib test OK ***
hello world
zlib version 1.2.8 = 0x1280, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
deflate.c:887:54: runtime error: left shift of negative value -1
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
*** zlib shared test OK ***
hello world
zlib version 1.2.8 = 0x1280, compile flags = 0xa9
uncompress(): hello, hello!
gzread(): hello, hello!
gzgets() after gzseek: hello!
deflate.c:887:54: runtime error: left shift of negative value -1
inflate(): hello, hello!
large_inflate(): OK
after inflateSync(): hello, hello!
inflate with dictionary: hello, hello!
*** zlib 64-bit test OK ***

DEBUG pre-processor causing Symbolic errors when zlib compiled for Static Release and built in a Debug Targeted Xcode Project

Issue:

In Xcode when building a Debug targeted project the IDE adds -DEBUG to the target. This is causing many errors for libraries that use zlib in it's release form and include headers which expose these parts of zlib when that -DEBUG command is turned on (default action).

Causing Symbolic link errors:

Undefined symbols for architecture i386:
  "_z_error", referenced from:
      _deflate in deflate.o
      _deflate_stored in deflate.o
      _deflate_slow in deflate.o
      _fill_window in deflate.o
      _longest_match in deflate.o
      _longest_match_fast in deflate.o
      _check_match in deflate.o
      ...
  "_z_verbose", referenced from:
      _deflate_stored in deflate.o
      _deflate_fast in deflate.o
      _deflate_slow in deflate.o
      _check_match in deflate.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Found in many places in zlib:

• #ifdef DEBUG https://github.com/madler/zlib/search?utf8=✓&q=%23ifdef+DEBUG&type=Code
• #ifndef DEBUG https://github.com/madler/zlib/search?utf8=✓&q=%23ifndef+DEBUG&type=Code

Proposed change:

Swap out DEBUG for ZLIB_DEBUG

Codes for reserved symbols are rejected even if unused

If PKZIP_BUG_WORKAROUND is not defined, zlib will incorrectly reject inflating files that contain definitions for literal/length symbols 286 and 287 or distance symbols 30 or 31, even if they are unused.

As far as I can tell, this should be allowed by the standard. Ironically, this behavior means that simply taking the table used for mode 01 and using it in a mode 10 block will result in zlib rejecting the file.