Starting an organization to manage the project?

[madler]

Starting an organization to manage the project?

[Dead2]

I havent really announced this officially yet, but I have been working on a fork of zlib. With the intentions being to get rid of legacy code (Require C99 and drop workarounds for ancient compilers/os), enable new optimizations (such as compiler intrinsics) and make it easier for contributers to contribute code. Currently I have merged in changes from both Intel and Cloudflare, and a couple more. Also lots of legacy code has been removed and some changes have been done to improve readability of the code, and so on.. Read the commit log for (a lot) more details.

Among the main challenges currently are copyright related, as I do not really know what more/less will be required. I have attempted contacting Mark Adler, but he has not answered my email (probably 6 months ago) and the mailing list archive was also down at that time and every time I have checked since.

Anyhow, if anybody would like to review, test and/or contribute to the fork then I would be glad to receive pull requests and/or comments. I have not had much time to work on zlib the last few months due to a lot going on at work, but I hope to get back to doing some more improvements soon.

https://github.com/Dead2/zlib-ng

I do not remember the numbers right now, but compression on x86-64 and armv6 (raspberry pi) was seeing major speed improvements compared to stock zlib (20-50% or so). More testing needs to be done, but it should be pretty stable at this point and it should be a win on all supported platforms in theory.

Sorry for highjacking the thread, but it seems a good idea to allow others to check out my fork.

PS: I do not really have any intentions of replacing zlib, but I hope to provide a good performance-oriented fork with a more community-driven approach. zlib itself will hopefully remain the extremely portable reference implementation.

[madler]

Ah, ok. I think I know how to fix the deflate bug. It's on my list.

[ProgramMax]

We've added code coverage measurement & reporting on Chrome's testing infrastructure. You can see an example here:
https://chromium-coverage.appspot.com/reports/558266/linux/chromium/src/third_party/zlib/report.html

We likely won't be getting the coverage up to 100% though. There are parts of zlib that Chrome doesn't currently intend to use.

We have also added some fuzz tests. Over time we'll add more, as well as additional unit tests. Zlib already has some unit tests but those aren't currently running on Chrome's testing infrastructure.

[ProgramMax]

There are a handful of zlib alternatives available. Each have their pros and cons.

But so many things rely on zlib itself. I think in an ideal world we'll keep zlib active and healthy.

[rhuijben]

I think http://www.zlib.net/ contains some pointers on how zlib is maintained and contact information to reach the maintainers.

[dandesousa]

@Dead2 , this is great. I think this is something we might be interested in and I'll forward the thread along to some of my colleagues who are looking into making the changes.

[madler]

I support the idea of a group of dedicated people to maintain zlib.

[rhuijben]

I hope the requirement for your fork would not be 'very strict C99' as that would block usage on most Windows systems. The Visual C++ compiler supports most C99 constructs, but not all. Most C99 based projects that I know require that it also stays compatible with that toolchain.

I would very much welcome new activity in this project... There are many small and larger improvements that could really use some central review. (And there are some bugs in current 'contrib' code that make me recommend against using those optimizations).

The current license of zlib is very liberal, I would hope that it stays that way. Switching to something viral/GPL based would require many current zlib users to stick with the current code, as those license are not compatible with the rest of our code. (In my case the entire code base is Apache 2 licensed, which is incompatible with plain GPL)

[Dead2]

rhuijben: I have gone the way of requiring C89 and preferring the use of C99 features where possible (the MS compiler luckily often includes non-standard ways of doing the same or nearly the same thing, requiring only minor workarounds).

Copyright wise my questions are more along the way of whether the header of the files can be modified to remove the copyright with name, since that header really becomes a bit of misinformation when the file has had major modifications. Having a central file of contributor names derived from the git commit log would be my preferred method.

License can likely not be changed from zlib (unless Mark Adler does this himself), but I really do not see any reason to do that anyway. Sure there is a lot of good GPL code that could be merged in, but I see no significant reasons to do so.

madler: That is good news :)
I would volunteer to become part of that group. Possibly you could head such a group and gradually distribute responsibility to the group or something along those lines?

I do not think zlib-ng should be merged into zlib though (just to make my opinion clear), since zlib-ng does cut out lots of legacy support and has some major changes. But that could be done in the future if that is what the community wants.

[tbeu]

So what is the state of the zlib project by now?

• There were 33 commits in development branch since April 2013 when v1.2.8 was released - so one commit by month on average.
• There are 50 open pull requests - some with overlapping content. Especially #120 shows how serious the zlib development is considered.

I am just curious if we ever can exepct a new official release?

[mtl1979]

Most of the fixes in development branch have been feature additions or bug fixes for minor operating systems... I'm assuming we need some major issue to be fixed before it warrants release of v1.2.9.

[vinniefalco]

@Dead2 I'm working on a header-only port of ZLib to modern C++ (no macros, object oriented). It supports raw deflate, and its for my HTTP/WebSocket library:
https://github.com/vinniefalco/Beast/tree/zlib/include/beast/zlib

[ProgramMax]

I think having a group of dedicated people to maintain zlib would be wise.

Zlib is a critically important project. A vast amount of the modern world uses it. But since it isn't a funded project, it cannot have a full-time developer. Rather, it needs to be faithfully maintained in free time, which is limited. More people would mean more total free time.

For what it is worth, @Adenilson and I have been trying to make sure any contributions coming in from Chromium are as easy-to-digest & accept as possible. We know the resources for review are limited and we want to pre-review strictly. I'm also extending this to the rest of Google where possible.

I suggest others who are contributing to zlib do the same.
Try to keep changes within a subdirectory of contrib/ if possible. This might mean copying files into contrib/, making changes to your copy, and providing a CMake or similar build file that uses your copied file instead. Make small, isolated pull requests.

As a community, our limiting resource is review time. The easier it is to understand and accept the pull request, the less of our limited review time it will use. Please treat our limited resource as precious. :)

[Adenilson]

I second what @ProgramMax said, we should try to help the mainstream project as much as we can.

[vinniefalco]

Integrate codecov and Travis CI, add tests, get coverage up to 100%.

[vinniefalco]

Maybe I am biased, but I think that your collective efforts would be better spent on improvements made to Beast's header-only port of Zlib, since it is now part of Boost (and I intend to write a paper proposing it for the C++ standard). My port is modern and uses C++11 idioms (for example, by replacing macros with lambdas).

[vinniefalco]

The decompressor is broken for windowBits==8 maybe start there

[madler]

How so?

[vinniefalco]

grr... I always get deflate and decompress confused:
https://github.com/madler/zlib/blob/master/deflate.c#L303

[Adenilson]

@vinniefalco does it using lambdas somehow improve the generated assembly (or the performance) than using macros?

While I personally like C++11/14, it is important to remember that there are quite a few cases (e.g. linux kernel, ARM embedded devices, etc) where we don't have access to a recent C++ compiler or can't use modern C++ features.

The use of plain C in zlib is a nice feature that improves portability.

[vinniefalco]

it is important to remember that there are quite a few cases...where

Yeah in those cases the "plain C" version might be better suited.