Are you going to build support for streams?

[thejoshwolfe]

Are you going to build support for streams?

[thejoshwolfe]

No, because it's not really possible without sacrificing correctness. The master table of contents is at the end of a .zip file, not the beginning, so a streaming API would need to either buffer the entire .zip file (defeating the purpose of a streaming interface) or rely on the "local file header" information. The local file headers are explicitly denounced in the spec as being unreliable copies of the "central directory" information at the end of the file, so trusting them would be wrong.

Every streaming API I've seen in other libraries has made this mistake without realizing it or caring that they're wrong. It's pretty rare to see a .zip file where the unreliable info is actually incorrect, so test-driven confidence will probably lead most naive library aythors astray.

I haven't looked closely yet at the library you linked, but the above would be my first guess at how they implemented it.

[meenie]

No, you are spot on and most likely the reason why every single streaming unzipping library I've found has had issues. Looks like I'll be loading the whole file into memory to unzip it rather than streaming it straight through to Swift.

Thanks for your time!

[thejoshwolfe]

No problem. I'm happy for an opportunity to soapbox :)

[andrewrk]

If you're expecting large zip files you might want to buffer in a temp file instead of memory. Depends on your use case of course.

[meenie]

@andrewrk: I'm not expecting huge ones and it's not a feature that will be used all that frequently. I'm using Hapi.js as the web server and they give you the chance to either send the uploaded files to the buffer or a temp file.

@thejoshwolfe: Interestingly enough, I tried to use your Buffer option but I was getting an error from your dependency on fd-slicer. It was saying that the use of process.nextTick in a recursive manner would be deprecated in the next release of NodeJS. It only happened with a zip file (500kB) that has 600+ files. So I've changed over to the default .open() method for now and it's working great! Thanks a ton!!

[andrewrk]

I changed fd-slicer to use setImmediate instead of nextTick and published a new version of yauzl. Want to try again with the buffer interface?

[meenie]

@andrewrk I should have been more observant, I didn't realise you were the author of fd-slicer :).

Yes, after that update it's working great! Thanks for that. Now I don't have to worry about cleaning up any temporary files.

Cheers!

[meenie]

I was just checking your personal website and I saw you were into a bit of music. You might find one of my libraries quite fun. It's called Band.js. You can find it here: https://github.com/meenie/band.js. Here's an example of what it can do: http://plnkr.co/edit/LG20SL?p=preview

[thejoshwolfe]

As this issue has surfaced again, I'd like to add a few more thoughts. It's possible to support a streaming unzip api and still be correct if there's a check at the end of the stream that can report an error that basically tells you that everything you thought about this zip was a lie. The more I think about how this API would work and how clients would be supposed to use it, the less I like it. So the summary is that I still don't think supporting this feature is a good idea.