corrupt zip file generation

[archiverjs]

corrupt zip file generation

[inolen]

To note, the corrupt archives all had the same file size, but a hex comparison showed that there were several places in the archive where data had been shifted around. I assumed that chunks of each stream were perhaps being incorrectly interleaved.

[ctalkington]

you may want to listen for the close event of output stream (to do your callback), as finalize doesn't guarantee that your streams have finished dealing with the data sent to them.

[ctalkington]

since archiver can handle being passed files right after each other and then dealing dealing with it internally 1 by 1. you really shouldn't need async at all on that part and there is a nice little helper for lazy loading streams that should ease open file limits.

https://npmjs.org/package/lazystream

[inolen]

@ctalkington I just had time to look into this finally, and you were 100% correct about waiting for events on the output stream. Thanks!

[inolen]

Actually, I lied, still hitting some issues. They appear to be the same as #38

[ctalkington]

yah, i really need to get a test environment for Mac so I can see whats going on as I have been unable to reproduce to this point.

[ctalkington]

bit of update from discoveries, this corruption seems to happen more when you are writing multiple zips at once and only when the source is a buffer and goes through zlib.

i messed with this some more, it appears if JUST the line to checksum is removed from the DeflateRawChecksum, it doesn't go crazy and start overwriting itself.

[ctalkington]

also to be noted this corruption also happens randomly and some times its none, one, two, or all the zips that end up corrupt. It almost makes me feel like its a global leak or there is some memory corruption going on from all the buffers being created. yet that doesn't really fit with the DeflateRawChecksum line removal fixing it...

EDIT: after the latest commits the DeflateRawChecksum line removal seems to have been a fluke as its back to erroring. I think its still related to something with buffers.

[ctalkington]

cc @ampgcat

[ctalkington]

during testing, it was found that a higher internal buffer highWaterMark seemed to reduce the occurrence of the issue cropping up. i will be releasing a bugfix in the next day with such a change as a temporary solution.

[ctalkington]

ok, im hoping the latest release eliminates the issue. i spent way too much trying to track down why deflate.pipe was causing such a random race condition so I've moved it back to event listeners for the time being.

[sokra]

It does fix my issue with corrupt zip files... Thanks 😄

[ctalkington]

closing. please open new issue if issue remains.

[reid]

Thank you @ctalkington for researching and fixing this! <3