Comments (14)
Ah wow, I didn't know that was possible since even the official zip tooling doesn't support it.
Looks complicated. I can see why it's not obvious...
Let me see if I can get that library working with this one.
from archiver.
I mean, that lib works as far as I can tell, I don't know of another pure Go solution. You can try the latest commit I just pushed and see if it works for you. But if you're doing custom concurrency stuff maybe just best to use that lib directly...
from archiver.
@mholt I'll give it a go and see how things fare. We haven't rolled out to production yet and so are still coding things up.
from archiver.
I don't think it's possible with the zip format, even the zip
command line tool temporarily extracts the archive and re-creates it, apparently: https://wpguru.co.uk/2018/12/how-to-add-files-to-an-existing-zip-archive-on-macos-and-linux/
Tar is appendable because the format is basically just like delimiter-separated files, so you can easily add to it.
Closing, but feel free to continue discussion if needed.
from archiver.
@mholt I found this library which does exactly what I need. I found some github issues online where this was requested by the below developer to be included in OOTB golang but core devs pushed back saying there wasn't enough need for it but if the community really wanted it they could look at putting it in.
https://github.com/STARRY-S/zip
from archiver.
Hmmm... I cannot get it to create a zip file that isn't corrupted (according to unzip -vl test.zip
). Were you able to get it to work?
from archiver.
D'oh -- just kidding. I forgot to close the Updater.
I have a commit that works with a single test I did (:sweat_smile:) so you can use it if you want!
from archiver.
@mholt nice! Let me know how it goes and if you're able to optimize things one way or another or "make the impl better" whatever that means in this context :) Once you've got something I can give it a go here. We've basically got a highly parallel process which keeps open the "zip updater", and as files are finished, we're appending them to the zip and then shipping that off. I don't think that other library is maintained or active anymore so if you can whip something up I'd be more than happy to take yours ;)
from archiver.
@mholt with this new addition, and the way you implemented it, is it possible here to use a different compressor to get a smaller zip size? We're not necessarily concerned about how fast it takes to build the zip but more if we can get the zip size as small as possible. The files inside are all textual.
from archiver.
@cdancy Set the Compression
field of your Zip
struct: https://pkg.go.dev/github.com/mholt/archiver/v4#Zip.Compression (e.g. zip.Deflate
)
from archiver.
@mholt I'm having no luck :( No matter what I use I can't get the zip of the file to not be significantly larger than what I get with the starry-zip library. Same 4 files I'm using there and here results in a 57K versus 764K respectively. Maybe I'm doing something wrong?
// have to use zip.NewWriter otherwise library was complaining that zip was not valid
zipWriter := zip.NewWriter(zipFile)
zipWriter.SetComment("Hello, World!")
zipWriter.Close()
zipper := archiver.Zip{
Compression: flate.BestCompression,
}
err = zipper.Insert(context.Background(), zipFile, files)
require.NoError(t, err)
from archiver.
@cdancy I think you might need to use zip.Deflate
instead of flate.BestCompression
, which is probably a uint8 that isn't recognized, so maybe it treats it as "store" instead of "compress".
from archiver.
@mholt yeah I tried that as well but still nothing. When I open the zip written by the starry golang lib I see Defl:N
compression used by default but no matter what I do here I only ever see Stored
. I'm on mac-os so not sure if that plays into things at all.
L105342MUS:kadiv14103107114 dancc$ unzip -vl example.zip
Archive: example.zip
Hello, World!
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
687922 Stored 687922 0% 12-31-1979 00:00 2f6acde1 file1.txt
354646 Stored 354646 0% 12-31-1979 00:00 9150eeac file2.txt
77921 Stored 77921 0% 12-31-1979 00:00 be9df062 file3.txt
390349 Stored 390349 0% 12-31-1979 00:00 c6890ae4 file4.txt
-------- ------- --- -------
1510838 1510838 0% 4 files
L105342MUS:kadiv14103107114 dancc$ unzip -vl example-1.zip
Archive: example-1.zip
Hello, World!
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
77921 Defl:N 5752 93% 00-00-1980 00:00 be9df062 file1.txt
390349 Defl:N 28739 93% 00-00-1980 00:00 c6890ae4 file2.txt
354646 Defl:N 18367 95% 00-00-1980 00:00 9150eeac file3.txt
687922 Defl:N 40838 94% 00-00-1980 00:00 2f6acde1 file4.txt
-------- ------- --- -------
1510838 93696 94% 4 files
I don't know ...
from archiver.
I'm not sure if reusing the zip file after the zip writer wrote to it is a good idea. What if you have a fresh open file for the insert?
from archiver.
Related Issues (20)
- File.Open given to the handler for Tar.Extract can't be used after the handler has returned HOT 5
- cannot archiver rar format HOT 2
- why '7z' not supported by Command HOT 2
- Unable to decrypt 7-ZIP file with password (err: breader.ReadByte: no data!) HOT 1
- Unable to decrypt 7-ZIP file with password (err: breader.ReadByte: no data!) HOT 1
- Is 7z supported by 4.0 Library? HOT 1
- RAR signature not found HOT 4
- If overwrite is enabled for unarchive, and the file being overwritten is a symlink, then it tries to follow the symlink, rather than replace it HOT 12
- Return io.Reader from archiver.FileSystem.Open(filename) HOT 1
- func FilesFromDisk filenames arg, if path seperator is not system defined, pathInArchieve may be wrong path HOT 7
- Tar files created inside the directory they are compressing are not walkable HOT 5
- file 2G.file: writing data: archive/tar: write too long HOT 1
- Listing files of an Encrypted RAR Without Password Input HOT 1
- The CompressFile function does not work HOT 2
- Unarchiving read-only folders fails HOT 1
- Can't create archive with broken symlinks HOT 3
- rardecode: RAR signature not found HOT 8
- Compare and sync two directories HOT 1
- WIll there be support for just generic Archive and Extract to a dir? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from archiver.