Change compression from LZMA to ZStandart

[romaniakovlev]

Change compression from LZMA to ZStandart

[RomanIakovlev]

Hi, I've had a quick look at this issue, and it seems the change is a bit more involved than it looks on the surface. ZStandard is a compression algorithm, not an archival format, that is, it can't organize multiple files into a single archive. The approach used now, 7Zip, supports both archival and compression (to be precise, it supports a collection of different compression formats, but not ZStandard).

Now, in order to use ZStandard compression method, it's required to come up with an alternative archival method, probably tar or something similar, to put all individual GeoJSON features into a single file first, and then apply compression to that file.

It's all definitely doable, and might be beneficial for the Timeshape project, if artifact size doesn't increase substantially, and decompression time is decreased, so let's do it! I obviously can't promise any timelines here, but if you're interested in giving it a try, I'll gladly accept a pull request from you. If you're interested in giving it a stab, and if you need guidance on the project setup or any other details about the project, besides what's already written in the README, please ask here, and I'll do my best to provide clarification.

[juarezr]

I did'nt read the source code of timeshape and did'nt know about it.
I have imagined that geojson was compressed first with lzma and then stored with no compression in a folder inside the library jar/zip file.
But now I found that my assumption was false. :)

[RomanIakovlev]

Ok, I've tried using ZStandard and I must say I'm positively surprised by its performance! I was expecting it to have a better decompression time, but in practice it also offers better compression time (mostly irrelevant for Timeshape users) and compression rate (5% better). Decompression time, at least on my laptop, has dropped significantly, and now TimeZoneEngine#initialize method takes only 1300 milliseconds for Tar+ZStandard, versus 4300 milliseconds for 7z+LZMA4. This new approach is definitely worth using, so I'll make a release with it.

[RomanIakovlev]

I've just released 2018d.6 that uses ZStandard for an increased compression rate and faster speed. Could you please give it a try and close this issue if all is fine on your side?

[RomanIakovlev]

Ok, I'll go ahead and close this one now.

[juarezr]

I will test the new version for getting the improvements.
Thanks for the hard work.

[juarezr]

It worked fine. 👍
Wondering if replacing Protobuff with Capn'Proto can give any performance while querying and if the performace/memory tradeoffs worths.

[RomanIakovlev]

Interesting regarding Cap'n Proto, I've created #29 to track this.