Comments (7)
I know about the feature, it is an additional filter in the xz format. It increases redundancy in the x86 code. I haven't really looked into it and nowadays it would be more important to implement the feature for AMD64 and ARM.
from xz.
The --x86
option is also effective for amd64 because their instruction sets are similar. A --arm
option would be really cool too!
from xz.
I'd really like to see --x86
added. As @rjoleary noted, this works on both x86 and amd64.
@ulikunitz what would be needed to get this support added?
ARM (and nowadays RISC-V) would also be nice, but I don't think there's an implementation in any language for RISC-V, and I don't think there's one in go for ARM.
from xz.
The infrastructure for filters is in the code. The filter interface in format.go must be implemented. The only implementation is currently lzmaFilter in lzmafilter.go.
What would help would be a write-up, how the filter works with examples. The xz spec says only "These filters convert relative branch, call, and jump instructions to their absolute counterparts in executable files. This conversion increases redundancy and thus compression ratio." Note that we have to maintain full compatibility to the xz tools.
When I read the spec, I wondered whether I would really need to identify executable segments in the COFF and ELF formats, which have 32 and 64 bit variants and how I would track the program counters. It is probably much simpler, but at the time it appeared quite complex to me and I would need to read source code to understand what the semantics are. It takes much less time to implement something from a good specification than to understand semantics from source code, which will be required here to maintain compatibility.
BTW RISC-V is not in the spec, so we would need to do something incompatible that would not be supported by the xz tools.
from xz.
It takes much less time to implement something from a good specification than to understand semantics from source code, which will be required here to maintain compatibility.
I assume the code linked by @rjoleary is already tested and encodes/decodes identically to the filter in liblzma, but I could be wrong. @rjoleary do you happen to remember what your implementation came from? A translation of liblzma code, a spec, or ???
from xz.
Fiano's Go implementation is a port of https://github.com/tianocore/edk2/blob/00f5e11913a8706a1733da2b591502d59f848a99/BaseTools/Source/C/LzmaCompress/Sdk/C/Bra86.c which came from liblzma which is in the public domain.
The filter is reversible even when run on arbitrary non-x86 data. The compression benefits are small (<10%), but cheap.
I'm not too sure exactly how it works. You might want to test it more than we did with different inputs and code coverage. It seems to have gone out of fashion in newer UEFI Firmware Volumes I've seen.
from xz.
Thanks for the link. The modification is indeed simpler than I expected, but still will require me to re-engineer semantics.
I'm currently working on a complete rewrite of the Lempel-Ziv compression and use a much simpler/faster LZ sequence generator, so don't expect the feature to be supported soon.
from xz.
Related Issues (20)
- Checksum None is valid HOT 4
- expose blockreader HOT 4
- Current maturity of project (and other semantics) HOT 3
- Achieving maximum xz compression HOT 2
- memory leak HOT 2
- [SECURITY] Implementation of readUvarint vulnerable to CVE-2020-16845 HOT 17
- How to use multi CPU work for compression? HOT 3
- Panic with invalid input HOT 2
- missing match limit, was "lzip" HOT 11
- How to write compress & decompress data in file? HOT 2
- Don't worry about this lssue HOT 1
- Missing common APIs like Reader:Close() Writer:Flush() HOT 3
- Out of Memory bug when using a large reader HOT 3
- high allocation ratio HOT 3
- Expose `processFile` function HOT 2
- Plan for rewrite branch HOT 4
- How i can compressed folder? HOT 1
- Equivalent of FastBytes? HOT 2
- Extract files from NSIS installer? HOT 1
- Add statement to README about xz backdoor HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from xz.