dol-sen/pydecomp

pydecomp's Issues

squashfs decompression passes invalid options to unsquashfs

When trying to decompress a squashfs file on Linux, unsquashfs is passed xattr-related flags that it cannot handle and fails to extract any files, printing usage help text.

https://github.com/dol-sen/pyDeComp/blob/master/DeComp/definitions.py#L379
https://github.com/dol-sen/pyDeComp/blob/master/DeComp/definitions.py#L70

rsync does not preserve hard links and does not provide other_options

Hi. COMPRESS_DEFINITIONS does not allow rsync to have other_options. Still, it uses only -a option that loses acls, xattrs, and hard-links during copy:

       --archive, -a            archive mode is -rlptgoD (no -A,-X,-U,-N,-H)

       --acls, -A               preserve ACLs (implies --perms)
       --xattrs, -X             preserve extended attributes
       --atimes, -U             preserve access (use) times
       --crtimes, -N            preserve create times (newness)
       --hard-links, -H         preserve hard links

For example, missing hard-links increase the size of directories more than 20x:

22M	/var/tmp/catalyst/tmp/hardened/livecd-stage1-core2-hardened-nomultilib-openrc-2023.03.04/usr/libexec/git-core/
502M	/var/tmp/catalyst/tmp/hardened/livecd-stage2-core2-hardened-nomultilib-openrc-2023.03.04/usr/libexec/git-core/

I'm looking for best solution here. Probably easiest solution will be to add H, A, and X options for rsync. Will you accept such change? Or should we add other_options for args definition, to allow catalyst check and set additional options and set these options there?

py3 compat handling w/popen results

the decomp module in contents.py doesn't work well with py3. this bit:

        try:
            proc = Popen(_cmd, stdout=PIPE, stderr=PIPE)
            results = proc.communicate()
            result = "\n".join(results)

will fail like so:

  File "/usr/lib64/python3.4/site-packages/DeComp/contents.py", line 66, in contents
    self._map[mode].cmd, self._map[mode].args, verbose)
  File "/usr/lib64/python3.4/site-packages/DeComp/contents.py", line 110, in _common
    result = "\n".join(results)
TypeError: sequence item 0: expected str instance, bytes found

the return of communicate is bytes in py3, so trying to use a string doesn't work. changing it to b"\n" will make the join work, but the result will still be bytes, so you'll need to also decode it. maybe something like:

            result = b"\n".join(results).decode("utf8")

squashfs_xz removes bcj filter when it is desired

amd64 and arm64 don't have bcj filters which match their name, but there are appropriate filters for those architectures.

amd64 should use the x86 filter
arm64 should use the arm filter