martinellimarco/t2sz

Compress a file into a seekable zstd with special handling for .tar archives

t2sz's Issues

Threading does not appear to work

Using the latest git commit, the -T option simply doesn't appear to work. t2sz will use only a single thread according to htop no matter how high I set it. This happens with -s and -S set to both high or low values, as well as without them set.

I am on Solus using zstd 1.5.2.

Memory corruption

git clone --depth=1 --single-branch https://github.com/stedolan/jq
tar -cf jq.tar -- jq
t2sz -vfo jq.tar.zst -- jq.tar

+ <null>
# END OF BLOCK (512)

+ <null>
# END OF BLOCK (512)

munmap_chunk(): invalid pointer
[1]    164384 abort (core dumped)  t2sz -vfo jq.tar.zst -- jq.tar

# OR:(sometimes): free(): invalid pointer

──────
(gdb) bt
#0  0x00007ffff7d2eef5 in raise () from /usr/lib/libc.so.6
#1  0x00007ffff7d18862 in abort () from /usr/lib/libc.so.6
#2  0x00007ffff7d70f38 in __libc_message () from /usr/lib/libc.so.6
#3  0x00007ffff7d78bea in malloc_printerr () from /usr/lib/libc.so.6
#4  0x00007ffff7d7901c in munmap_chunk () from /usr/lib/libc.so.6
#5  0x00007ffff7d7dcdb in free () from /usr/lib/libc.so.6
#6  0x00005555555552bc in main (argc=<optimized out>, argv=<optimized out>) at /home/user/.cache/aurutils/airy/t2sz-git/src/t2sz/src/t2sz.c:372
──────
(gdb) f 6
#6  0x00005555555552bc in main (argc=<optimized out>, argv=<optimized out>) at /home/user/.cache/aurutils/airy/t2sz-git/src/t2sz/src/t2sz.c:372
372         free(ctx->outFilename);
──────
(gdb) l
367             }
368         }
369
370         compressFile(ctx);
371
372         free(ctx->outFilename);
373         free(ctx);
374
375         return 0;
376     }
──────
(gdb)

==167002== Invalid free() / delete / delete[] / realloc()
==167002==    at 0x483F9AB: free (vg_replace_malloc.c:538)
==167002==    by 0x1092BB: main (t2sz.c:372)
==167002==  Address 0x1fff000109 is on thread 1's stack

Dictionary support?

I was wondering if t2sz supports dictionaries.

CMakeLists does not respect CMAKE_INSTALL_PREFIX

Line 15 in CMakeLists hardcodes an install prefix which does not respect the CMAKE_INSTALL_PREFIX variable.

t2sz/CMakeLists.txt

Line 15 in 72bf850

install(TARGETS t2sz DESTINATION "/usr/bin/")

Changing this to:

install(TARGETS t2sz)

corrects the problem and allows cmake to utilize the specified install prefix.

Allow using stdin

Currently t2sz mmaps the input file, so it has to be a physical file. This makes t2sz difficult to use in pipelines, where some stream is generated and needs to be compressed, but we don't want the uncompressed stream persisted to disk.

There's no particular reason we couldn't just read blocks into a buffer, though.

Usability for non-tar archives

It might be nice to be able to run t2sz against non-tar-format files, eg: t2sz image.ext4 -o image.ext4.sz. It'd be useful for mounting disk images with ratarmount, for example. My tool pixz allows this sort of thing for writing indexed xz files, and I'd love to see the same functionality for zstd.

Rounding error

I was compressing a 250GB tar archive with -s 50K and it oddly enough it seems to have stopped at 4GB:
ERROR: Invalid tar header. If this is not a tar archive use raw mode (-r)
Maybe the same problem as with martinellimarco/libzstd-seek#3?

t2sz/src/t2sz.c

Lines 74 to 75 in 84dab3b

	uint32_t compressedSize;
	uint32_t decompressedSize;

t2sz/src/t2sz.c

Line 104 in 84dab3b

uint32_t numberOfFrames;