rayrobdod/png-inflate

A clean filter for git that decompresses png images

png-inflate's Issues

Don't assume that IEND is the last chunk of a file

For two purposes: first to avoid data loss when given an invalid PNG file (the file is already corrupt so what data is there to loose, but still), and because IEND is not the last chunk of an MNG file.

This probably involves detecting where an EOF is in ::png::Chunk::read and telling the difference between "there is no chunk to read" and "read an incomplete chunk"

Add support for MNG

MNG has a very similar format to PNG - just with additional chunks and a different magic number.

• Check whether any MNG chunks have compressed parts
• Check whether the MNG chunks that do not have compressed parts are safe to copy, either marked as such or would be safe for this algorithm to copy
• Figure out how to check whether two MNG files have the same semantics

Truncate output file

If the program output is to a file, a file already exists at that location, and the program output is smaller than the existing file contents, then the output file is not shrunk, and the file contains garbage data at the end.

Options:

1. Open output file as truncate. Simple, but if the input file and output file are the same file, have to read the file before opening the output file
2. File::set_len. Would have to be done at the end of the program, and need to be a special case when closing a file (whereas all current special casing is when opening the file)(using a deconstructor, maybe?)
3. Two-stage file saving. Even more steps than set_len - especially if attempting to preserve file permissions, does rust have an atomic "create temporary file" method - but has the best resistance to data loss

Add support for APNG chunks

APNG is an extension to PNG adding several non-standard chunks (although the current w3c editor's draft of the PNG spec mentions the APNG chunks)

• Check which APNG chunks have compressed parts
• Check whether the APNG chunks that do not have compressed parts are safe to copy, either marked as such or would be safe for this algorithm to copy
• Figure out how to check whether two APNG files have the same semantics

chunks longer than 2^31 bytes

Although encoders and decoders should treat the length as unsigned, its value shall not exceed 2³¹-1 bytes.

-- the PNG spec (Second Edition)

• An oversized inflated IDAT (or fdAT - but this would require renumbering all the APNG chunks, which is the same problem as merging adjacent fdATs) can be split;
  • this could mean that chunk inflating is no longer always a one-to-one transformation
  • this could mean that Chunk struct could hold data that is illegal to write - which can already happen through the ConcatinateIdats step, or theoretically since Chunk doesn't currently encapsulate its typ field. Being idiomatic would probably require forcing Chunk to always be a valid-to-write chunk. (have separate png-legal Chunk and a not-neccessarily-legal Chunk types? seems overengineered). And this would require adding a SplitIdats step.
• An oversized inflated zTXT, iCCP or iTXT would be unrecoverable. Those chunks can't be split with a preserved meaning, and an algorithm to inflate as much as possible without going over 2MB is not worth the time or effort.
• any other chunk wasn't decompressed, and thus was an input error to begin with, but could be checked at read time

Also, might want to be able to integration-test this without having to commit a 2MB file into the source repository. Which would mean having to let the derive crate generate files in addition to listing existing paths.

Check whether this filter does result in a smaller git repository

I mean, that is the only practical use of this I've thought of so far.

Remember to disable the computer's sleep mode before trying this again.

Maybe also compare against a tree-filter that recompresses with pngout, or one that uses both pngout and this.

https://github.com/eliatlarge/FEMultiPlayer-V2 and https://github.com/thatswhatyouget/tpp-progress are probably two repositories with a lot of images (even if the images are dwarfed by the music files and generated js files respectively), but their images are all poorly compressed anyway. Maybe one of the pret ones.

#! powershell -File
$CACHE=$env:TMP + '\.hash_inflate\'
$FILTER='C:\Users\Raymond\AppData\Local\Temp\rust\release\png_inflate.exe'

ForEach ($file in $args) {
	$hash = $(Get-FileHash -LiteralPath $file).Hash
	$hashpath = $($CACHE + $hash)
	if ($(Test-Path -LiteralPath $hashpath)) {
		Copy-Item -LiteralPath $hashpath -Destination $file -Force
	} else {
		&"$PNG_FILTER" $file $hashpath
		Copy-Item -LiteralPath $hashpath -Destination $file -Force
	}
}

Set-ExecutionPolicy Unrestricted -Scope Process
cd $env:TMP
git clone https://github.com/thatswhatyouget/tpp-progress.git tpp-progress6
cd tpp-progress6
&git filter-branch --tree-filter "find -name '*.png' -exec C:/Users/Raymond/AppData/Local/Temp/filter.ps1 {} +"
cd ..
git clone file://C:/Users/Raymond/AppData/Local/Temp/tpp-progress6 tpp-progress7
# Then check the sizes of the `.git` drectories of a fresh clone of the original repository with the newly produced one