Allow for option to have .gz be scanned as external SHA1

[Obiwantje]

Allow for option to have .gz be scanned as external SHA1

[uwedeportivo]

If you use the flag -include-zips=true, it will do the same to .gz files as well

[Obiwantje]

This seems not to work. Below is the output of a scan of 3 files (2x .gz and 1x .sha) you can see the .sha file being indexed - but the 2 .gz are not (in this case the .gz files are NOT gzipped files)

Log file created at: 2014/02/23 11:23:30
Running on machine: thejay-EX58-UD5
Binary: Built with gc go1.2 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0223 11:23:30.684304 32213 db.go:120] Loading DB
I0223 11:23:30.684709 32213 kv.go:117] Loading Generation File
I0223 11:23:30.685018 32213 kv.go:124] Loading Dats DB
I0223 11:23:30.690905 32213 kv.go:131] Loading CRC DB
I0223 11:23:30.696786 32213 kv.go:138] Loading MD5 DB
I0223 11:23:30.704757 32213 kv.go:145] Loading SHA1 DB
I0223 11:23:30.714209 32213 kv.go:152] Loading CRC -> SHA1 DB
I0223 11:23:30.719560 32213 kv.go:159] Loading MD5 -> SHA1 DB
I0223 11:23:30.723947 32213 db.go:127] Done Loading DB in 0s
I0223 11:23:30.723995 32213 depot.go:96] Depot init
I0223 11:23:30.724009 32213 depot.go:106] establishing size of /mnt/roms02/roms
I0223 11:23:30.761062 32213 depot.go:114] Initializing Depot with the following roots
I0223 11:23:30.761102 32213 depot.go:118] root = /mnt/roms02/roms, maxSize = 34TB, size = 28TB
I0223 11:23:30.761108 32213 depot.go:123] Depot init finished
I0223 11:23:30.761162 32213 service.go:98] Service init
I0223 11:23:30.761176 32213 service.go:109] Service init finished
I0223 11:23:41.034077 32213 service.go:627] service starting archive
I0223 11:23:41.034226 32213 service.go:632] starting progress broadcaster
I0223 11:23:41.034865 32213 worker.go:227] starting archive roms
I0223 11:23:41.034885 32213 worker.go:250] initial scan of /home/thejay/romba/include-zips=true to determine amount of work
I0223 11:23:41.034897 32213 worker.go:250] initial scan of /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms to determine amount of work
I0223 11:23:41.054659 32213 worker.go:119] visiting path /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/amigasrc.cpio.gz, current common root is
I0223 11:23:41.054685 32213 worker.go:127] new current common root is /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/amigasrc.cpio.gz
I0223 11:23:41.054691 32213 worker.go:119] visiting path /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/pubsrc.cpio.gz, current common root is /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/amigasrc.cpio.gz
I0223 11:23:41.054702 32213 worker.go:127] new current common root is /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms
I0223 11:23:41.054706 32213 worker.go:119] visiting path /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/pubsrc.cpio.gz.sha, current common root is /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms
I0223 11:23:41.054713 32213 worker.go:127] new current common root is /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms
I0223 11:23:41.054728 32213 worker.go:259] found 3 files and 23MB to do. starting work...
I0223 11:23:41.054839 32213 worker.go:192] starting worker 0 for archive roms
I0223 11:23:41.054847 32213 worker.go:198] processing file /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/amigasrc.cpio.gz
I0223 11:23:41.054929 32213 worker.go:192] starting worker 1 for archive roms
I0223 11:23:41.054937 32213 worker.go:192] starting worker 2 for archive roms
I0223 11:23:41.054941 32213 worker.go:192] starting worker 4 for archive roms
I0223 11:23:41.054944 32213 worker.go:192] starting worker 7 for archive roms
I0223 11:23:41.054939 32213 worker.go:192] starting worker 5 for archive roms
I0223 11:23:41.054955 32213 worker.go:192] starting worker 6 for archive roms
I0223 11:23:41.054944 32213 worker.go:192] starting worker 3 for archive roms
I0223 11:23:41.054964 32213 worker.go:221] exiting worker 6 for archive roms
I0223 11:23:41.054959 32213 worker.go:221] exiting worker 5 for archive roms
I0223 11:23:41.054958 32213 worker.go:198] processing file /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/pubsrc.cpio.gz
I0223 11:23:41.054961 32213 worker.go:198] processing file /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/pubsrc.cpio.gz.sha
I0223 11:23:41.054970 32213 worker.go:221] exiting worker 4 for archive roms
I0223 11:23:41.054965 32213 worker.go:221] exiting worker 3 for archive roms
I0223 11:23:41.054974 32213 worker.go:221] exiting worker 7 for archive roms
I0223 11:23:41.092012 32213 kv.go:432] indexing rom pubsrc.cpio.gz.sha
I0223 11:23:41.092435 32213 kv.go:443] declaring crc e0e6f913 -> sha1 4ee1ece0c8fb42fd946f802b5b7aefdf616e2571 ampping
I0223 11:23:41.092454 32213 kv.go:453] declaring md5 8dca28778eaecfeb0e80c24ff149e05d -> sha1 4ee1ece0c8fb42fd946f802b5b7aefdf616e2571 ampping
E0223 11:23:41.103796 32213 worker.go:205] failed to process /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/amigasrc.cpio.gz: cgzip: failed to inflate (-3): incorrect header check
I0223 11:23:41.104005 32213 worker.go:188] to be implemented: handle file that caused error: /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/amigasrc.cpio.gz
I0223 11:23:41.104019 32213 worker.go:221] exiting worker 0 for archive roms
E0223 11:23:41.104025 32213 worker.go:317] master found worker error cgzip: failed to inflate (-3): incorrect header check
E0223 11:23:41.131395 32213 worker.go:205] failed to process /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/pubsrc.cpio.gz: cgzip: failed to inflate (-3): incorrect header check
I0223 11:23:41.131458 32213 worker.go:188] to be implemented: handle file that caused error: /mnt/roms02/Seedbox/data/!0000Feed to ROMba/!random roms/pubsrc.cpio.gz
I0223 11:23:41.131470 32213 worker.go:221] exiting worker 1 for archive roms
E0223 11:23:41.131501 32213 worker.go:317] master found worker error cgzip: failed to inflate (-3): incorrect header check
I0223 11:23:41.242071 32213 worker.go:221] exiting worker 2 for archive roms
I0223 11:23:41.243441 32213 worker.go:333] Worker errors happened. First error was cgzip: failed to inflate (-3): incorrect header check.
I0223 11:23:41.243459 32213 worker.go:336] Done.
I0223 11:23:41.243495 32213 worker.go:363] finished archive roms
total number of files: 3
number of files processed: 3
number of files with errors: 2
total number of bytes: 23MB
number of bytes processed: 23MB
elapsed time: 0s
throughput: 110MB/s
E0223 11:23:41.243508 32213 service.go:650] error archiving: cgzip: failed to inflate (-3): incorrect header check
I0223 11:23:41.243519 32213 service.go:662] service finished archiving
I0223 11:23:41.243526 32213 service.go:638] stopped progress broadcaster