Support for GZ format

[juliaio]

Support for GZ format

[hpoit]

Very painful process of understanding it and turning it into Julia for registry https://tools.ietf.org/html/rfc1952#page-6 (GZ format specification)

https://gist.github.com/hpoit/007ace3657b94fffd68530464a7d5ad9

If anyone knows of an existing code block representative of this please let me know! Otherwise I'm continuing on my gist.

About the magic number, not sure if it's true
"There is a magic number at the beginning of the file. Just read the first two bytes and check if they are equal to 0x1f8b."

From the spec
"ID2 (IDentification 2)
These have the fixed values ID1 = 31 (0x1f, \037), ID2 = 139
(0x8b, \213), to identify the file as being in gzip format."
I believe this is what I have to look for in io to detect GZ.

[hpoit]

Wait, what?

julia> using DataFrames

julia> airline_df = readtable("/Users/randyzwitch/airline/1987.csv.gz");

julia> size(airline_df)
(1311826,29)

julia> typeof(airline_df)
DataFrame  (use methods(DataFrame) to see constructors)

I think I just have to see how readtable identifies it, if it does.

[hpoit]

   # (2) Path is GZip file
    elseif endswith(pathname, ".gz")
        io = gzopen(pathname, "r")
        nbytes = 2 * filesize(pathname)

https://github.com/JuliaStats/DataFrames.jl/blob/19821b9060a691646114c064e1bc0b8e290c8f66/src/dataframe/io.jl#L932-L935

Hi @SimonDanisch, would this pass for registry and identifying a file is a GZIP?

[hpoit]

@SimonDanisch don't worry about it, I actually tried julia> df = readtable("/file.csv.gz") and it takes forever. Back to binaries.

[hpoit]

Okay, I think I've got the right question now: how do I read the first two bytes of a file to check if they are equal to 0x1f and 0x8b. That's what I'm trying to answer.

If the first two bytes are equal to those, then the file is in GZIP format.

[hpoit]

Hi @SimonDanisch, can I PR this? Is it correct?
add_format(format"GZIP", [0x1f, 0x8b], ".gz", [:Libz])
Let me test it.

[hpoit]

Hi @SimonDanisch, here goes the test, not sure I passed it

julia> Pkg.test("FileIO")
INFO: Testing FileIO
these tests will print warnings: 
Library "NotInstalled" is not installed but is recommended as a library to load format: ".not_installed"
Should we install "NotInstalled" for you? (y/n):
Library "NotInstalled" is not installed but is recommended as a library to load format: ".not_installed"
Should we install "NotInstalled" for you? (y/n):
invalid is not a valid choice. Try typing y or n
Should we install "NotInstalled" for you? (y/n):
this test will print warnings: 
Errors encountered while loading "test.multierr".
All errors:
   ErrorException("1")
   ErrorException("2")
Fatal error:
Test Summary: | Pass  Total
  FileIO      |  108    108
INFO: FileIO tests passed

julia> 

Should I just PR for you to see what I've done?

[timholy]

A PR would be great!