fruits.parquet generated by test/integration.js is unreadable by Hadoop parquet-tools 1.9.0

[ironsource]

fruits.parquet generated by test/integration.js is unreadable by Hadoop parquet-tools 1.9.0

[ZJONSSON]

Also - if you want to avoid the headache of building and configuring parquet-tools you can simply add this to your .bashrc (or paste in console) and use docker to take care of everything.

parquet-tools() { docker run -w /home -v ${PWD}:/home nathanhowell/parquet-tools $@; }

You have to be in the same directory as the parquet file you want to inspect (since current directory will be mounted to the docker as /home). You can then use the tools directly on any parquet file, i.e.:

parquet-tools dump fruits.parquet

[eliezershindler]

I'm getting Error:TypeError: Cannot read property 'num_values' of null when trying to read 'fruits.parquet' using the read functionality of the module

[eliezershindler]

I can read it fine when using @drauschenbach tool above

[sfescape]

I get the num_values error when writing fields with null values. Not writing those fields when their value is null avoided the issue.

[ZJONSSON]

You might want to check out this PR here #56 which has some fixes to RLE encoding and does verification of the generated files with parquet-mr.

I think you should be able to install this branch simply by:

npm install zjonsson/parquetjs#0c7948d4fa64acf76e481256422c6f4a6ba56815