Implement statistics

[ironsource]

Implement statistics

[ZJONSSON]

Ideally his might require pushing the shredding operation further down the stack, ideally when we are writing pages for each column chunk. Instead of pushing the fully shredded record on rowBuffer we could push a flattened record (i.e. de-nested) in the same order as the shredder would put them.

The encodeColumnChunk would receive the raw values and perform shredding of all records within each each page ( max/min can be easily accessed at this point and added to metadata for each row and each columnChunk). This would also make splitting into pages way easier as we wouldn't have to reverse-engineer the rLevel and dLevel.

Thoughts?

[ZJONSSON]

Here is a quick WIP of pages - passes integration tests ZJONSSON@e643f72

[dobesv]

Did a version of this library with support for statistics ever get out there? Maybe a fork somewhere? I'd like to have these available.

[jeffbski-rga]

@dobesv ZJONSSON's fork has statistics implemented. parquetjs-lite https://github.com/ZJONSSON/parquetjs

I realize this post is quite old, but might be useful for others that stumble on it.