Invedsstigate Tungsten integration/support

[filodb]

Invedsstigate Tungsten integration/support

[velvia]

I've just wrapped up a performance comparison, and by the way you can run the SparkReadBenchmark using jmh:run and uncomment the tungsten disable to see what the effect is.

• Tungsten is automatically used by default for all DataFrame queries in Spark 1.5.x and up
• It introduces a ~1 second startup time for the first time a query is run, due to code gen
• Perhaps 20% improvement in in-memory scan speeds over having Tungsten off, in the benchmarks. However in real life Spark Shell use, the improvements are not necessarily visible. Perhaps more visible in joins.
• Major areas of use includes UnsafeRow, an off-heap, byte-serialized representation of intermediate rows; and more efficient cache friendly structures for joins and grouping aggregations. UnsafeRow is not really useful for us, we already have our data in a much more efficient columnar byte-serialized representation.

I think until the vectorized computation stuff comes in, there won't be too much of a gain yet.