DirectMemory size detection

[filodb]

DirectMemory size detection

[velvia]

Asked this question on StackOverflow:
http://stackoverflow.com/questions/32831361/how-to-detect-amount-of-java-8-direct-memory-left

Note that we need DirectMemory because it is used for the MemTable. Originally we used Heap, but caused huge GC issues and stalls. Moving to off-heap made ingestion much much smoother. If we implement our own MemTable using Filo buffers instead of MapDB, and it is off-heap, we might not have to worry about direct memory.

[velvia]

Took a different approach. Instead of using DirectMemory we just use the heap with MemoryDB which is byte serialized, less GC pressure. Now able to ingest 4 million row DataFrame completely.