Fine tune and document Rocks DB parameters

[tkstanczak]

Fine tune and document Rocks DB parameters

[tkstanczak]

After reviewing closely main changes suggested for now would be:

• bigger write cache but smaller write cache number - this change is more by intuition based on reading
• multi threaded compaction to increase the throughput on SSDs - seems to be bringing great results

• in my case the DB has some bad score - 1.1 - 1.3 on all levels which cleared quickly with 500MB/s reads/writes with multi threaded compactions

• will need to manage DB size parameter to be split into memtables (currently at 128MB) + blok cache; at 300GB my indexes took 3GB when all loaded to memory - when allocating 4GB memory it will be just fine to keep everything in - in such case we can set to keep filters and indexes in block cache and just control size of the block cache
• need to look at index/filter partitioning - it seems that we would need to update our RocksDB version / update the RocksDbSharp wrapper
• bottommost compression is not set - this may help, although even without it seems that our DB sizes are smaller than reported by Geth (due to the way we handle storage - worth analyzing)
• we use Snappy on high levels - can check how we could enable LZ4 which was suggested by RocksDb creators to be always a better choice
• need to check if options.OptimizeForPointLookup(128) actually works as the logs are suggesting that this setting is entirely ignored and 8MB * 4 block cache is used instead

[tkstanczak]

Need to look at the RocksDbSharp to check why the code below would not work:
// RocksDbSharp.Native.Instance.rocksdb_create_cache_lru([capacity]) // should be able to create LRU cache here
// tableOptions.SetBlockCache() // should be able to pass the LRU IntPtr here