strategies to reduce write amplification

[facebook]

strategies to reduce write amplification

[mdcallag]

Your suggestions for reducing write-amp are valid. However getting more data into memory isn't trival. If you make the memtable huge then queries of it and inserts into it can be slow. An alternative is to keep old memtables in memory and then merge several of them to create a larger L0 file. RocksDB also has alternate memtable implementations where you sacrifice some features provided by the skip list to get better performance.

Having spent a lot of time thinking about write-amplification this year, I am happy to read what others write about it. With RocksDB we now have LevelDB-style (leveled) and HBase-style compaction (universal). I think you are describing write-amp for leveled compaction above and write-amp for it is usually much worse than for universal. However, I don't understand why you include the "2*" term in the cost formula above. I think that might be doubling the write-amp that is likely to occur. The 2X will account for the reads & writes that can be done during compaction, but I call that compaction amplification, not write amplification.

I use the following estimate for write-amp with leveled compaction when there are N levels -- L0, L1, L2, ... LN-1.

• +1 for the redo log
• +1 for the write to an L0 file
• +1 for compaction to L1 assuming size(L0 files in compaction) == size(L1)
• +10 for each level that follows assuming default growth ratio of 10 between levels was used

So my estimate is 3 + 10*X where X is the number of levels excluding L0 and L1. If you use the --stats_per_interval and --stats_interval options with db_bench when doing a benchmark that writes data it will show the amount of reads and writes per level and the total write-amp and the value printed there tends to match the estimated cost I describe here.

My calculation has included the read IO, so I include "2*" term.
Given the speed of read/write is 126k/17k, just about 7.5, Then I think the io-amp is the bottleneck. Maybe making the memtable huge or keep more memtables in memory is a worthy strategy.

And another non trivial strategy is to split range dynamicly like bigtable or hbase. Then it can keep the number of levels low.

[mdcallag]

RocksDB has two compaction algorithms -- leveled and universal. leveled is what you we discussed earlier on this thread. universal is more like hbase and write amplification is much lower with it. Try db_bench --compaction_type=1. But we really need to provide more docs for it.

[dhruba]

We have added a new compaction algorithm called FIFO compaction:
6de6a06

FIFO compaction allows a database to behave more like a TTL-based cache where older entries are automatically purged. This setting can be configured such that write-amplification is really low, close to 1 or 2.

[somnathr]

Will there be any performance impact for using this FIFO compaction with level db?
Basically, I want to know what are the trade-offs for doing this ?

[igorcanadi]

LevelDB?

FIFO compaction is currently very low overhead and write amplification, but might increase space and read amplification. It is also TTL based, so it deletes oldest records automatically.

[somnathr]

Sorry, I mean Rocksdb :-)
Thanks for the information. But, delete is very expensive in flash. Did you guys have any benchmark data (performace and WA induced) at scale on flash for the Rocksdb ?

[igorcanadi]

These are our performance benchmarks: https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks

[igorcanadi]

Closing this since it's not really an issue. Please post in https://www.facebook.com/groups/rocksdb.dev/ if you wish to discuss this further.