Summary : RocksDB looks like using more memory than the size of total memory specified in rocksdb::Options. It leads Linux kernel to kill a process that is using RocksDB after using physical memory and swap.
Created bg thread 0x7f748f3ff700
./master.sh: line 1: 29105 Killed
cat /var/log/messages
Nov 28 15:34:57 userver90-157 kernel: Out of memory: Kill process 29105 (ssdb-server) score 335 or sacrifice child
Nov 28 15:34:57 userver90-157 kernel: Killed process 29105, UID 1000, (ssdb-server) total-vm:9259152kB, anon- rss:4827800kB, file-rss:804kB
I was trying to load 150GB of data into a system that has only 16GB of physical memory. Insertion rate was 15000 key/value pairs per second. Key size is 32 bytes, Value size varies from 100 to 2KB.
Following is the list of option values I set.
options.create_if_missing = true;
options.filter_policy = rocksdb::NewBloomFilterPolicy(10);
// 512M
options.block_cache = rocksdb::NewLRUCache(512 * 1048576);
// 32K
options.block_size = 32 * 1024;
// 64M
options.write_buffer_size = 64 * 1024 * 1024;
options.max_open_files = 200;
options.compression = rocksdb::kNoCompression;
Options that are not set :
block_cache_compressed
memtable_factory
table_factory
Is there a single option parameter that can be set to limit the maximum size of memory that RocksDB can use?
Following is from the result of profiling with valgrind using massif.
The process is using 1.6GB. If I do not stop the process, it continues to allocate memory resulting in killing the process by Linux kernel.
n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)
69 227,407,665,278 1,718,009,376 1,717,227,820 781,556 0
99.95% (1,717,227,820B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->97.68% (1,678,140,930B) 0x475E17: rocksdb::ArenaImpl::AllocateNewBlock(unsigned long) (arena_impl.cc:72)
| ->97.68% (1,678,140,930B) 0x475E99: rocksdb::ArenaImpl::AllocateFallback(unsigned long) (arena_impl.cc:43)
| ->77.34% (1,328,755,428B) 0x450D72: rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&) (arena_impl.h:81)
| | ->75.00% (1,288,490,112B) 0x471990: rocksdb::(anonymous namespace)::MemTableInserter::Put(rocksdb::Slice const&, rocksdb::Slice const&) (write_batch.cc:201)
| | | ->75.00% (1,288,490,112B) 0x471D89: rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler_) const (write_batch.cc:84)
| | | ->75.00% (1,288,490,112B) 0x47225C: rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteBatch const_, rocksdb::MemTable_, rocksdb::Options const_, rocksdb::DB_, bool) (write_batch.cc:232)
| | | ->75.00% (1,288,490,112B) 0x43A489: rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch_) (db_impl.cc:2856)
| | | | ->75.00% (1,288,490,112B) 0x41C03B: BinlogQueue::commit() (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | | ->75.00% (1,288,490,112B) 0x40D8AD: SSDB::set(Bytes const&, Bytes const&, char) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | | ->75.00% (1,288,490,112B) 0x422670: proc_set(Server_, Link_, std::vector<Bytes, std::allocator > const&, std::vector<std::string, std::allocatorstd::string >) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | | ->75.00% (1,288,490,112B) 0x41F15F: Server::ProcWorker::proc(ProcJob) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | | ->75.00% (1,288,490,112B) 0x42A650: WorkerPool<Server::ProcWorker, ProcJob>::run_worker(void) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | | ->75.00% (1,288,490,112B) 0x4E3084F: start_thread (in /lib64/libpthread-2.12.so)
| | | | ->75.00% (1,288,490,112B) 0x5CDC94B: clone (in /lib64/libc-2.12.so)
| | | |
| | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->02.34% (40,265,316B) 0x47188C: rocksdb::(anonymous namespace)::MemTableInserter::Delete(rocksdb::Slice const&) (write_batch.cc:221)
| | ->02.34% (40,265,316B) 0x471D55: rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler) const (write_batch.cc:92)
| | ->02.34% (40,265,316B) 0x47225C: rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteBatch const, rocksdb::MemTable_, rocksdb::Options const_, rocksdb::DB_, bool) (write_batch.cc:232)
| | ->02.34% (40,265,316B) 0x43A489: rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch_) (db_impl.cc:2856)
| | ->01.95% (33,554,430B) 0x41D027: BinlogQueue::del_range(unsigned long, unsigned long) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->01.95% (33,554,430B) 0x41D148: BinlogQueue::log_clean_thread_func(void_) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->01.95% (33,554,430B) 0x4E3084F: start_thread (in /lib64/libpthread-2.12.so)
| | | ->01.95% (33,554,430B) 0x5CDC94B: clone (in /lib64/libc-2.12.so)
| | |
| | ->00.39% (6,710,886B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->20.34% (349,385,502B) 0x475F23: rocksdb::ArenaImpl::AllocateAligned(unsigned long) (arena_impl.cc:65)
| ->10.55% (181,193,922B) 0x48B638: rocksdb::SkipList<char const*, rocksdb::MemTableRep::KeyComparator&>::Insert(char const* const&) (skiplist.h:196)
| | ->10.55% (181,193,922B) 0x48B337: rocksdb::(anonymous namespace)::SkipListRep::Insert(char const_) (skiplistrep.cc:22)
| | ->10.55% (181,193,922B) 0x450D0C: rocksdb::MemTable::Add(unsigned long, rocksdb::ValueType, rocksdb::Slice const&, rocksdb::Slice const&) (memtable.cc:155)
| | ->08.20% (140,928,606B) 0x471990: rocksdb::(anonymous namespace)::MemTableInserter::Put(rocksdb::Slice const&, rocksdb::Slice const&) (write_batch.cc:201)
| | | ->08.20% (140,928,606B) 0x471D89: rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler_) const (write_batch.cc:84)
| | | ->08.20% (140,928,606B) 0x47225C: rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteBatch const_, rocksdb::MemTable_, rocksdb::Options const_, rocksdb::DB_, bool) (write_batch.cc:232)
| | | ->08.20% (140,928,606B) 0x43A489: rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch_) (db_impl.cc:2856)
| | | ->08.20% (140,928,606B) 0x41C03B: BinlogQueue::commit() (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.20% (140,928,606B) 0x40D8AD: SSDB::set(Bytes const&, Bytes const&, char) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.20% (140,928,606B) 0x422670: proc_set(Server_, Link_, std::vector<Bytes, std::allocator > const&, std::vector<std::string, std::allocatorstd::string >) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.20% (140,928,606B) 0x41F15F: Server::ProcWorker::proc(ProcJob) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.20% (140,928,606B) 0x42A650: WorkerPool<Server::ProcWorker, ProcJob>::run_worker(void) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.20% (140,928,606B) 0x4E3084F: start_thread (in /lib64/libpthread-2.12.so)
| | | ->08.20% (140,928,606B) 0x5CDC94B: clone (in /lib64/libc-2.12.so)
| | |
| | ->02.34% (40,265,316B) 0x47188C: rocksdb::(anonymous namespace)::MemTableInserter::Delete(rocksdb::Slice const&) (write_batch.cc:221)
| | ->02.34% (40,265,316B) 0x471D55: rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler) const (write_batch.cc:92)
| | ->02.34% (40,265,316B) 0x47225C: rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteBatch const, rocksdb::MemTable_, rocksdb::Options const_, rocksdb::DB_, bool) (write_batch.cc:232)
| | ->02.34% (40,265,316B) 0x43A489: rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch_) (db_impl.cc:2856)
| | ->02.34% (40,265,316B) 0x41D027: BinlogQueue::del_range(unsigned long, unsigned long) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | ->02.34% (40,265,316B) 0x41D148: BinlogQueue::log_clean_thread_func(void_) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | ->02.34% (40,265,316B) 0x4E3084F: start_thread (in /lib64/libpthread-2.12.so)
| | ->02.34% (40,265,316B) 0x5CDC94B: clone (in /lib64/libc-2.12.so)
| |
| ->09.79% (168,191,580B) 0x48A7B6: rocksdb::SkipListFactory::CreateMemTableRep(rocksdb::MemTableRep::KeyComparator&, rocksdb::Arena_) (skiplist.h:196)
| ->09.79% (168,191,580B) 0x451E7D: rocksdb::MemTable::MemTable(rocksdb::InternalKeyComparator const&, std::shared_ptrrocksdb::MemTableRepFactory, int, rocksdb::Options const&) (memtable.cc:52)
| ->09.37% (161,061,264B) 0x42F694: rocksdb::DBImpl::MakeRoomForWrite(bool) (db_impl.cc:3136)
| | ->09.37% (161,061,264B) 0x43A052: rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch_) (db_impl.cc:2808)
| | ->08.98% (154,350,378B) 0x41C03B: BinlogQueue::commit() (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.98% (154,350,378B) 0x40D8AD: SSDB::set(Bytes const&, Bytes const&, char) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.98% (154,350,378B) 0x422670: proc_set(Server_, Link_, std::vector<Bytes, std::allocator > const&, std::vector<std::string, std::allocatorstd::string >) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.98% (154,350,378B) 0x41F15F: Server::ProcWorker::proc(ProcJob) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.98% (154,350,378B) 0x42A650: WorkerPool<Server::ProcWorker, ProcJob>::run_worker(void) (in /home/widerplanet/kangmo/refs/ssdb-rocks/ssdb-server)
| | | ->08.98% (154,350,378B) 0x4E3084F: start_thread (in /lib64/libpthread-2.12.so)
| | | ->08.98% (154,350,378B) 0x5CDC94B: clone (in /lib64/libc-2.12.so)
| | |
| | ->00.39% (6,710,886B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.42% (7,130,316B) in 1+ places, all below ms_print's threshold (01.00%)
|
->02.23% (38,237,022B) 0x4A3354: rocksdb::ReadBlockContents(rocksdb::RandomAccessFile, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockContents, rocksdb::Env_, bool) (format.cc:88)
| ->02.04% (35,115,915B) 0x49792A: rocksdb::BlockBasedTable::ReadFilter(rocksdb::Slice const&, rocksdb::BlockBasedTable::Rep_, unsigned long_) (block_based_table_reader.cc:405)
| | ->02.04% (35,115,915B) 0x499D01: rocksdb::BlockBasedTable::GetFilter(bool) const (block_based_table_reader.cc:822)
| | ->02.04% (35,115,915B) 0x49C5A2: rocksdb::BlockBasedTable::Open(rocksdb::Options const&, rocksdb::EnvOptions const&, std::unique_ptr<rocksdb::RandomAccessFile, std::default_deleterocksdb::RandomAccessFile >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_deleterocksdb::TableReader >) (block_based_table_reader.cc:321)
| | ->02.04% (35,115,915B) 0x49621F: rocksdb::BlockBasedTableFactory::GetTableReader(rocksdb::Options const&, rocksdb::EnvOptions const&, std::unique_ptr<rocksdb::RandomAccessFile, std::default_deleterocksdb::RandomAccessFile >&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_deleterocksdb::TableReader >) const (block_based_table_factory.cc:26)
| | ->02.04% (35,115,915B) 0x456DAC: rocksdb::TableCache::FindTable(rocksdb::EnvOptions const&, unsigned long, unsigned long, rocksdb::Cache::Handle__, bool_, bool) (table_cache.cc:76)
| | ->02.04% (35,063,961B) 0x457021: rocksdb::TableCache::NewIterator(rocksdb::ReadOptions const&, rocksdb::EnvOptions const&, unsigned long, unsigned long, rocksdb::TableReader*, bool) (table_cache.cc:104)
| | | ->01.24% (21,363,233B) 0x42E626: rocksdb::DBImpl::FinishCompactionOutputFile(rocksdb::DBImpl::CompactionState, rocksdb::Iterator_) (db_impl.cc:2004)
| | | | ->01.13% (19,425,164B) 0x436942: rocksdb::DBImpl::DoCompactionWork(rocksdb::DBImpl::CompactionState_, rocksdb::DBImpl::DeletionState&) (db_impl.cc:2378)
| | | | | ->01.13% (19,425,164B) 0x437FCD: rocksdb::DBImpl::BackgroundCompaction(bool_, rocksdb::DBImpl::DeletionState&) (db_impl.cc:1821)
| | | | | ->01.13% (19,425,164B) 0x43DC85: rocksdb::DBImpl::BackgroundCallCompaction() (db_impl.cc:1713)
| | | | | ->01.13% (19,425,164B) 0x47E23F: rocksdb::(anonymous namespace)::PosixEnv::ThreadPool::BGThreadWrapper(void_) (env_posix.cc:1362)
| | | | | ->01.13% (19,425,164B) 0x4E3084F: start_thread (in /lib64/libpthread-2.12.so)
| | | | | ->01.13% (19,425,164B) 0x5CDC94B: clone (in /lib64/libc-2.12.so)
| | | | |
| | | | ->00.11% (1,938,069B) in 1+ places, all below ms_print's threshold (01.00%)
| | | |
| | | ->00.80% (13,700,728B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->00.00% (51,954B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.18% (3,121,107B) in 1+ places, all below ms_print's threshold (01.00%)
|
->00.05% (849,868B) in 1+ places, all below ms_print's threshold (01.00%)