Comments (7)
E0108 16:48:53.001791 315636 PoolRebalancer.cpp:50] Rebalancing interrupted due to exception: PoolId 80 is not a compact cache, eventually ending with a segfault at: facebook::cachelib::MemoryPoolManager::getPoolById(signed char) const+0x20
Yeah. It sounds like a code path expects compact cache to be present, but it is infact not a compact cache. I suppose in the setup, you don't create any compact caches and have just a single default pool ?
One suggestion would be to stop pool rebalancer explicitly on startup through this api and see if the problem resurfaces as a different one.
PoolId XX is not a compact cache
this is thrown by the api here and from my quick digging, it should not be called by any pool rebalancing code.
from cachelib.
Thanks @sathyaphoenix. I indeed don't create a compact cache and I indeed only use a single pool (the code above is practicly all the code that uses CacheLib). Stopping the pool rebalancer as you suggested allowed my runs to complete without an issue, don't know if it solved or just hid the real issue but it's a step forward. What's the impact of having no pool rebalancer? Is it an acceptable setup for a single cache and a single pool? I wanna make sure I'm not degrading performance with that.
from cachelib.
@erangi: running without pool rebalancing may cause memory inbalance (e.g. some allocation sizes have no memory to allocate while others have excessive free memory).
Can you share your cache setup and repro instruction? (Both with the rebalancing crash and with the change that made it go away).
Looking at the error message, it smells like corruption if you just have a single pool. The message suggests there exists a PoolId 80 (81st pool created).
Rebalancing interrupted due to exception: PoolId 80 is not a compact cache
from cachelib.
@therealgymmy the cache setup is in my first post, it's pretty straightforward. The only unconventional thing I do is wrap this client code in JNI so I can benchmark CacheLib using YCSB (which is Java). The extended YCSB and JNI wrapper are WIP so I'd rather not publicly share them but I can share them privately if that helps. I agree this looks like memory corruption but it's consistent over runs and machines (except the pool ID, which is often 80 but varies).
FWIW, our benchmarks use a fixed key and value sizes (less realistic but simpler...), so I don't think this synthetic use case will be very challenging for a memory manager.
from cachelib.
@erangi: if it's fixed key and value size, then you can disable rebalancer. It won't do much anyway.
from cachelib.
Sounds good, thanks!
from cachelib.
The extended YCSB and JNI wrapper are WIP
@erangi I 'm sorry to bother you. I also want to benchmark CacheLib using YCSB. But I 'm not famliar with YCSB.
Would you be willing to share the code or insights from your work on this integration?
Thank you
from cachelib.
Related Issues (20)
- Fail to build dependency fbthrift (with errors reported in fmt) HOT 5
- make clean option for contrib/build.sh HOT 1
- build error about fizz on ubuntu22.04 HOT 2
- CDN trace expected behavior HOT 2
- Enable FDP for CacheBench HOT 26
- qDepth Support for NVM Cache HOT 6
- Questions about trace files when running cachebench HOT 2
- Running simple-cache-example gives an error, flag 'v' was defined more than once HOT 6
- OSS build broken as of May, 2024 -> PRs are all blocked HOT 7
- No build support for Fedora37 OS HOT 4
- failed to build CacheLib following document HOT 2
- Build fails on debian-10 HOT 2
- Segmentation fault while fetching refcount HOT 2
- Minimum Limit For Cache Allocation? HOT 1
- build failed when building dependency 'fbthrift' HOT 3
- Build issue with CacheLib with missing source files HOT 3
- [Seeking Volunteers] Add new builds to CacheLib HOT 2
- Build is failing with error: ‘fmt::v10::detail::type_is_unformattable_for<const facebook::cachelib::navy::Status, char> _’ has incomplete type 1600 | type_is_unformattable_for<T, typename Context::char_type> _; | ^ HOT 4
- NavySetup should not involve the MockDevice
- How to configure folly for logging CacheLib
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cachelib.