Some questions on cachelib allocator, pool and pool size

[facebook]

Some questions on cachelib allocator, pool and pool size

[sathyaphoenix]

I'd recommend not spinning multiple instances of cachelib and instead use the pools to partition the memory available. The motivation for this is in-line with what @sjoshi6 brought up. Copying 32 bytes might not be a significant cpu impact compared to leaving fragmented memory that is harder to manage across multiple instances. If you'd like to not use std::string (since it uses heap allocation past 20 bytes and incurs a malloc + copy), you can still allocate memory on stack, copy the contents and wrap the stack memory into a Item::Key to call the find apis as long as the calls happen in the same call stack. Lots of performance critical applications do this trick.

[therealgymmy]

How do we know the cache is created? Do we only need to check cache == nullptr?

This will throw if there're bad configs. (You will see std::invalid_argument()). Cache is created if the constructor didn't throw. Typically we recommend you create cache as a std::unique_ptr<...>, so it is easy for you to move it around and destroy.

Is there any easier way to accommodate the overhead?

Can you clarify this question more? What overheads are you referring to?

Is there any better idea to support the feature that same key in different pools?

We currently do not support this. Because we share the index across all memory pools, the keys must be unique. On insertion, theoretically we can allow you to pass us two keys (prefix + actual key) and we just memcpy them into the item memory. However, on lookup, we have to concatenate because we need the key to be (prefix + actual key). And, I suspect it is the lookup that's the most expensive here (which we don't have a good way to solve it).

Have you measured how much perf overhead this is? (If keys are small <15 bytes, this shouldn't incur heap allocation if using std::string). If overhead is too much, you should consider using multiple CacheLib instances instead of cache pools.

[tangliisu]

Thank you for the quick reply!

Can you clarify this question more? What overheads are you referring to?

I think we can't directly add pool in the following way
cache->addPool(name1, 30GB); cache->addPool(name2, 15GB)
my understanding is fixed overhead needed to manage cache in cachelib so the memory allocated to two pools is less than 45GB.
Then the way I allocate memory to pool1 and pool2 is

cache->addPool(name1, cache_->getCacheMemoryStats().cacheSize * 30 / (30 + 15))
cache->addPool(name2, cache_->getCacheMemoryStats().cacheSize * 15 / (30 + 15))

but i am not sure if it is a good practice. How do you always set the pool size when multiple pools are needed

Have you measured how much perf overhead this is? (If keys are small <15 bytes, this shouldn't incur heap allocation if using std::string). If overhead is too much, you should consider using multiple CacheLib instances instead of cache pools.

We didn't do the perf test right now but will do it soon. Does using multiple cachelib instances impact the perf or have other disadvantage compared to using multiple cache pools in a Cachelib instance?

[sjoshi6]

There are some experimental cachelib features which might be very useful for us in the future such as "Automatic pool resizing", "Memory Monitor". I believe we won't be able to leverage them well, if we have multiple cachelib instances in a process.

A typical key-size we encounter is ~32bytes.

[tangliisu]

@sathyaphoenix that's a great suggestion, thank you! I have another question on persistent cache. If we want to enable persistent cache, we have to set something as follows.

config.enableCachePersistence(path);
Cache cache(Cache::SharedMemNew, config);

Does this local path store metadata only or store the whole old cache instance? If only storing metadata, could i expect the size of metadata is very small?

[sathyaphoenix]

Does this local path store metadata only or store the whole old cache instance? If only storing metadata, could i expect the size of metadata is very small?

It is only storing some metadata, which should be less than a KB. You can have this be on any file system and is not performance critical. All the data and any heap metadata is persisted either through shared memory or on-device. The metadata stored within files in the cache directory is some limited information to recover all other pieces of information.

[sathyaphoenix]

@tangliisu I'll close this ticket since the original questions are answered. Please feel free to open a new one if you have any additional questions :)