Comments (3)
I was able to reproduce the error on my laptop with a lot of message that creates a lot of new filters so of course it reached the "too many open files" nearly instantly.
[bloomd]
tcp_port = 8673
udp_port = 8674
data_dir = /data/bloomd
log_level = INFO
cold_interval = 1800
flush_interval = 300
initial_capacity = 10001
default_probability = 0.0001
workers = 4
use_mmap = 1
sudo lsof -p `pidof bloomd` | grep "/data/" | wc -l
472
$ulimit -Hn
unlimited
$ulimit -Sn
2560
Apr 8 15:57:07 MBP.local bloomd[77074] <Critical>: Failed to create new file: /data/bloomd/bloomd.f1.2012-10-2/data.000.mmap for filter f1.2012-10-2. Err: Too many open files
Apr 8 15:57:07 MBP.local bloomd[77074] <Error>: Failed to create SBF: f1.2012-10-2. Err: -24
Apr 8 15:57:07 MBP.local bloomd[77074] <Error>: Failed to fault in the filter 'f1.2012-10-2'. Err: -24
Do I have other choice than incrementing ulimit?
from bloomd.
The underlying issue here is that bloomd maintains open file handles for all the filters that are currently in memory. What is happening is that you are eventually just hitting the limit of file handles, so when you try to create a new filter, bloomd returns an internal error.
When bloomd is restarted, it starts without opening the on disk files, and only loads them when read, so it allows more filters to be made for a while.
The simplest solution is to just raise the ulimit values to something huge, that you are unlikely to ever hit. Otherwise, it would require a very large change to bloomd to actively manage the open file handles and work around the limits.
I hope that helps!
from bloomd.
Ok! I had the same conclusion as you do, thank you for your time!
from bloomd.
Related Issues (20)
- Syslog facilities HOT 2
- rename filter command HOT 7
- Connection drops when flush thread starts. HOT 6
- Higher false positive rate than expected HOT 2
- Abandoned? HOT 4
- can not build on ARM CPU HOT 1
- Can't compile on alpine linux HOT 2
- Failed to accept() connection! Too many open files. HOT 1
- race condition in filtmgr_check_keys HOT 1
- Question: cuckoo filter HOT 1
- Repo can update to the latest version HOT 1
- in_memory = 0 HOT 1
- Is there any client lib implemented in cīŧ HOT 1
- Does bloomd have problems with more than 1 billion entries? HOT 1
- bitmap_from_filename & bf_from_bitmap HOT 1
- Time outs [+question] HOT 1
- cast-function-type error in bloomd.c HOT 2
- clear command report filter is not proxied HOT 2
- Why does the probability value (FP) decrease from initial setting.
- Build errors (cc1: all warnings being treated as errors) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. đđđ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google â¤ī¸ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bloomd.