Comments (22)
For reference, the backtrace reduces to:
0x515d92 - std::string format_backtrace(bool), src/backtrace.cc:198
0x512f54 - void report_fatal_error(const char*, int, const char*, ...), src/errors.cc:65
0x44b85f - void pool_t::worker_t::on_error(), src/extproc/pool.cc:216
0x5f3d49 - void linux_event_watcher_t::on_event(int), src/arch/io/event_watcher.cc:66
0x608e26 - void epoll_event_queue_t::run(), src/arch/runtime/event_queue/epoll.cc:114
0x60975e - void* linux_thread_pool_t::start_thread(void*), src/arch/runtime/thread_pool.cc:141
which is one of the most sensical backtraces I've seen yet, it seems like our symbol files are good for something after all. This appears to be the same as issue #35, though, closing that one since it has less information.
Thanks for including the version you're using, by the way, it makes this much easier.
As for the error itself, I'm not sure what could cause an error on the worker socket, but I'll see what I can find. It seems that we should be robust enough to handle this with (at the worst) only failing a query.
from rethinkdb.
Thanks, wasn't sure if those were the same since the other one happened in different circumstances.
This one seems to crop up reproducibly (on this machine: a VirtualBox VM on my Mac) when I run certain queries, though the same queries work fine if I run it on an EC2 box with more or less the same data.
Let me know if there's a good way to get more debugging info from this.
from rethinkdb.
If there is a certain set of queries that can reproduce this, it would be helpful to know what they are, and we could try reproducing it here.
from rethinkdb.
Actually, it seems that all (?) queries make it fail? Running
r.db('ka').table('raw_feedback').count().run();
(in the admin UI) runs for about 15 seconds and then the server dies.
from rethinkdb.
Looks like something is borked about that table; operations on another table that I just made work fine.
from rethinkdb.
Hmm is the table still borked if you shutdown and then restart the server? If so would it be possible for you to share the relevant data files with us or do they contain sensitive information?
from rethinkdb.
Not sure what you mean -- the server crashes and I've been starting it back up each time.
The data files aren't particularly sensitive but the table is 1.5 GB so is a little impractical to transport.
from rethinkdb.
We'll see if we can get a large file transfer going.
from rethinkdb.
I'd add rethinkdb datafiles tend to respond well to compression. I've seen about a factor of 10 decrease in size which could make this a bit more palatable.
from rethinkdb.
K got my hands on a data file for this looking in to it.
from rethinkdb.
So I have downloaded and run your data files and been unable to get any sort of crash with them. This was at f533ac9 and was run on an ubuntu machine rather than a virtualbox. I'm going to look in to 1.2.5 and see if I can reproduce it. If not next step is virtual box.
from rethinkdb.
spicyj alright I've had no luck with the specific binary from v1.2.5 which leads me to believe the VM is blame. You mentioned in IRC that you wouldn't mind sending me the VM image. That would be incredibly helpful if you wouldn't mind. I believe you have my email address.
from rethinkdb.
Hmm, so I just got the VM and ran your query. For me it crashed because the rethinkdb process ran out of memory and was killed. This is easy enough to fix but it seems to match pretty well with your symptoms could this be the problem? Running rethinkdb with only 512 Mb of memory isn't fully recommended but it can be done by decreasing the cache size of the tables.
from rethinkdb.
That could definitely be the case. How'd you determine that it ran out of memory?
from rethinkdb.
If this is indeed the case, we should give a good error message in case of out-of-memory conditions (e.g. if malloc fails, crash with an intelligent error message). Of course even printing a good error message in those conditions may not be possible, in which case we should ideally detect that the condition might occur and print a message in advance. Tricky, but certainly doable and will save people a lot of grief.
from rethinkdb.
I was under the impression that modern Linux only allows malloc and friends to fail when the address space is exhausted, and instead deploy the OOM killer when memory is exhausted. So that would be hard to gracefully error on.
We could in theory periodically check how much memory we're using and warn if we're using too much.
from rethinkdb.
Ah, I didn't know about that. But in any case, yes, I was proposing the latter.
EDIT: actually scratch that, I was proposing both, but given the OOM stuff, only the latter actually makes sense. (Although it might be possible to detect that you're being killed by the kernel and log it, though that seems unlikely to me)
from rethinkdb.
Yeah I got hit with the oom killer so we can't do too much about that. We
can keep an eye on memory usage vs total system memory.
On Monday, November 19, 2012, coffeemug wrote:
Ah, I didn't know about that. But in any case, yes, I was proposing the
latter.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/38#issuecomment-10541021.
from rethinkdb.
Well, of course every responsible sysadmin would turn overcommiting off on a server, so malloc would indeed fail.
http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting
(the every in there is ironic. I know that many people don't do it, and there might even be occasional reasons not to)
Also, malloc can still fail with overcommiting on, as some heuristic is apparently at work in the kernel that decides whether and how much to overcommit memory.
from rethinkdb.
There was a patch designed to notify processes of low memory conditions so they can take some action -- http://lwn.net/Articles/267013/. Unfortunately I can't find the relevant block device on Ubuntu 11.10, so either most distros disable it in the kernel, or the patch never went through. I'm not sure what a good way to detect when OOM killer spins up is, but it'd be great if we could find one so we can warn the user.
from rethinkdb.
@spicyj In my case it popped up with a message that the process had been killed due to memory usage in the main window (the one I got when I started the virtual box session). If this is indeed the case and you need to run rethinkdb in low memory conditions the way to do it is by decreasing the sizes of the tables. This can be done in the optional settings of the create table dialogue. I ran with 256 MB of cache and still hit the oom although it did take a lot longer. I think you need to take it down as low as 128 total for it to work.
from rethinkdb.
So unless I'm mistaken we've concluded that this was an oom-killer problem and since we have other issues open that deal directly with that issue (#98, #97) I'm going to close this.
from rethinkdb.
Related Issues (20)
- download.rethinkdb.com is down, 502 Bad Gateway HOT 2
- Evaluate Profile-Guided Optimization (PGO) on RethinkDB
- error: to_string called on an uninitialized ip_address_t, addr_type: 0 compiling rethinkdb on Raspberry HOT 6
- RethinkDB not fully supported on Raspberry PI OS Bullseye (32/64 bit) HOT 10
- Reasonable to change hard-coded cluster size? HOT 5
- help bro my issue = warn: Problem when checking for new versions of RethinkDB: HTTP request to update.rethinkdb.com failed. HOT 1
- cluster connect/reconnect timeout HOT 1
- Installation fails in Kubuntu 23.10 HOT 4
- Generate web_assets.cc in a repeatable file order HOT 1
- Avoid full paths of coffeescript files in generation of web_assets.cc HOT 2
- Rethinkdb 2.4.4 release list HOT 11
- Support protobuf 25
- Return multiple changes feed
- Cache miss rate measurements HOT 4
- Something i forgot to had when having the default doc like so
- Connect rethinkdb directly from browser ? HOT 1
- Why exactly do rethinkDB add null characters to response / request JSON ? HOT 5
- I could not install RethinkDB in Ubuntu 24.04 HOT 3
- How to handle NaN error in javascript Driver? HOT 3
- Is it just me or is there a problem with the download page? HOT 11
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rethinkdb.