Crashes almost immediately

[rethinkdb]

Crashes almost immediately

[coffeemug]

Thanks Slava. We're a little inundated with questions, so it'll take some time sort everything out, but we'll investigate this as soon as we can.

[slava-vishnyakov]

Sure, let me know how I can help investigate it. And thanks for looking into it. :)

[srh]

Hi Slava, our current status when investigating this bug is that we've created a Linux Mint 13 VM inside of VirtualBox (with the same uname -a string except for machine name) and failed to reproduce the problem. The server starts up normally.

This is on a fresh installation of Linux Mint 13, running in a VirtualBox VM on a Ubuntu 12.04 host (running on an i7-2760QM on some Intel 330 or 520 SSD), the filesystem being ext4, no weird installation options.

Does this behavior happen consistently for you?

[slava-vishnyakov]

Thanks for looking.

Yes, it's consistent. I tried deleting the folder. I tried running under sudo, etc..

This is also VirtualBox with Minn 13 upgraded to latest versions of packages.

Is there a way to enable debug or something like that? Or something I can provide that will give a hint of what's going on? Like strace, core dump or something?

[srh]

Thanks for the quick response. I think running (strace rethinkdb 2>&1) > trace.txt might be very helpful.

Edit: Sorry, ignore this.

[srh]

Sorry, I meant, try running strace -o trace -f rethinkdb from an empty directory.

Here's the metadata file opening output we get here.

2117  stat("rethinkdb_data/metadata", 0x1d83e80) = -1 ENOENT (No such file or directory)
2117  open("rethinkdb_data/metadata", O_RDWR|O_CREAT|O_DIRECT|O_NOATIME, 0644) = 17
...
2117  stat("rethinkdb_data/metadata", {st_mode=S_IFREG|0644, st_size=524288, ...}) = 0
2117  open("rethinkdb_data/metadata", O_RDWR|O_CREAT|O_DIRECT|O_NOATIME, 0644) = 17

[slava-vishnyakov]

$ cat trace | grep metad
2903  stat("rethinkdb_data/metadata",  <unfinished ...>
2903  open("rethinkdb_data/metadata", O_RDWR|O_CREAT|O_DIRECT|O_NOATIME, 0644) = -1 EINVAL (Invalid argument)

Does that help?

[coffeemug]

@slava-vishnyakov - what are your mount args for ext4? Are you doing anything special?

[slava-vishnyakov]

I don't think that I'm doing anything special.

This is from fstab:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID=ec69baac-863f-42b8-8845-27822ba2b187 /               ext4    errors=remount-ro 0       1

[slava-vishnyakov]

Oh wait, can the encrypted home folder be the reason?

[slava-vishnyakov]

Yep, it was due to running in encrypted home folder. Runned outside of it - everything works.
You can probably add that to a list of reasons,

Thanks for help, guys!

[jdoliner]

Awesome we should add this to known issues.

[daniel-toppatch]

I'm seeing this same issue from within a Vagrant-mounted volume:

vagrant@vagrantbase:/opt/repo$ rethinkdb --config-file /opt/repo/conf/rethinkdb.conf --web-static-directory /opt/repo/rethinkdb/current/web
info: Running rethinkdb 1.3.2-0ubuntu1~precise (GCC 4.6.3)...
info: Loading data from directory /opt/repo/var/rethinkdb/instances
Version: rethinkdb 1.3.2-0ubuntu1~precise (GCC 4.6.3)
error: Inaccessible database file: "/opt/repo/var/rethinkdb/instances/metadata": Invalid argument
       Some possible reasons:
       - the database file couldn't be created or opened for reading and
         writing
       - the database file is located on a filesystem that doesn't support
         O_DIRECT open flag (e.g. some encrypted or journaled file systems)
       - user which was used to start the database is not an owner of the file
Trace/breakpoint trap

[coffeemug]

I believe this is fixed in 1.4 (which is going to be out in a few days). In 1.4 (and next), if rethink can't open a file under O_DIRECT, it falls back to the page cache and gives the user a warning about potential performance ramifications.