theycallmeswift/node-mongodb-s3-backup

A node.js package that makes syncing a MongoDB database to S3 simple.

node-mongodb-s3-backup's Issues

How to backup all database?

hello, I want to backup all the mongo data to the s3, what should I write my json file ?

TypeError: mime.lookup is not a function error

I get this error after running mongodb_s3_backup -n config.sample.json.
Do you know what can cause this?

/usr/lib/node_modules/mongodb_s3_backup/index.js:262
        throw(err);
        ^

TypeError: mime.lookup is not a function
    at /usr/lib/node_modules/mongodb_s3_backup/node_modules/knox/lib/client.js:386:28
    at FSReqWrap.oncomplete (fs.js:82:15)

time of 11:59 gives bad cron

I used a time of 11:59 and got:

1 Oct 23:59:17 - [info] MongoDB S3 Backup Successfully scheduled (11 59 * * *)

The cron format is MINUTE HOUR, not HOUR MINUTE.

If I use instead "cron": "59 11 * * *" I get what I expect (plus it wakes up and actually does my backup)

Doesn't work via crontab

IMO, it is better to just use Amazon's CLI and Mongo dump via SH. I had trouble with this and found the following to work well but there is more setup of course.

https://gist.github.com/eladnava/96bd9771cd2e01fb4427230563991c8d

knox should be upgraded to latest

Current knox 0.4 gives the following error while uploading to aws S3:

[error] Error: Hostname/IP doesn't match certificate's altnames

After manually changing to knox 0.8.3 things work fine.

Unexpected errors. Can't tell if succeeded or failed.

I'm going to try restoring from the backup. But it seems like it should not have logged errors. If they were genuine errors, where would I find a log?

root@mydb-mongodb:/mongodb-s3-backup# mongodb_s3_backup -n config.json
21 Feb 23:20:34 - [info] Loading config file (/root/mongodb-s3-backup/config.json)
21 Feb 23:20:34 - [info] Starting mongodump of mydb
21 Feb 23:20:34 - [error] 2016-02-21T23:20:34.250-0500 writing mydb.tasks to /tmp/mongodb_s3_backup/mydb/tasks.bson
21 Feb 23:20:34 - [error] 2016-02-21T23:20:34.254-0500 writing mydb.projects to /tmp/mongodb_s3_backup/mydb/projects.bson
21 Feb 23:20:34 - [error] 2016-02-21T23:20:34.255-0500 writing mydb.notes to /tmp/mongodb_s3_backup/mydb/notes.bson
21 Feb 23:20:34 - [error] 2016-02-21T23:20:34.256-0500 writing mydb.entries to /tmp/mongodb_s3_backup/mydb/entries.bson
2016-02-21T23:20:34.259-0500 done dumping mydb.projects (58 documents)
2016-02-21T23:20:34.259-0500 writing mydb.users to /tmp/mongodb_s3_backup/mydb/users.bson
2016-02-21T23:20:34.260-0500 done dumping mydb.entries (23 documents)
2016-02-21T23:20:34.260-0500 writing mydb.migrations to /tmp/mongodb_s3_backup/mydb/migrations.bson
21 Feb 23:20:34 - [error] 2016-02-21T23:20:34.262-0500 done dumping mydb.tasks (189 documents)
21 Feb 23:20:34 - [error] 2016-02-21T23:20:34.264-0500 done dumping mydb.users (3 documents)
2016-02-21T23:20:34.264-0500 writing mydb.meteor_oauth_pendingCredentials to /tmp/mongodb_s3_backup/mydb/meteor_oauth_pendingCredentials.bson
2016-02-21T23:20:34.265-0500 done dumping mydb.migrations (1 document)
2016-02-21T23:20:34.265-0500 writing mydb.meteor_accounts_loginServiceConfiguration to /tmp/mongodb_s3_backup/mydb/meteor_accounts_loginServiceConfiguration.bson
2016-02-21T23:20:34.266-0500 done dumping mydb.meteor_oauth_pendingCredentials (0 documents)
21 Feb 23:20:34 - [error] 2016-02-21T23:20:34.268-0500 done dumping mydb.notes (24 documents)
2016-02-21T23:20:34.268-0500 done dumping mydb.meteor_accounts_loginServiceConfiguration (0 documents)
21 Feb 23:20:34 - [info] mongodump executed successfully
21 Feb 23:20:34 - [info] Starting compression of mydb into mydb_2016_2_21_1456114834218.tar.gz
21 Feb 23:20:34 - [info] successfully compress directory
21 Feb 23:20:34 - [info] Attemping to upload mydb_2016_2_21_1456114834218.tar.gz to the mydb-mongodb-backups s3 bucket
21 Feb 23:20:34 - [info] Successfully uploaded to s3
21 Feb 23:20:34 - [info] Successfully backed up mydb
21 Feb 23:20:34 - [info] Removing /tmp/mongodb_s3_backup/mydb
21 Feb 23:20:34 - [info] Removing /tmp/mongodb_s3_backup/mydb_2016_2_21_1456114834218.tar.gz
root@mydb-mongodb:/mongodb-s3-backup# ls

Add docs on how to use via public functions

Thought it might be a common use-case to use this module via its "API".

Like so:

var myConfig = require('./config.json');
myConfig.mongodb.password = process.env.MONGODB_PASSWORD;
myConfig.s3.secret = process.env.S3_SECRET;
var mongoBackup = require('mongodb_s3_backup');

// inside cron\task
mongoBackup.sync(myConfig.mongodb, myConfig.s3);

@theycallmeswift This seems to work, but not sure if I'm missing anything.

How can I run this in background ?

Right now I need to keep the terminal open to keep this app running

Support backing up multiple DBs?

How about support for multiple databases?

How to create a weekly and monthly backup plan

I want to backup my database every weekly and monthly, How can i do that

Error

Hi,

I'm trying to make a backup, but I'm having the following issue:

$ mongodb_s3_backup -n configDB.json
29 May 10:25:05 - [info] Loading config file (/configDB.json)
29 May 10:25:05 - [info] Starting mongodump of heroku_app31122212

events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn ENOENT
at errnoException (child_process.js:980:11)
at Process.ChildProcess._handle.onexit (child_process.js:771:34)

Do you have any clue what could be happening?

Thanks

How about an option to clean up old backups

Would that be a useful addition?

Should be very simple using
http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectExpiration.html

Node version is pretty dated, on pre v1 currently

Thanks for building this for sure but you should update the version because its on like v7 now.

Running it on Heroku gives error spawn ENOENT

I have tried running it on Heroku Scheduler.
I have placed the command
mongodb_s3_backup -n node_modules/mongodb_s3_backup/config.sample.json
but when it runs on Heroku it gives the following error for some reason.

[info] Loading config file (/app/node_modules/mongodb_s3_backup/config.sample.json)
[info] Starting mongodump of heroku_app31146145
events.js:72
throw er; // Unhandled 'error' event
^
Error: spawn ENOENT
at errnoException (child_process.js:1001:11)
at Process.ChildProcess._handle.onexit (child_process.js:792:34)

When I run it locally from my terminal it runs fine.
The error also appears when I run to Heroku terminal using the command
heroku run bash
to connect to the remote terminal.

Max upload size

Hi, Could you list the max-upload size somewhere please?

Thanks

Cannot find module '../build/Release/bson'

{ Error: Cannot find module '../build/Release/bson'
at Function.Module._resolveFilename (module.js:470:15)
at Function.Module._load (module.js:418:25)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
at Object. (/Users/vanraar/Documents/Projects/js-playground/mongodb/node_modules/mongo-dump-stream/node_modules/bson/ext/index.js:15:10)
at Module._compile (module.js:571:32)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3) code: 'MODULE_NOT_FOUND' }
js-bson: Failed to load c++ bson extension, using pure JS version

How can i restore the backup

How can I restore the database backup, I didn't see any documentation