mafintosh/flat-file-db

Fast in-process flat file database that caches all data in memory

flat-file-db

Fast in-process flat file database for Node.js that supports JSON and caches all data in memory. All data is persisted to an open file using a append-only algorithm ensuring compact file sizes and strong consistency.

npm install flat-file-db

Usage

Pass a database file to use to the flat-file-db constructor and wait for the database to open. When it is open all data has been loaded into memory.

var flatfile = require('flat-file-db');
var db = flatfile('/tmp/my.db');

db.on('open', function() {
	db.put('hello', {world:1});  // store some data
	console.log(db.get('hello')) // prints {world:1}

	db.put('hey', {world:2}, function() {
		// 'hey' is now fully persisted
	});
});

If you don't want to wait for it to open use flatfile.sync

var db = flatfile.sync('/tmp/my.db');
console.log(db.get('hello')); // prints {world:1}

If you issue multiple writes the last one will always win

for (var i = 0; i < 10; i++) {
	db.put('test', {count:i}, ...);
}
console.log(db.get('test')); // {count:9} which also the persisted value of 'test'

API

• db = flatfile(path, opts) Create a new db instance. Per default fsync is called on all puts. To disable this set opts.fsync = false

• db = flatfile.sync(path, opts) Same as above except you do not need to wait for the open event

• db.put(key, val, [cb]) Insert or update new key

• db.del(key, [cb]) Delete a key

• db.get(key) -> doc Get the value of a key

• db.has(key) -> bool True if db has key

• db.keys() -> list Get all keys as an array

• db.clear([cb]) Clears the database of all values

• db.close() Close the database

Events

• db.on('open') Fired when the db is open and ready for use.

• db.on('close') Fired when the db is fully closed

• db.on('drain') All puts and deletes are flushed to disk

License

MIT

flat-file-db's Issues

Is there a quick way to wipe the DB?

Wondering what would be the easiest way to wipe the DB but keep the file (or I guess delete the file if that's better).

db.put function duplicates information

when i run
db.put('hello', {world:1});
and then at a later time after closing and reopening i run that again

i get two entries of hello?? why is this so?? from the api i am never gonna get the previous value of hello so why does the value still persists.. is there a way not to make such things happen so that the file size gets reduced

fails to read larger database

Hi.

I don't quite know how to reproduce this yet, but I have a database that fails to read. There were some small entries added first, but then a large entry was added, and updated a few times.

trace:

# node backup-glacier.js 
oops : TypeError: Cannot call method 'push' of undefined TypeError: Cannot call method 'push' of undefined
    at free (/root/cronscripts/node_modules/flat-file-db/index.js:92:21)
    at /root/cronscripts/node_modules/flat-file-db/index.js:107:10
    at Array.forEach (native)
    at populateFreelist (/root/cronscripts/node_modules/flat-file-db/index.js:105:10)
    at parseDatabase (/root/cronscripts/node_modules/flat-file-db/index.js:80:2)
    at Database.openSync (/root/cronscripts/node_modules/flat-file-db/index.js:220:2)
    at Function.open.sync (/root/cronscripts/node_modules/flat-file-db/index.js:246:5)
    at /root/cronscripts/backup-glacier.js:37:23

file: http://manticore.2y.net/temp/backup.jdb.gz (1.8M)

Call fsync to be truly "fully persisted"

There should probably be a way to turn fsync on/off (currently always off). Without calling fsync, the data can be in a write cache and not reach the filesystem.

It's a correctness/performance tradeoff that you could let the user make.

error putting a long value

Is there any lenght limit for the value while doing a put on db?

i'm doing:

var flatfile = require('flat-file-db');
var db = flatfile.sync('mydb');
var key = 'key1'
var value = '';
db.put(key,value)

JsonObjectPassedAsParam is a parameter. the snippet of code is working, but when i try to write particularly json objects, it give me the following error:
Cannot read property 'length' of undefined

As attachment, one of large jsonObject impossibile to write.

Fabrizio
json_impossibile_to_write.txt

leveldown compatibility

hey it would be super awesome if this was a leveldown, all you have to do is require the test suite from https://github.com/rvagg/abstract-leveldown e.g. https://github.com/maxogden/level.js/blob/master/test.js