derbyjs/racer

Realtime model synchronization engine for Node.js

Racer

Racer is a realtime model synchronization engine for Node.js. By leveraging ShareDB, multiple users can interact with the same data in realtime via Operational Transformation, a sophisticated conflict resolution algorithm that works in realtime and with offline clients. ShareDB also supports PubSub across multiple servers for horizontal scaling. Clients can express data subscriptions and fetches in terms of queries and specific documents, so different clients can be subscribed to different overlapping sets of data. On top of this sophisticated backend, Racer provides a simple model and event interface for writing application logic.

Demos

There are currently two demos, which are included in the racer-examples repo.

• Pad – A very simple collaborative text editor.
• Todos – Classic todo list demonstrating the use of Racer's model methods.

Features

• Realtime updates – Model methods automatically propagate changes among browser clients and Node servers in realtime. The racer-browserchannel adapter is recommended for connecting browsers in realtime.

• Realtime query subscriptions – Clients may subscribe to a limited set of information relevant to the current session. Both document and realtime query subscriptions are supported. Currently, arbitrary Mongo queries are supported.

• Conflict resolution – Leveraging ShareDB's JSON Operational Transformation algorithm, Racer will emit events that bring conflicting client states into eventual consistency. In addition to their synchronous API, model methods have callbacks for handling the resolved state after a server response.

• Immediate interaction – Model methods appear to take effect immediately. Meanwhile, Racer sends updates to the server and checks for conflicts. If the updates are successful, they are stored and broadcast to other clients.

• Offline – Since model methods are applied immediately, clients continue to work offline. Any changes to the local client or the global state automatically sync upon reconnecting.

• Unified server and client interface – The same model interface can be used on the server for initial page rendering and on the client for user interaction. Racer supports bundling models created on the server and reinitializing them in the same state in the browser.

• Persistent storage – Racer uses ShareDB to keep a journal of all data operations, publish operations to multiple frontend servers, and automatically persist documents. It currently supports MongoDB, and it can be easily adapted to support other document stores.

• Access control – (Under development) Racer will have hooks for access control to protect documents from malicious reads and writes.

• Solr queries – (Under development) A Solr adapter will support updating Solr indices as data change and queries for realtime updated search results.

Installation

Racer requires Node v16. You will also need to have a MongoDB and a Redis server running on your machine. The examples will connect via the default configurations.

$ npm install racer

Tests

Run the tests with

$ npm test

Usage

Racer can be used independently as shown in the examples, but Racer and Derby are designed to work especially well together. Racer can also be used along with other MVC frameworks, such as Angular.

For now, Racer is mostly documented along with Derby. See the Derby model docs.

MIT License

Copyright (c) 2024 by Brian Noguchi and Nate Smith

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

racer's Issues

Getting errors trying to npm install

problem

npm ERR! cb() never called!
npm ERR!  
npm not ok

what I did to fix issue

modifed package.json

  "dependencies": {
    "socket.io": "~0.9.x",
    "socket.io-client": "~0.9.x",
    "es5-shim": "~1.2.x",
    "browserify": "~1.10.x",
    "uglify-js": ">=1.2.5",
    "share": "0.4.1",
    "node-uuid": "1.3.3"
  },
  "devDependencies": {
    "mocha": "~0.9.x",
    "expect.js": "~0.1.x",
    "coffee-script": "~1.2.x",
    "console.color": "~0.0.4",
    "racer-journal-redis": "*",
    "racer-pubsub-redis": "*",
    "racer-db-mongo": "*"
  },
  "engines": {
    "node": "0.4 || 0.5 || 0.6"
  },

id - confusion - byId() and where('_id')/where('id') .within([<id#1>, ...]) doens't return scoped model

With new query system, most likely because ids are named ids as well as _ids, whenever you do:

store.query.expose('test', 'fromArrayOfIds', function( ids ) {
return this.byId( ids );
// return this.where('_id').within( ids ); -- Will work exactly as above
// return this.where('id').within( ids ); -- Won't work
});

...

model.subscribe( model.query('test').fromArrayOfIds( my_array_of_ids ), function( err, scoped_tests ) {

console.log( model.get('test') ); // Returns array of test objects - ie. will return expected result
console.log( scoped_tests.get() ); // Returns undefined - ie. won't return expected result

});

Probably related to the confusion mentioned in Issue #20

[Bug] Subscribing multiple queries using array returns cross-results

When subscribing mutliple queries, their results get crossed. Each scopedModel... returned in the Callback function carries the results of its query plus the results of the other queries.

Expected:
Scoped model "seniors" should only have 1 ocurrency of each person over 50 years old.

Actual Result:
In this case, scoped model "seniors" will contain people over 50, plus people over 50 that live in Lisbon, plus another copy of everyone that satisfies the first two queries.

---

    peopleOlderThan = model.query('people').olderThan(50)
    peopleFrom = model.query('people').from('Lisbon')
    people = model.query('people').everyone()
    queries = [
                peopleOlderThan
                peopleFrom
                people
            ]

    model.subscribe queries..., (err, seniors, residents, everyone) ->
                console.log seniors.get()

---

The same happens if the queries are performed independently.

---

model.subscribe peopleOlderThan, (err, seniors) -> console.log seniors.get()
model.subscribe peopleFrom, (err, residents) ->  console.log residents.get()
model.subscribe people, (err, everyone) ->  console.log everyone.get()

---

The first query will return correctly.
The second one will have its own result plus the result from query one.
The third one will return the results from all three queries.

Cannot find module es5-shim

I am not clear on how to get the examples running. Starting from an empty C:\racer directory on Windows XP with node v0.6.3 and npm 1.1.0-alpha-3:

C:\racer>npm install racer
npm WARN [email protected] package.json: bugs['web'] should probably be bugs['url']
[email protected] ./node_modules/racer
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected] [email protected])
├── [email protected] ([email protected] [email protected] [email protected] [email protected] [email protected] [email protected] detect
[email protected])
├── [email protected] ([email protected] [email protected] [email protected])
└── [email protected]

C:\racer>node node_modules/racer/examples/pad/server.js

node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Cannot find module 'express'
at Function._resolveFilename (module.js:334:11)
at Function._load (module.js:279:25)
at Module.require (module.js:357:17)
at require (module.js:368:17)
at Object. (C:\racer\node_modules\racer\examples\pad\server.js:2:11)
at Module._compile (module.js:432:26)
at Object..js (module.js:450:10)
at Module.load (module.js:351:31)
at Function._load (module.js:310:12)
at Array.0 (module.js:470:10)

C:\racer>npm install express
[email protected] ./node_modules/express
├── [email protected]
├── [email protected]
├── [email protected]
└── [email protected]

C:\racer>node node_modules/racer/examples/pad/server.js

node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
Error: Cannot find module 'connect-gzip'
at Function._resolveFilename (module.js:334:11)
at Function._load (module.js:279:25)
at Module.require (module.js:357:17)
at require (module.js:368:17)
at Object. (C:\racer\node_modules\racer\examples\pad\server.js:3:8)
at Module._compile (module.js:432:26)
at Object..js (module.js:450:10)
at Module.load (module.js:351:31)
at Function._load (module.js:310:12)
at Array.0 (module.js:470:10)

C:\racer>npm install connect-gzip
[email protected] ./node_modules/connect-gzip
├── [email protected]
└── [email protected]

C:\racer>node node_modules/racer/examples/pad/server.js
info - socket.io started
Go to http://localhost:3011/racer

C:\racer\node_modules\racer\node_modules\browserify\lib\wrap.js:352
throw new Error('Cannot find module ' + JSON.stringify(mfile)
^
Error: Cannot find module "es5-shim" from directory "C:\racer\node_modules\racer\lib"
at EventEmitter.require (C:\racer\node_modules\racer\node_modules\browserify\lib\wrap.js:352:19)
at C:\racer\node_modules\racer\node_modules\browserify\lib\wrap.js:430:14
at Array.forEach (native)
at EventEmitter.require (C:\racer\node_modules\racer\node_modules\browserify\lib\wrap.js:429:27)
at C:\racer\node_modules\racer\node_modules\browserify\lib\wrap.js:156:14
at Array.forEach (native)
at EventEmitter.addEntry (C:\racer\node_modules\racer\node_modules\browserify\lib\wrap.js:154:22)
at C:\racer\node_modules\racer\node_modules\browserify\index.js:126:15
at Function.bundle (C:\racer\node_modules\racer\node_modules\browserify\index.js:211:12)
at C:\racer\node_modules\racer\lib\racer.js:64:48

Queries Readme link is broken on derbyjs.com

This section is out of date. Please see the Queries Readme for now.

Should link to https://github.com/codeparty/racer/blob/master/lib/descriptor/query/README.md

accessControl session var goes out of scope? maybe?

Still don't know exactly what's going on, figured I'd file after knowing better the bug - but I just can't figure it out. Using the accessControl boilerplate from README.md (see my own implementation here), this.session resolves as undefined at various times:

TypeError: Cannot read property 'userId' of undefined
    at Object.<anonymous> (/Users/lefnire/Dropbox/tyler/Sites/personal/habitrpg/derby/lib/server/setupStore.js:14:37)
    at guard (/Users/lefnire/Dropbox/tyler/Sites/personal/habitrpg/derby/node_modules/derby/node_modules/racer/lib/accessControl/accessControl.Store.js:156:14)

it seems that maybe this.session goes out of scope? The most common way I can replicate this is by closing my laptop and opening it again - which is pretty edge-case I know - but still it fully crashes the server. However, it does crop up otherwise from time to time.

Many changes at once cause erratic behavior

I have some code which calls many gets and sets at once in rapid succession (whether by _.each or async.forEach). The issue is this: sometimes when the code is called, the mutations never get sent to the server - such that on a refresh, the objects are as they were before. Sometimes when that happens, upon refresh the server jumps into a model.set() loop which attempts to call the previously failed sets forever and eventually crashes the server.

This has been my #1 issue as long as I've been developing my app, but I've never been able to pinpoint it enough to file a specific issue - and it happens intermittently. The code starts here (it's spaghetti right now, I'm cleaning it up). I've always assumed it's either because I'm calling set() too fast back-to-back, but it's just a theory. I've also tried model._memory.flush() immediately after all the sets are finished to try to force-commit the changes (is that the right function to call?), but to no avail.

Maybe I should create a video of the problem, it's kinda funky & hard to explain - but definitely debilitating.

racer examples and test

Hi, i'm running ubuntu 11.10 and had trouble building redis-2.2.111-scripting (missing math headers) so resorted to redis unstable branch which appears to include scripting support. I'm unable to get any of the racer examples to work correctly

1. no Todos appear and can't add a new todo
2. "Letters" doesn't ever load in the browser
3. The pad appears, but with "undefined" in the text area and does not sync to another browser page on update.

The follwing lines are characteristic of the messages in the console when trying to update something in the apps model, either in the examples of in an app that I have written myself (based on the examples).
debug - websocket writing 5:::{"name":"txnErr","args":[{},"c.1"]}
debug - websocket writing 5:::{"name":"txnErr","args":[{},"c.2"]}
debug - websocket writing 5:::{"name":"txnErr","args":[{},"c.3"]}

When I run the racer tests with "make test" the following occurs and the tests do not complete (keyboard interrupt after the test stalls for a number of minutes):

100% 310 tests

.
uncaught undefined: Error: ERR Error running script (call to f_8ce78e66e229c050e8ff46a503279dfdce271e82): [string "func definition"]:2: attempt to index global 'os' (a nil value)
at RedisReplyParser. (/home/carl/src/racer/node_modules/redis/index.js:256:27)
at RedisReplyParser.emit (events.js:67:17)
at RedisReplyParser.send_error (/home/carl/src/racer/node_modules/redis/lib/parser/javascript.js:266:14)
at RedisReplyParser.execute (/home/carl/src/racer/node_modules/redis/lib/parser/javascript.js:125:22)
at RedisClient.on_data (/home/carl/src/racer/node_modules/redis/index.js:422:27)
at Socket. (/home/carl/src/racer/node_modules/redis/index.js:66:14)
at Socket.emit (events.js:67:17)
at TCP.onread (net.js:327:14)

uncaught undefined: Error: ERR Error running script (call to f_8ce78e66e229c050e8ff46a503279dfdce271e82): [string "func definition"]:2: attempt to index global 'os' (a nil value)
at RedisReplyParser. (/home/carl/src/racer/node_modules/redis/index.js:256:27)
at RedisReplyParser.emit (events.js:67:17)
at RedisReplyParser.send_error (/home/carl/src/racer/node_modules/redis/lib/parser/javascript.js:266:14)
at RedisReplyParser.execute (/home/carl/src/racer/node_modules/redis/lib/parser/javascript.js:125:22)
at RedisClient.on_data (/home/carl/src/racer/node_modules/redis/index.js:422:27)
at Socket. (/home/carl/src/racer/node_modules/redis/index.js:66:14)
at Socket.emit (events.js:67:17)
at TCP.onread (net.js:327:14)

uncaught undefined: Error: 'Lua lock script should return valid timeout and transaction count' timed out
at Object._onTimeout (/home/carl/src/racer/node_modules/expresso/bin/expresso:980:23)
at Timer.ontimeout (timers.js:84:39)

^C
Failure: Only 1 of 7 suites have been started

make: *** [test-serial-fast] Interrupt

Table editor page of 'sink' example - doesn't property add column

The bug seems to be in racer/lib/mutators/mutators.Model.js (version 0.3.13)

push: {
  type: ARRAY_MUTATOR,
  insertArgs: 1,
  fn: function() {
    var args, at, callback, current, path;
    args = 1 <= arguments.length ? __slice.call(arguments, 0) : [];
    if (at = this._at) {
      //          if (typeof (path = args[0]) === 'string' && (current = this.get()) && !Array.isArray(current)) {
      if (typeof (path = args[0]) === 'string' && (current = this.get(path)) && Array.isArray(current)) {
        args[0] = at + '.' + path;
      } else {
        args.unshift(at);
      }
    }
    if (typeof args[args.length - 1] === 'function') {
      callback = args.pop();
    }
    return this._sendToMiddleware('push', args, callback);
  }
},

The original line is commented with the fixed line following it.

can't install [email protected]

npm install [email protected] ==> error in package.json file at line "colors": "git://github.com/Marak/colors.js.git#master"

Warning: connection.session() MemoryStore is not designed for a production environment

I still get the MemoryStore warning when running Derby in production, even though I'm using store.sessionMiddleware with MongoStore

Warning: connection.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.

My server definitely has a memory leak somewhere, and it has something to do with Mongo, because after RAM has filled up (over the course of a few hours) this guy will crash the server:

/var/www/vhosts/habitrpg/node/node_modules/racer-db-mongo/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/server.js:481
        throw err;
              ^
RangeError: Maximum call stack size exceeded

Maybe these two are related?

Racer assumes path input is valid

The following example code:

model.push('news', {title: params.body.title, abstract: params.body.abstract})

causes the following error:

..../default/node_modules/derby/node_modules/racer/lib/Memory.js:99
      throw new TypeError("" + arr + " is not an Array");
            ^
TypeError: [object Object] is not an Array
    at Object.Memory.push (..../default/node_modules/derby/node_modules/racer/lib/Memory.js:99:13)
    at applyTxn (..../default/node_modules/derby/node_modules/racer/lib/transaction.js:109:32)
    at Object.module.exports.applyTxn (..../default/node_modules/derby/node_modules/racer/lib/transaction.js:82:12)
    at applyTxn (..../default/node_modules/derby/node_modules/racer/lib/adapters/pubsub-memory/channel-interface-query.js:49:15)
    at Function.Object.create.publish.value [as publish] (..../default/node_modules/derby/node_modules/racer/lib/adapters/pubsub-memory/channel-interface-query.js:29:18)
    at PubSub.publish (..../default/node_modules/derby/node_modules/racer/lib/pubSub/PubSub.js:56:10)
    at Store.module.exports.proto.publish (..../default/node_modules/derby/node_modules/racer/lib/pubSub/pubSub.Store.js:179:20)
    at publish (..../default/node_modules/derby/node_modules/racer/lib/txns/txns.Store.js:113:15)
    at next (..../default/node_modules/derby/node_modules/racer/lib/middleware.js:7:26)
    at module.exports.events.middleware.txn (..../default/node_modules/derby/node_modules/racer/lib/txns/txns.Store.js:106:11)

This is because 'news' ought to read 'news.item'. However, this is not at all clear from the output. In fact, it appears that the code actually thought my input was valid, which is a bug in itself. Changing 'news' to 'news.item' causes this error to vanish.

The following happens if you use mongodb as a backend:

..../default/node_modules/derby/node_modules/racer/lib/Store.js:304
      throw new Error('No persistence handler for ' + method + '(' + args.join
            ^
Error: No persistence handler for push(news,[object Object],1)
    at next (..../default/node_modules/derby/node_modules/racer/lib/Store.js:304:13)
    at next (..../default/node_modules/derby/node_modules/racer/lib/Store.js:312:14)
    at Store._sendToDb (..../default/node_modules/derby/node_modules/racer/lib/Store.js:320:10)
    at Store.module.exports.bufferifyMethods.methodNames.forEach.Klass.(anonymous function) (..../default/node_modules/derby/node_modules/racer/lib/util/async.js:51:47)
    at writeToDb (..../default/node_modules/derby/node_modules/racer/lib/txns/txns.Store.js:102:15)
    at next (..../default/node_modules/derby/node_modules/racer/lib/middleware.js:7:26)
    at Object.Lww.incrVer (..../default/node_modules/derby/node_modules/racer/lib/modes/lww.js:33:12)
    at next (..../default/node_modules/derby/node_modules/racer/lib/middleware.js:7:26)
    at Object.module.exports.events.init.store.eachContext.context.guardWrite (..../default/node_modules/derby/node_modules/racer/lib/accessControl/accessControl.Store.js:50:45)
    at accessController (..../default/node_modules/derby/node_modules/racer/lib/txns/txns.Store.js:92:17)

Again, the code assumes my input is valid, and proceeds to fail deep within racer code.

More query methods

We have some basic query methods (where, equals...). It`s enough for simply scenarios. Whan about to add some more?

How to get length of collection and not to get all data of it? Some kind of 'count' method would be needed.

There is 'contains' method for seek value in array. For simple search scenarios 'contains' in string method is absolutly needed.

Also some logic as 'and', 'or' don`t harm.

May be worth to stick to mondodb and it`s queries? Then put queries in separete module, so queries for different db can be written in future.
Universal queries approach is good, but less expressive.

Errors from query execution are silently swallowed

If a query provider returns an error, the error is swallowed silently and an empty result set is returned.

The problem stems from Store._fetchAndCompileQueryData, which accepts a callback with only one parameter.
The parameter is supposed to indicate whether the result is a single item. If the query returns an error, the error is also passed to this parameter.

This method is called by Store.middleware(), but I'm not sure how to report errors through the middleware chain.

Date values set from the client side are persisted as strings

I have noticed this with the MongoDB backend, but I suspect this applies to other adapters as well.

Steps to reproduce:

1. create a route which sets a date on some field: model.set("users.42.updated_at", new Date())
2. access this route directly from the server
3. look at the data persisted in the database, it should be something like this: "updated_at" : ISODate("2012-09-20T11:57:07.637Z")
4. now access this from the client side (e.g. with a link pointing to '/')
5. look at the data persisted in the database, it should be something like this: "updated_at" : "2012-09-20T11:57:03.243Z"

I suspect this is a side effect of how JSON.stringify and JSON.parse handles Date objects.

Ability to use distinct on a field inside of a query

I'd love to be able to use distinct to get a list of unique fields.

In mongo:

db.collection.distinct('field')
or
db.collection.distinct('field', { key: 'value' })

Edit: I tried figuring out how to build out the query myself, but couldn't really get anywhere. I'll keep trying, but I'm having a tough time wrapping my head around the whole suite.

redis auth support

Please add support for client.auth in redis. Without it you no one can use Heroku, dotcloud, or other paas with this framework. Thanks.

Error in multiple queries

racer : 0.3.13
derby : 0.3.13
racer-db-mongo : 0.3.13

I have these queries.

peopleOlderThan = model.query('people').olderThan(50)
peopleFrom = model.query('people').from('Lisbon')

I subscribe them, in either of these ways:

model.subscribe peopleOlderThan, peopleFrom, (err, seniors, peopleFrom) -> 

or

model.subscribe peopleOlderThan, (err, seniors) -> 
        model.subscribe peopleFrom, (err, residents) -> 

When I subscribe them, I get the following error

C:\LogViewer\node_modules\derby\node_modules\racer\lib\descriptor\query\scope.js:187
      throw new Error('Uncaught edge case: ' + method + ' ' + require('util').
            ^
Error: Uncaught edge case: set { '0':
   [ 'people.502a5385769d510064000001',
     { name: 'Pedro Abrunhosa',
       age: 52,
       city: 'Lisbon',
       nif: '66666',
       id: 502a5385769d510064000001 } ],
  '1':
   { name: 'Pedro Abrunhosa',
     age: 52,
     city: 'Lisbon',
     id: 502a5385769d510064000001 } }
    at Model._1 (C:\LogViewer\node_modules\derby\node_modules\racer\lib\descriptor\query\scope.js:187:13)
    at Model.EventEmitter.emit (events.js:115:20)
    at Model.mixin.events.init.memory (C:\LogViewer\node_modules\derby\node_modules\racer\lib\refs\index.js:39:19)
    at Model.EventEmitter.emit (events.js:91:17)
    at Model.module.exports.proto._addData (C:\LogViewer\node_modules\derby\node_modules\racer\lib\descriptor\descriptor.Model.js:162:16)
    at module.exports.proto.subscribe (C:\LogViewer\node_modules\derby\node_modules\racer\lib\descriptor\descriptor.Model.js:87:14)
    at Object.module.exports.server._addSub._clientIdPromise.on.res.send (C:\LogViewer\node_modules\derby\node_modules\racer\lib\descriptor\descriptor.Model.js:208:13)
    at Object.module.exports.events.middleware.middleware.subscribe.add._res.send (C:\LogViewer\node_modules\derby\node_modules\racer\lib\pubSub\pubSub.Store.js:120:39)
    at Object.module.exports.events.middleware.middleware.fetch.add._res.send (C:\LogViewer\node_modules\derby\node_modules\racer\lib\descriptor\descriptor.Store.js:105:27)
    at module.exports.events.middleware.middleware.fetchQuery.add.store._fetchAndCompileQueryData.done (C:\LogViewer\node_modules\derby\node_modules\racer\lib\descriptor\query\query.Store.js:72:17)

C:\LogViewer>

accessControl still allows access to collection

App uses mongoDB which has a collection 'ziink' with a few documents.

In server/index.js
  store.query.expose('ziink', 'test', function(id) {
    return {name: 'ziink'};
  });

  store.accessControl = true;
  store.queryAccess('ziink', 'test', function (id, accept) {
    accept(false);
  });

In app/index.js

ready(function(model) {
  window.model = model;
});

In browser

    model.query('ziink').fetch(function(err, users){console.log(users.get())})

This fetches all documents.

Derby should not allow any access to data that is not explicitly allowed when accessControl is true.

Racer + Backbone.js?

Could we use Racer with Backbone.js on the client? If so would you be able to give some insight as to what this configuration looks like?

I know derby doesn't use backbone but our app is already built with backbone. Could we use racer to sync it's models?

Thanks for any information or insight!

Offline support using localStorage

It would be nice to have a HTML5 localStorage back-end for racer. This would allow an application to work in a truly offline fashion and still have the synchronization capability provided by racer.
I thought this could be similar to the mongodb back-end but on the client side.

The app would get bootstrapped the first time it connects to the server and it then can be used offline. The data would be stored persistently in the browser. As soon as the user is online again it gets synced back to the server as it would usually be done.

Maybe this is already possible using racer but I was not able to find anything about this.

Validation and authorisations through server-side `before` events?

You could give it an API similar to on *, but with an action callback instead of the out parameter.

listener = model.before method, path, (captures..., args..., action, isLocal, passed)->
  if authorised and valid
    action null # attempt to perform the actual model request.
  else
    action "Error message"  # call call the error callback of the action without attempting the perform it.

data.splits in treeLookup leaks memory

Every time a model path is looked up, the full path is split by ., and the resulting split array cached in data.splits.

As far I can tell, this cache is never cleared, so memory usage will grow unboundedly.

The cache doesn't do much good, either, since the array is cloned on every lookup (for mutations)

Why is this there?

Real documentation

I'm considering building something similar to racer simply because the project doesn't have good enough documentation and the bloody thing is in coffeescript.

Real documentation would be appreciated. (and use javascript in your examples in your documentation ಠ_ಠ)

Update Docs: root models must be Objects and not Strings or Arrays

Root array methods doesn't work.

model.set 'list', []
model.push 'list', 1  # doesn't work

model.set 'list.list', []
model.push 'list.list', 1  # works

Seems like Derby allows root arrays, so the above code should work?

Updated via IRC:

<quangv> i think model.set 'list.items', []   model.push 'list.items', 'test'   works though
<quangv> model roots need to be a namespace or something?
<switz> yes
<switz> list is your id
<switz> therefore if you set list, it has to be an object
<switz> it can't be an array or a string
<switz> or a number
<quangv> hey switz, dang I just filed a bug https://github.com/codeparty/racer/issues/71
<quangv> http://derbyjs.com/#paths  made it seem that fruits : [] is allowed...
<quangv> switz, that's weird so it can't be a string either?
<switz> local models can be strings
<switz> local models are prefaced with _
<switz> and are not saved to the server
<switz> or database
<quangv> so model.set('title', 'Fruit store') shouldn't be used either?
<switz> no
<quangv> i see
<switz> but you can do page.render('index', {title: 'Fruit store'})
<quangv> okay I am going to copy & paste our convo to the bug i just filed and suggest the documentation be changed.

queries (using derby) doesn't follow established naming convention of ids

Using paths, one can do:

model.subscribe( 'document.*', function( err, scoped_model ) {

    assert( 
          model.get( 'document.' + <id#1> )
        ===
          {
            _id: 'test',
            other_field: 'test'
          }
      ); // TRUE

  });

But using queries, this is the current behavior:

model.subscribe( model.query('document').where('_id').within( [ <id#1>, <id#2>, ... <all ids> ] ), function( err, scoped_model ) {

    assert( 
          model.get( 'document.' + <id#1> )
        ===
          {
            _id: 'test',
            other_field: 'test'
          }
      ); // FALSE

      // But
      assert( 
          model.get( 'document.' + <id#1> )
        ===
          {
            id: 'test',
            other_field: 'test'
          }
      ); // TRUE


  });

Notice the difference of the name of the id field. Also worth noticing is that it's still (only) possible to query on the _id field.

(PS. I'm using the racer-mongo adapter and a MongoDB in the backend - dunno if that info helps though)

ability to use xhr-polling

Looks like https://github.com/codeparty/racer/blob/master/src/Store.js#L88 sets up IO for websockets. It appears configurable (racer.get('transports')), but I can't figure any way to configure it. Also, as per setting up Socket.IO on Heroku, io.set("polling duration", 10); is necessary, but there's no room for it in the code here.

Is there a way to configure IO settings in a DerbyJS / Racer app? If not, any chance of adding that capability?

Examples aren't working

For instance: http://todos.racerjs.com/racer

Allow access to underlying socket.io object

So socket.io can be further configured if necessary:

racer = require 'racer'
store = racer.createStore()

store.io.set 'log level', 1

New queries not working.

I have got following code:

store.query.expose('users', 'potentialWinners', function(min){
    return this.where('bid').gte(1).sort(['bid', 'asc']);
});

var query = model.query('users').potentialWinners(1);

So far so good, but as soon as I try to fetch something with this query, I get an ugly error.

model.fetch(query, function(e, pw){
    console.log(pw.get());
});

Here is the error from console

TypeError: Cannot read property 'query' of undefined

Can't set reference model

This is a derby app:

I am fetching a query of docs and then filtering it. If it has audioOnly it goes into

model.ref '_audioVideos', videoModel.filter({where:{'audioOnly': true}})

, and if not it goes into

model.set '_videos', videoModel.filter({where:{'audioOnly':false}}).get()

This works great, but later, I try to set something on '_audioVideos' through an x-bind. According to the derby docs ("Setting a property of the reference path modifies the underlying data"), I should be able to do this, but I get an error. I'm simply trying to set an existing field from false to true.

Console test:

> model.set('_audioVideos.0.report', true)
Error: set on undefined refList child _$queries._3.results.4fb0694edd78bc5ad304785a.report

It looks as though Racer is checking for the _id when it should be using the index. Should be: _$queries._3.results.3.report

Model.js / Uncaught TypeError: Cannot call method 'apply' of undefined

function eventListener seems to get a null callback. Will update if/when I get more details. Don't know if this is a serious issue or now, so please close if it's not.

Uncaught TypeError: Cannot call method 'apply' of undefined Model.js:270
(anonymous function) Model.js:270
EventEmitter.emit events:68
module.exports.proto._applyMutation txns.Model.js:433
module.exports.proto._applyTxn txns.Model.js:410
module.exports.proto._specModel txns.Model.js:444
resend txns.Model.js:229

Unable to unsubscribe queries

When unsubscribing queries I get the error 'ReferenceError: queryJson is not defined'

The error is in unregisterSubscribe

I tried fixing the error, but the entire unsubscribe call chain is full of bugs (at least for queries) and I don't understand how it's supposed to work so I can't fix it.

To test this you can simply define any query and then run:

var q = model.query('row').all(); //my test query
model.subscribe(q, function(){});
model.unsubscribe(q, function(){});

Query byId() not run again for an ID even if returned fields needed are different

  store.query.expose 'users', 'getEmail',  (userid) ->
    @byId(userid).only(['email']);

  store.query.expose 'users', 'getName',  (userid) ->
    @byId(userid).only(['name']);

If you run the first query for a given user and then run the second query for the same user, you still get only the 'email' field. This is with 'racer-db-mongo'

Racer loads data twice

Using Derby, I subscrube to all data on server side, so data loads inline with with first response from server.
When scripts are loaded to browser and websocket connection estabilished, socket emits 'resyncWithStore', so all data loads to browser second time.
I think that it`s bug in reconnect.Store.js file.

"PubSub" => "pubSub" ?

Should the folder PubSub not be called pubSub? In my envirnoment I received an error the module "pubSub" could not be found. So I renamed the folder and it works.

model.fetch() returns null in middleware

I'm almost done with a Derby example in EveryAuth. The problem is in findOrCreateUser(). It uses model.fetch(query, ..) to determine if the user exists, and model.set(...) to create new user otherwise. However, that model.fetch keeps returning {err:null, user:null}, yet the same query run outside of EveryAuth's middleware (aka, on the client) works properly and returns a user.

Here's the code where it's flopping, and check out server/index.js for which locations call these functions.

Default route redirects to http://127.0.0.1:3010/lobby instead of $DOMMAINNAME/lobby

Just a heads up

racer-db-mongo 0.3.12 causes an odd "null" error object to be thrown

The following occurs:

..../nodejs/default/node_modules/racer-db-mongo/node_modules/mongoskin/node_modules/mongodb/lib/mongodb/connection/server.js:428
        throw err;
              ^
null

With the following installation:

[email protected] node_modules/gzippo
└── [email protected]

[email protected] node_modules/express
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected])

[email protected] node_modules/racer-db-mongo
└── [email protected] ([email protected])

[email protected] node_modules/derby
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected])
└── [email protected] ([email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected])

This should happen simply by trying to use mongodb as a storage backend. racer-db-mongo 0.3.11 works.

model.unsubscribe always requires a callback

If I use model.unsubscribe(path), I get following exception:

ReferenceError: finishAfter is not defined
    at Function.unsubscribe (C:\git-repositories\winery\winery-modeller\node_modules\racer\lib\descriptor\query\QueryHub.js:109:18)

If I submit an empty callback function, everything works fine.

False warning re MemoryStore when using connect-mongo

Set NODE_ENV=production in chat example and you get the warning:

Warning: connection.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.

Racer patches the memory store to redirect to a persistent store, yet 'connect' gives the above warning.

The easiest fix might be to check for the persistent store and alert user to ignore the following warning.

Sync/Pub/Sub between two or multiple node process through racer-db-mongo ?

Is it possible to create racer server in one node process and a racer client in another node process and they are able to pubsub data modification between them through racer-db-mongo plugin ?

Allow persistent server models that can be used with #bundle

Currently, Model#bundle will cause post-bundle method invocations that depend on Model#store to fail. It would be nice to mark a model as always being connected to the store on the server. Here's an example scenario with the proposed new syntax:

var model = store.createModel({persistent: true});  // NEW

// Now, we can do this successfully!
physicsEngine.eachStep( function (x, y) {
  model.set('games.global.pos', {x: x, y: y});
});

model.subscribe('games.global', function (err, game) {
  model.bundle(...);
});

app.on('all players left game room', function () {
  model.cleanup(); // NEW
});

Include compiled "lib" folder in racer module (windows users)

Hi,

I'm using windows for developing with DerbyJS and Racer, but I can only use racer if I install it using 
           - "npm install racer"
           - installing it through Derby package.json (which is the same as npm install racer)

DerbyJs's repository already has the compiled "lib" folder. Could you add the same for racer, so windows users can use the latest commited version, instead of the latest version in the npm repo?

Thank you

Apps created using 'derby new appname' dont work

I've created a derby app using 'derby new test' and tried to run it but got an error after requesting...

Details here:
http://stackoverflow.com/questions/13475899/derby-app-created-using-derby-new-test-not-working

Query byId() reads in the entire collection from the db (mongodb) before selecting the one to return

This is with 'racer-db-mongo'. Obviously this inefficient and could cause major problems.

Bug in the 'pad' example

opening pad.racerjs.com/racer I get this error in the javascript browser console:
"Uncaught Error: Cannot find module '../../node_modules/share/lib/types/text' "

readPathAccess & writeAccess not executing

I can't get store.readPathAccess() & store.writeAccess() to be called. See the console.log which isn't executed here, and the resultant "Unauthorized: No access control declared" here

The three demos in the README are down

[Feature Request] Validation Before CRUD

Hi,

 I see theres yet to be implemented a way for performing validation, for example, before inserting data into a DB. 

 Since Racer saves us a lot of trouble and work by doing everything under the hood, this poses a problem when dealing with mal-intentioned users, or client-side validation faults. 

 I feel it is very important to have this last line of defence. I might be out of my way, suggesting this. You probably already thought about this, but I gave a quick read in the issues and commit sections and didn't find anything about it.

 I'm using DerbyJS, and since most of the code is client side, I think this is indispensable.