mikemintz/rethinkdb-websocket-server

Node.js WebSocket server that proxies to RethinkDB. Supports query validation.

rethinkdb-websocket-server's Issues

Support websocket compression

In index.js, perMessageDeflate is set to false (here). There's a accompanying comment indicating that this was because of a bug in websockets (here). This bug has been fixed and it would be great if compression was supported in this package, potentially even by default.

I did some testing in my environment and everything seems to work perfectly when compression is enabled, and delivers significant reductions in network traffic.

Query logging prints binary data and long arrays, flooding stdout

When unsafelyAllowAnyQuery is on, rethinkdb-websocket-server prints every query with all of its data to the console. This includes r.binary() stuff like uploaded files and arrays with binary data (say, when storing UInt8Arrays in the DB as a part of the object). A couple of megabytes sent to stdout can really cause havoc. I think the data should be trimmed in the output after an X number of characters, plus there should be an option to turn off showing queries altogether. There are legitimate use cases for not using any validation and running the server, e.g. when running unit tests that should bypass validation or when running the server locally and just wanting to connect to the server instance from a web page, with full control.

Row-level security / automatic filters

It would be great if a client could do something like r.table('turtles').filter({name: 'Ralph'}), and the server would automatically inject the herdId filter (from your example) in between the .table and the .filter provided by the client, so that from a client's perspective, the only data in the DB is what is available to them. This could ease the burden not only on user-level security, but help to support multi-tenancy (e.g. filtering every table on tenantId).

Support react-native

Let's open

#9

Since React Native support isn't there. If it is please create an issue to document it with a clear example. Currently getting "Unhandled rejection ReqlTimeoutError: Could not connect to localhost:3030 operation timed out".

This error takes about 30 seconds to show up. If we turn off the server it shows up right away, so it seems that it kind of works at first but then fails to reconnect.

If someone could help me make this work because it actually can work, I'd be happy to write the documentation.

Support for RethinkDB 2.3.x

Will you be updating the client and server to support 2.3.x in the future? What would need to be done to update?

I think the following change in the 2.3 driver is breaking it at the moment.

Disallowed calling r.wait, r.rebalance and r.reconfigure on the global scope to avoid confusing semantics. (#4382)

TypeError: _rethinkdb2.default.wait is not a function at Object.<anonymous> (/home/tersiusk/WebstormProjects/pinkdot/node_modules/rethinkdb-websocket-server/dist/ReqlTermExamples.js:109:32)

Pass request object along with urlQueryParams

urlQueryParams seems extremely dependent on authentication with the same server. If I want to authenticate via another server (to include third-party) and then have a JWT for example, I need access to the request headers. Passing the request object as a second parameter would allow this and then the JWT could be authenticated.

Native reql syntax in query whitelist

It might be nice to express queries in the whitelist using the natural JS reql syntax, rather than the protocol AST syntax we currently use. Instead of this:

RQ(
  RQ.INSERT(
    RQ.TABLE("messages"),
    {body: RQ.ref("body"), userId: RQ.ref("userId"), createdAt: RQ.NOW()}
  )
).opt("db", RQ.DB(cfg.dbName))
.validate(...)

we'd allow writing this:

r.db(cfg.dbName)
 .table('messages')
 .insert({body: RP.ref("body"), userId: RP.ref("userId"), createdAt: r.now()})
 .validate(...)

The biggest challenge is being able to log rejected queries in this format, so the application developer can copy/paste into their whitelist. The reason we have the current whitelist syntax is because it's derived directly from the AST sent over the wire protocol.

My hypothesis is that we can go from AST to JS reql syntax with the following algorithm:

(1) Start at the inner-most 0th arg of the nested arrays.

let x = query;
while (Array.isArray(x.args[0])) {
  x = x.args[0];
}

(2) Work our way from the inside to the outside. Each time we go out one level, append the term name of the outer term to the current query, and use arguments after the 0th as its arguments, and options as an object for the last argument. For each argument, recursively apply this algorithm, making special cases for terms like FUNC and MAKE_ARRAY.

x = x.parentTerm;
result += '.' + termNames[x.termId] + '(';
result += x.arguments.slice(1).map(formatTerm).join(', ');
if (x.options) result += ', ' + formatOptions(x.options);
result += ')';

Error: Uncaught SyntaxError: Unexpected token in JSON

We've been experiencing a weird bug off and on over the past few months, and lately it seems to be getting worse.

• Symptom: a syntax error in JSON parsing on the client. It's like part of the JSON message gets dropped or mangled in transmission. For example:

Error: Uncaught SyntaxError: Unexpected token , in JSON at position 289587

• Clients: Chrome on OS X and Windows. It seems to occur much more frequently on OS X, and possibly not at all on Windows (we haven't been able to repro on Windows)
• Versions: rethinkdb-websocket-server v0.5.1, react-rethinkdb v0.5.1
• It seems to occur much more frequently (possibly exclusively) when the returned object is big / complicated.
• Sometimes it happens 50% of the time for big objects, sometimes 0%, sometimes 100% of the time
• We're using wss://

I'm going to keep looking into this but I don't have many ideas about where to start. Does anything jump out?

ws version bump due to RangeError: out of range index crashing server

Is there much involved in bumping the ws version to >1.1.1? For now I manually bumped the version in my projects and it's working fine.

I've been seeing this bug a lot: websockets/ws#778

Authentication with username/passsword

Hi, it seems that it is currently not possible to connect to rethinkdb providing username and password. Since authKey is not present in the latest version of rethinkdb, the only way to get this proxy working with db is leaving db unsecured.

Meteor-style allow/deny functions

Great work on this! I'm excited to try it out.

I haven't yet grokked the queryWhiteList bit - I think I need to read the code a couple more times. I was wondering if you had any thoughts about the feasibility of an auth API similar to the allow/deny functions of Meteor? I can't tell yet how much effort that would take. I also like Firebase's approach.

In both cases there is a clean, easy-to-read mapping of data paths to functions which allow or deny writes and reads, given user/session data as well as snapshots of the current state of database when relevant.

using server only session data in filter

I have a situation where the user sends a token from client via query params and I use the same to validate the session and store the the userId in the RP session.

options.sessionCreator = function (params) {
  let token = params.token;
  let user = params.user;
  if (!token || token == "" || !user || user == "") {
    return Promise.reject('Invalid auth token');
  } else {
    // use teade to validate the user from user service
    const clients = {
      users: {
        host: process.env.SERVICE_RPC_HOST,
        port: process.env.CXI_USER_PORT
      }
    }
    const data = {
      token: token,
      user: user
    }
    const authenticator = require('./middlewares/authenticator');

    return Promise.promisify(authenticator.auth)(clients, data)
      .then(function(result){
        return Promise.resolve({userId: result.data.id});
      }).catch(function(err){
        return Promise.reject(err.message);
    })
  }
};

now I want to add this userId in the filter of a query but I dont have this id in the frontend.

So I am looking for a way to either

1. add a filter to the query on the server (maybe in validate) and not send the aforementioned filter from frontend
2. modify the filter data on the server no matter what is sent from the front end.

// whitelisted query
r.table("trades").orderBy({index: allowSpecificFieldOrderBy(RP, "datetime")})
            .filter(r.row("currency").eq(allowAnyString(RP)))
            .filter(r.row("isExecuted").eq(true))
            .filter(r.row('userId').eq(
                 /* what can I use here? */
             ))
            .pluck("datetime", "rate", "id")
            .changes({"includeInitial": true}).opt("db", r.db("cxihub"))
            .validate(function (refs, session) {
                // console.log(refs, session)
                // { userId: '' } { userId: 1 }
                return true;
            })

Automatic reconnect on db connection interruption

If the connection to the database gets interrupted (e.g. database is restarted), the server seems to crash permanently.

It would be great if the server could reconnect when the connection is available again.

For example, this stacktrace is what happens to the Chat example when the database is restarted (and in perpetuity until the websocket server is restarted).

22:21:18.676 172.17.0.5:54391 Error in sessionCreator
ReqlDriverError: Connection is closed.
    at ReqlDriverError.ReqlError [as constructor] (/app/node_modules/rethinkdb/errors.js:23:13)
    at new ReqlDriverError (/app/node_modules/rethinkdb/errors.js:68:50)
    at TcpConnection.Connection._start (/app/node_modules/rethinkdb/net.js:413:13)
    at /app/node_modules/rethinkdb/ast.js:142:29
    at Get.TermBase.run (/app/node_modules/rethinkdb/ast.js:131:12)
    at AuthManager.js:17:47
    at tryCatcher (/app/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/app/node_modules/bluebird/js/release/promise.js:497:31)
    at Promise._settlePromise (/app/node_modules/bluebird/js/release/promise.js:555:18)
    at Promise._settlePromiseCtx (/app/node_modules/bluebird/js/release/promise.js:592:10)
    at Async._drainQueue (/app/node_modules/bluebird/js/release/async.js:130:12)
    at Async._drainQueues (/app/node_modules/bluebird/js/release/async.js:135:10)
    at Immediate.Async.drainQueues [as _onImmediate] (/app/node_modules/bluebird/js/release/async.js:16:14)
    at processImmediate [as _immediateCallback] (timers.js:383:17)

Connect trouble with Firefox

Hi.

I'm having an issue with firefox clients failing to connect to rethinkdb-websocket-server.
Connect works as expected for Chrome and chromium browsers.

The error message is also quite amusing :) Any idea what is going on here?

Ingar

Support rethinkdb auth key

We have some basic security on our RethinkDB instances with the built-in authKey support for securing drivers. However, this library doesn't seem to support that.

RethinkDB 2.2 compatibility

I'm using 0.3.2 and now I'm getting this error

/app/node_modules/rethinkdb-websocket-server/dist/ReqlTermExamples.js:109
  WAIT: _rethinkdb2['default'].wait(),
                               ^
TypeError: undefined is not a function

I haven't had any problems with that before..

Support react-native

Hi, I'm trying to use the library but It never connects with rethinkdb server, log just prints:
04:14:52.518 ::ffff:192.168.99.1:54361 Connect
04:15:12.548 ::ffff:192.168.99.1:54361 webSocket closed
04:15:12.549 ::ffff:192.168.99.1:54361 dbSocket closed

Code on nodejs:

console.log('Server online!');
//First Part
var r = require('rethinkdb');
r.connect({host: "rethinkdb", port: '28015'}, function (err, connection) {
    r.db('db').table('users').insert({
        "id": 3,
        "title": "Lorem ipsum",
        "content": "Dolor sit amet"
    }).run(connection, function (err, result) {
        console.log(result);
        connection.close();
    });
});


//Second Part
var express = require('express');
var http = require('http');
var wsListen = require('rethinkdb-websocket-server').listen;
var options = {};
var httpServer = http.createServer();
options.dbHost = 'rethinkdb';
options.dbPort = '28015';
options.unsafelyAllowAnyQuery = true;
options.httpServer = httpServer;


wsListen(options);
httpServer.listen(3000);

As you can see on the first part I'm testing the connection directly, second part is not responding

Thanks for your help

Duplicate lines of code in Connection.js

When looking around this code I noticed a duplicate

https://github.com/mikemintz/rethinkdb-websocket-server/blob/master/src/Connection.js#L29
https://github.com/mikemintz/rethinkdb-websocket-server/blob/master/src/Connection.js#L45

Perhaps I'm missing something here?

Use with existing Express/port

I'd like to use this with an existing Express application that is already listening on a port.

Is it possible to do that, i.e. plug this in to a specific route on an existing web server, instead of listening on its own port?

cant query list using filter

cant query white list using filter

r.table('table')
.filter(r.row("amount").ge(100))
.run(conn, (err, cursor) => {
cursor.toArray((err, results) => {

	    	})

//queryWhiteList

r.table('table')
.filter(
r.row("amount").ge(RP.check((actual, refs, session) => actual === 100))
)
.opt("db", r.db("dice"))

Before/after-execute query callbacks

It would be nice if developers could add before-execute and after-execute callback functions to their queries in the whitelist. Some use cases:

• Send push notifications after a row is inserted
• Record analytics for a given action
• Record some state in the user's session object
• Modify a query before inserting into database (e.g. password hashing could happen in before-execute instead of separate ajax call).

Technically, before-execute is already possible using the .validate() method. Are there use cases where we'd want a separate before-execute callback with different semantics? Some thoughts:

• The .validate() function must return a truthy value or the query will be rejected. Having non-validation callbacks end in return true might clutter things or lead to bugs.
• Should we be able to modify the query before forwarding it to RethinkDB? Would there be issues if rethinkdb-websocket-client got results for a query different from what it thought it executed?

For after-execute, there are two possible variants: one that gets called right after the query is sent to RethinkDB, and another that gets called right after the query response is received from RethinkDB. The former is trivial to implement, and the latter requires parsing RethinkDB responses and tracking query tokens.

Query validation of array data

What's the best way to validate a query like this? For example, to ensure id > 0

r.table('test').insert([{id: 1}, {id: -1}]) // Query

r.table('test').insert(RP.check(arr => arr.every(e => e.id > 0))) // Whitelist entry

Whitelisting like this fails, I think because arr is a parsed query as opposed to a js array. I'm not sure how to loop through it.

Proper way to use on Heroku?

I've been having a lot of trouble deploying this app on heroku and using it in conjunction with the client. What is the proper way to securely run this in production, with security? I get immediate timeouts when I try to use TLS.