mongoosejs/mquery

Expressive MongoDB query builder

mquery's Issues

read prefs are broken

When passing a mongo.ReadPreference instance, the query fails.

Error: Illegal readPreference mode specified, primary

mongo shell compatible

Guard against RangeErrors

I got a RangeError: Maximum call stack size exceeded from within mquery. I am not 100% sure yet how this happened and with what data, but it was after trying to do an update operation on a Mongoose model.

Maybe mquery could implement a guard against this though?

Relevant stack trace portion:

node_modules/mongoose/lib/model.js in Function.update at line 2059
node_modules/mongoose/lib/query.js in Query.update at line 2192
node_modules/mquery/lib/mquery.js in Query.update at line 2042
node_modules/mquery/lib/mquery.js in Query.merge at line 1655
node_modules/mquery/lib/utils.js in Object.merge at line 161
node_modules/mquery/lib/utils.js in merge at line 161
node_modules/mquery/lib/utils.js in merge at line 161
node_modules/mquery/lib/utils.js in merge at line 161
node_modules/mquery/lib/utils.js in merge at line 161
node_modules/mquery/lib/utils.js in merge at line 161
node_modules/mquery/lib/utils.js in merge at line 161

And this goes on:

Add support for per query timeouts

MongoDB 2.6 now supports per query timeouts. Simply append .maxTimeMS(1000) to the query to take advantage of it.

See http://mongodb.github.io/node-mongodb-native/driver-articles/anintroductionto1_4_and_2_6.html#maxtimems for more details.

Expose underlying cursor object for finds

See Automattic/mongoose#1908.

browser compatible

validate options

some operations like distinct do not accept limit, sort, etc.

error when attempted

How to perform nested queries?

If I need to do write a query like this, where the AND is nested under an OR. How do I accomplish this? Does mquery support it? Or do I need to use regular syntax? In other words, can I pass a query as a parameter to OR in this case:

{
        _id: objId,
        $or: [{ 
            $and: [
                {
                    mayView: {
                        $all: ['owner']
                    }
                }, 
                {
                    owners: {
                        $all: [userId]
                    }
                }
            ]
        }, {
            mayView: {
                $all: [userRole]
            }
        }]
}

hint() does not support strings?

It seems like this is indeed the case, unless I'm missing something.
https://docs.mongodb.com/manual/reference/method/cursor.hint/#cursor.hint

According to mongodb documentation ^ , it is possible to pass a string (index name) to the hint() method.

Would really love this to be implemented, to be honest.

$geoWithin

http://docs.mongodb.org/manual/reference/operator/geoWithin/

count with hint

Why can't I use count with hint? So that count operation would take advantage of provided index. I can do this with both mongodb native driver and mongodb shell.

Missing "insert" command

These is an "update" command, but these is no "insert" command. Why? Can you implement this, so I can use the same mquery semantic. It would by nice, if you include all collection base commands like "ensureIndex" and so on.

Outdated bluebird

Hey, have you considered updating bluebird to newer 2.x version? I would be glad if it was at least v2.9.31 because it fixes this bug.
Thanks.

support pre-exec hook

provide way for users to receive/modify conditions (casting for example), options, & fields before executing the query.

Data loss with findOneAndUpdate, $push, and Buffer

Please see the original issue here.

I ran into the issue above while using mongoose which I think might originate with mquery.

• If this is an mquery issue, we can migrate it to this project and close the linked issue.
• If this isn't an mquery issue, feel free to close this issue.

Query On Using Mod in MQuery

Hi Heckmann

I have been sourcing for information on how to do the following in mongo but always fail till I saw your git.
Data (data format: id, price, flag)
id_1, 11, false
id_2, 22, false
id_3, 33, false,
id_4. 44. false
id_5, 55. false.
id_6, 66. false

I am using TS and will like even row be flag as true. How can I do it using mquery. Can help? Thank

$geoNear

http://docs.mongodb.org/manual/reference/command/geoNear/

Add support for sorting by text search score

With MongoDB 2.6 text search is finally available! Text search results can be sorted by text score, but the syntax is different from standard sorts.

To sort by text search score, the following needs to happen:

1. Project the text score to a key name:

Model
.find()
.select({ score: { $meta: "textScore" });

2. Add the sort value:

Model
.find()
.select({ score: { $meta: "textScore" })
.sort({score: { $meta: "textScore" }});

Aggregate method

Is there any plans to add an aggregate() method any time soon, or do you have any pointers for how to add such a thing myself?

.select({ ... }) fails when there's a field named 'length'

Automattic/mongoose#3903

Different mongo driver versions in mquery vs mongoose

As of now, we have the following versions:

[email protected] node_modules/mongoose
...
├── [email protected] ([email protected], [email protected])
└── [email protected] ([email protected], [email protected])

As you can see, [email protected] requires [email protected], and [email protected] which itself requires [email protected]. Two versions of mongodb driver are loaded, which is +150ms to startup time and a source of magic incompatibility errors.

Keeping versions in sync would help, but is prone to forgetting, etc. Maybe it's better to provide external mongodb driver reference to the mquery before use, i.e.:

mongodb = require('mongodb')
mquery = require('mquery')(mongodb);

$isolated

http://docs.mongodb.org/manual/reference/operator/isolated/

Chained query incorrectly merges conditions

The following query should return no results because an id cannot satisfy both conditions:
model.find({id:'a'}).find({id:{$ne:'a'}})
However, only the last condition is processed, resulting in the condition:
{id:{$ne:'a'}}
Instead of overwriting the first condition, the two conditions should be combined with $and so that the condition is:
$and:[{id'a'},{id:{$ne:'a'}}]

The following edit in Query.prototype merge is an example of what may work (tested)

Query.prototype.merge = function (source) {
  if (!source)
    return this;
...

//plain object
//NEW CODE
    if (Object.keys(source).length) {
        if (Object.keys(this._conditions).length) {
            if (this._conditions.$and && this._conditions.$and.length)
                this._conditions.$and.push(source)
            else
                this._conditions = {$and: [this._conditions, source]};
        }
        else
            this._conditions = source;
    }
//END NEW CODE

  return this;
}

Conditions are not parsed inside $not expression

Hi,

I'm using Mongoose.js now. While I'm developing my back-end application, I found a bug about $not expression.

If I query using a condition below, it is parsed well.
ex1)
let condition = {"createdAt":{"$gte":"2016/09/02 00:00:00", "$lte":"2016/09/02 23:59:59"}};
mongoose log:
=> Mongoose: collection.find({ createdAt: { '$gte': new Date("Thu, 01 Sep 2016 15:00:00 GMT"), '$lte': new Date("Fri, 02 Sep 2016 14:59:59 GMT") } }, { fields: undefined })

But if I use $not expression inside this condition, date string isn't parsed.
ex2)
let condition = {"createdAt":{$not:{"$gte":"2016/09/02 00:00:00", "$lte":"2016/09/02 23:59:59}}}
mongoose log:
=> Mongoose: collection.find({ createdAt: { '$not': { '$gte': '2016/09/02 00:00:00', '$lte': '2016/09/02 23:59:59' } } }, { fields: undefined })

What should I do to fix this?

Sincerely,
Jeongsang Baek

How to `mquery.or` multiple queries?

I have a table that can be filtered using various OR and AND clauses on any of the fields.

var nameNotEmpty = mquery.where('name').ne('')
var nameIsNotMike = mquery.where('name').ne('mike')
var ageQuery = mquery.where('age').gt(21)

Is it possible to combine the three queries using OR clause?

Something like:

var allQueries = [nameNotEmpty, nameIsNotMike, ageQuery];
var finalMquery = mquery.or(allQueries);

I am trying this but I am getting a cyclic dependency detected error. Not sure if it is due to mquery or due to something else.

npm install mquery - not working!

There is some issue with sliced module.

yogsagot@debian:~/Documents/tarotlogexpert/app$ npm install mquery
npm ERR! path /home/yogsagot/Documents/tarotlogexpert/app/node_modules/.staging/sliced-76f37b1b
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename '/home/yogsagot/Documents/tarotlogexpert/app/node_modules/.staging/sliced-76f37b1b' -> '/home/yogsagot/Documents/tarotlogexpert/lib/node_modules/mquery/node_modules/sliced'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR! /home/yogsagot/.npm/_logs/2018-04-18T18_21_36_524Z-debug.log

specify requirements of a collection object

Required

• find
• findOne
• count
• distinct
• update
• remove
• findAndModify
• aggregate
• geoNear
• geoHaystackSearch

Maybe

Unsure of these.

No Use

Don't need a query builder for these.

• createIndex
• ensureIndex
• indexInformation
• dropIndex
• dropAllIndexes
• reIndex
• options
• isCapped
• indexExists
• indexes
• stats
• mapReduce
• group
• save

Failure in utils.js - clone()

While trying a MongoDB update through mongoose (on a document containing a reference to another Model), the clone() function errors out on utils.js:32 complaining about obj.constructor.name not given, which IMO stems from an (empty?) object passed down and created via Object.create(null), so that the constructor property is not set.

stack trace:
Function.Model.findByIdAndUpdate - mongoose/lib/model.js:1338:32
Function.Model.findOneAndUpdate - mongoose/lib/model.js:1244:13
Query.findOneAndUpdate - mquery/lib/mquery.js:2036:10
Query._mergeUpdate - mquery/lib/mquery.js:2194:11
Object.mergeClone - mquery/lib/utils.js:176:17
clone - mquery/lib/utils.js:33:20
Object.cloneObject - mquery/lib/utils.js:66:13
clone - mquery/lib/utils.js:30:20
Object.cloneArray - mquery/lib/utils.js:98:14
clone - mquery/lib/utils.js:32:35

Vulnerability issue in debug package

Vulnerability is in ms package used by debug. Please refer to https://snyk.io/vuln/npm:ms:20151024.

debug package v2.6.7 has addressed this issue.

permissions

Is there a reason that permissions.js only has values for findOneAndUpdate and findOneAndRemove but not for find?

https://github.com/aheckmann/mquery/blob/master/lib%2Fpermissions.js

add ability to get `.find()` cursor directly

For make benefit glorious explain() function for applications like https://github.com/vkarpov15/mongoose-explain, re: mongodb-js/mongodb-core#96 (comment) since setting explain via option to find() is not technically officially supported by the driver.

overwrite option for findAndModify ops

like update()

mongo env

Perhaps rather than merge everything into a single file, we just build/concat it? Unless you guys are adding a bare-minimum module system to the V8 powered shell. ;-)

npm install --save doesn't install mquery

Here's what happens:
npm install mquery
installs the package in node, but doesn't add it in my package.json as a dependency. But when I
var mquery = require("mquery")
the module works fine, this means it's actually installed.
But when I
npm install --save mquery
it throws an error saying that
"cannot find module mquery", and the package is neither installed on node nor added in package.json.
Any help will be appreciated

geoHayStack

update should retain key order

Long causes Client Error: bad object in message: invalid bson type

Using the MongoDB Long/NumberLong type in queries can cause a few different errors in mquery. For example:

mquery(collection).update(
    {_id: "example"},
    {_id: "example", number: mongodb.Long.fromString("12345")},
    {upsert: true},
    function(err, num) {
      if (err) throw err;
      console.log("Updated " + num);
    }
);

Causes Client Error: bad object in message: invalid bson type while

mquery(collection).update(
    {_id: "example"},
    {number: mongodb.Long.fromString("12345")},
    function(err, num) {
      if (err) throw err;
      console.log("Updated " + num);
    }
);

Causes Client Error: bad object in message: bson length doesn't match what we found

The root of the problem appears to be that utils.clone and utils.cloneObject do not properly handle the mongodb.Long type (and likely Timestamp and possibly other non-native BSON types).

This could be fixed by adding another obj.constructor special case to clone for these types. Alternatively, the reason that the clone doesn't currently work is that functions are cloned by converting their source to a number and calling the Function constructor, which doesn't work for the functions on the Long prototype. In either case, it shouldn't be too bad to fix.

Cheers,
Kevin

Aggregation Support

Please add the support for aggregation pipeline

find fails with mix of including and excluding fields

find fails with mix of including and excluding fields. Here is the error message:

MongoError: You cannot currently mix including and excluding fields. Contact us if this is an issue.
      at Object.toError (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/utils.js:110:11)
      at Cursor.nextObject.self.queryRun (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/cursor.js:634:54)
      at Cursor.close (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/cursor.js:903:5)
      at Cursor.nextObject.commandHandler (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/cursor.js:634:21)
      at Db._executeQueryCommand (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/db.js:1670:9)
      at Server.Base._callHandler (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/connection/base.js:382:41)
      at Server.connect.connectionPool.on.server._serverState (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/connection/server.js:472:18)
      at MongoReply.parseBody (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
      at Server.connect.connectionPool.on.server._serverState (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/connection/server.js:430:20)
      at EventEmitter.emit (events.js:96:17)
      at _connect (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:191:13)
      at EventEmitter.emit (events.js:99:17)
      at Socket.exports.Connection.createDataHandler (/var/local/webroot/github-forks/mquery/node_modules/mongodb/lib/mongodb/connection/connection.js:384:22)
      at Socket.EventEmitter.emit (events.js:96:17)
      at TCP.onread (net.js:397:14)

Error: Cannot find module 'debug' (node v8+)

debug package used in /lib/mquery.js:11:13 is no where to be found. The package is not available in node or not added to mquery as a dependency package to be installed.

node stream

maximum call stack size exceeded

hey mr heckmann!
there is an error in mquery that is triggered by mongoose. thought I'd make a link from here to the mongoose error
Automattic/mongoose#1920

Include collection name in debug logs

Faced with a stream of logs being written by mquery, it's difficult to folllow what's happening:
mquery findOne +118ms { _id: 53ee3e60dffe60d65e6044f7 } { fields: undefined }
mquery findOne +2ms { _id: 5416dcecca114e3e3d480301 } { fields: undefined }
mquery findOne +1ms { _id: 543d54431bd9b060511d5bc6 } { fields: undefined }
mquery findOne +0ms { _id: 545d09ca602db438755d537b } { fields: undefined }
mquery findOne +1ms { _id: 54608d1ffb62257748086449 } { fields: undefined }
mquery findOne +1ms { _id: 546a241dab87fe2440cd4b93 } { fields: undefined }
mquery findOne +1ms { _id: 5489a989ba0d857295785bbf } { fields: undefined }

It would help to include the collection name:
mquery findOne +118ms flights { _id: 53ee3e60dffe60d65e6044f7 } { fields: undefined }
mquery findOne +2ms flights { _id: 5416dcecca114e3e3d480301 } { fields: undefined }

And for this to apply to all calls; find, findOne, count, distinct, etc

Will open a PR.

GeoJSON support for nearSphere

see Automattic/mongoose#1482

Error: field selection and slice cannot be used with count - permissions too strict for count?

To begin, I saw your response on the mongoose issue here: Automattic/mongoose#1772

However, I don't understand how using count on a query with limited or sliced fields is nonsensical or invalid in a way that justifies throwing an Error and refusing to continue. Regardless of the fields selected (or how the selected fields are sliced), the count should remain the same.

mongo seems to act how I'd expect:

> db.events.find().count()
454

> db.events.find( {}, { name: true } ).count()
454

> db.events.find( {}, { name: true, presentations: { $slice: 1 } } ).count()
454

My use case where this matters is that I have created an API that takes a Query. I take that query and turn it into a constructor, then use the constructor to build a query for the count and a query for a limited range of data (this is ultimately for a pagination middleware):

var queryConstructor = aQuery.find( req.query.criteria ).toConstructor();

var countQuery = queryConstructor().count();

var dataQuery = queryConstructor().sort( req.query.sort );
dataQuery = dataQuery.limit( req.query.limit );
dataQuery = dataQuery.skip( ( req.query.page - 1 ) * req.query.limit );

Everything is fine until I receive a query that selects fields or slices. I've come up with a few workarounds:

• Accept a separate, nearly identical Query that is used for the count - a bit ugly and repetitive for calls to the API
• Clear out the query's internal _fields value before calling count on the query - seems unnecessary, and it touches internal state directly, but there isn't any API for this currently that I saw
• Create a new query using the query's internal _conditions value - again, with the internal state manipulation

I'm also having trouble finding documentation (mongodb, mongoose, or otherwise) indicating that some of the other operations that are disallowed in permissions.js for the count operation actually need to be disallowed. In particular, select, slice, sort, comment, and hint all seem fine for use with count. I appreciate the effort to keep people from shooting themselves in the foot, but the current permissions seem a little too strict.

> db.events.find( {}, { name: true, presentations: { $slice: 1 } } ).comment('hi').sort({ name:1 }).count()
454

If you can point me to any documentation that justifies these particular cases, I'd be happy to review it and submit a PR if the justification has changed since the current permissions were implemented.

explain customization hooks

things like _fieldsForExec and _optionsForExec can be overridden.

Order of multiple sort fields is not preserved

mquery does not preserve the order of multiple sort fields. Sort fields and sort order are stored in object properties (see https://github.com/aheckmann/mquery/blob/master/lib/mquery.js#L1234). An array should be used instead. In addition it should be possible to pass an array to sort, e.g. sort([{ field1: 1 }, { field2: -1 }]).

MongoDB native driver also accepts an array, see docs (http://mongodb.github.io/node-mongodb-native/2.0/api/Cursor.html#sort) and code (http://mongodb.github.io/node-mongodb-native/2.0/api/lib_cursor.js.html#line512).

comment() cannot be combined with findAndModify

need to confirm. findAndModify is a comment, not a query.

Use native promises

mquery for Titanium Mobile Framework

Having hard-time to bring mquery into Appcelerator Titanium. It has many dependencies. We liked mquery, it perfectly fits our needs.

We have 3 use cases.

1. Use with Nodejs environment, we have queries stored in DB in proprietory format (expressions), we use mquery to generate mongo query (100% find query). No issues with that.
2. Use with Appcelerator Titanium client side native apps, we don't use direct Mongo Connection at mobile apps, we use NeDB, a lightweight Javascript query based Mongo engine for mobile apps. NeDB does respect many mongo find query options. We are working on bringing mquery dependent libraries, having difficulties with their dependencies.
3. Mobile online. Use mquery to generate the find query, convert to JSON, pass to server, get the result through JSON API. No Issues.

If anyone help us or make us to understand to make a bundled mquery package to integrate into mobile apps, that would be great. Also I am hoping that mquery would continue its path with consideration our use cases, abstract the collection from direct mongo reference.