bigplum/lua-resty-mongol

A ngx_lua driver for mongodb -- Deprecated for not updating with the mongodb version

lua-resty-mongol's Issues

sort crash

when sort key is null will be crash

fix it

local sort_f = function(a, b)
            if not a and not b then return false end
            if not a then return true end
            if not b then return false end
            if not a[key] and not b[key] then return false end
            if not a[key] then return true end
            if not b[key] then return false end
            if asc == 1 then
                return a[key] < b[key]
            else
                return a[key] > b[key]
            end
        end

Date support?

How to insert date field? like a new Date('Dec 03, 1924'),

获取表内所有数据时返回空

使用 cursor = col:find(query, returnfields, num_each_query) 进行查询数据库的时候返回空

local cursor = table::find()

这样使用不行么？

Cannot initialize module

When trying to run the module I constantly get the following error

*[error] 21872#0: 2 lua entry thread aborted: runtime error: /etc/nginx/lua/mongol.lua:11: attempt to call method 'new' (a nil value)

The code I have is the following

-- Includes
local p = "/usr/local/Cellar/lua-resty-mongol/devel/sharelua-resty-mongol/lualib/"
local m_package_path = package.path
package.path = string.format("%s?.lua;%s?/init.lua;%s", p, p, m_package_path)

-- Initialize
local mongo = require "resty.mongol"
if not mongo then
    ngx.say("Could not get mongol instance")
end
conn = mongo:new()

If I change the include path I get module not found as expected, so it's definitely including the module, just not working correctly.

Am I missing something obvious?

Thanks
/Marc

how to use MOngodb 3.0 + SCRAM-SHA-1 authenticate mechanism?

When using Monngo db 3.0 with auth enabled, the default authentication enabled, Its SCRAM-SHA-1.
Now when i try to authenticate using db:auth(username, password) method, it fails with teh err
Failed to authenticate admin@gridfs with mechanism MONGODB-CR: AuthenticationFailed: MONGODB-CR credentials missing in the user document

请问这个库带连接池么？

如果不带连接池 是每次都请求后都要close()？

How to get all records from cursor?

Maybe I misunderstood something, but how? Aren't cursor:next and pairs methods, which generates query to db on each record, instead of getting all needed by one request? col:getmore seems to be right method, which will return any number of recs, but my cursor id somehow always is false

Dealing onlyarray in 'to_bson' function may cause data loss

As we know, when we iterate table with pairs in lua, it doesn't guarantee the order as we defines in table.
http://www.lua.org/manual/5.2/manual.html#pdf-next

if onlyarray then
    if t_k == "number" and k >= 0 then
        if k >= high_n then
            high_n = k 
            seen_n [ k ] = v 
        end 
    else
        onlyarray = false
    end 
end

the code above deals array in to_bson in bson.lua, array value will store in seen_n only if k >= high_n, however this condition won't always be true. It means we may lost some value and results in 'null' stores in mongodb.
I think seen_n [ k ] = v should be moved out of the if condition and we can hold all the data.

Massive refactor and things

Hey bigplum,

tl;dr :: I'm refactoring this entire library to look more like lua and less like C. Also adding a rockspec for this and the resty bindings library. Should I make a PR or just create a couple new repos?(EG:: are you still interested in maintaining this)

So I am writing an event based web framework that's quite different from anything in any language for ngx_lua. It's called lusty https://github.com/Olivine-Labs/lusty and consists of many tiny modules. I'm currently writing a RESTful storage adapter for mongo(IE::fire put event, old document gets deleted and new gets inserted into configured collection, multiple adapters allowed for each model type so you can store data in multiple database types easily using predicates) and the interface for this library is wonky and the code is messy and I'm OCD about those things.

Basically I'm looking to rewrite this codebase and break every api it has. I'm also talking to daurnimator about perhaps merging this fork upstream and doing a pull request to his repo. My goals here are to unify the disparate repositories for this project and hopefully get one really nice version out there for everyone using openresty/ngx_lua to enjoy.

Thanks,
DorianGray

请帮忙看看这个问题

使用openresty1.4.7 lua_resty_mongol链接MongoDB Nginx配置如下:

location /mongo {

            content_by_lua '

                    local mongo = require "resty.mongol"
                    conn = mongo:new()
                    conn:set_timeout(1000)
                    local ok,err = conn:connect("192.168.1.100","27017")
                    if not ok then
                            ngx.say("链接失败"..err)
                            return
                    end

                    local db,err = conn:new_db_handle("igewen")
                    if not db then
                            ngx.say(err)
                            return
                    end
                    db:auth("igewen","123456")
                    local col = db:get_col("accounts")
                    local res = col:find_one({salt="0f2f93"})
                    ngx.say(res["email"])
            ';

}

访问出现500错误。
从Nginx debug日志看不出什么，不过Mongo日志出现下面的错误：

Mon Dec 17 21:31:26 [initandlisten] connection accepted from 192.168.1.102:58293 #3 (1 connection now open)
Mon Dec 17 21:31:26 [conn3] run command igewen.$cmd { getnonce: true }
Mon Dec 17 21:31:26 [conn3] command igewen.$cmd command: { getnonce: true } ntoreturn:1 keyUpdates:0 reslen:65 0ms
Mon Dec 17 21:31:26 [conn3] run command igewen.$cmd { authenticate: true, user: "igewen", nonce: "1bb17b627b06af9d", authenticate: true, key: "7a5c1317d90485e9245a89c6138ff655" }
Mon Dec 17 21:31:26 [conn3] authenticate db: igewen { authenticate: true, user: "igewen", nonce: "1bb17b627b06af9d", authenticate: true, key: "7a5c1317d90485e9245a89c6138ff655" }
Mon Dec 17 21:31:26 [conn3] command igewen.$cmd command: { authenticate: true, user: "igewen", nonce: "1bb17b627b06af9d", authenticate: true, key: "7a5c1317d90485e9245a89c6138ff655" } ntoreturn:1 keyUpdates:0 locks(micros) r:170 reslen:84 0ms
Mon Dec 17 21:31:26 [conn3] Socket recv() errno:104 Connection reset by peer 192.168.1.102:58293
Mon Dec 17 21:31:26 [conn3] SocketException: remote: 192.168.1.102:58293 error: 9001 socket exception [1] server [192.168.1.102:58293]
Mon Dec 17 21:31:26 [conn3] end connection 192.168.1.102:58293 (0 connections now open)

我的MongoDB版本是2.2.2。是否是与DB版本不兼容？

谢谢，支持！

How to read from replicaset secondery

Hi, is there a way to read from secondery of my replicaset? or is there a way to send "slaveOk=1" as command?

How to update using $set

I have tried to do this

local n, err = sessions:update({ username = username }, { $set = { sessionid = base32 } }, 1, 0, 1)

But it returned this unexpected symbol near '$'

So how do I do this correctly?

How to sort document before query

I've try to sort document by some filed and then query using skip and limit, since

colmethods:query(query, returnfields, numberToSkip, numberToReturn, options) 

have no sorting parameter or using

colmethods:find(query, returnfields, num_each_query)

which return cursor but cursor method have no skip function

is it possible to do something like this using lua-resty-mongol

insert an array but got nil element

If I insert an array like this

col:insert({ {photos={"1","2"} } })

I'll get an null element like this
{ "_id" : ObjectId("536650e213cb857d4ebfd65c"), "photos" : [ null, "1", "2" ] }

Is this a bug?

How can I use aggregate？

How can I use aggregate？
such as：
db.mycol.aggregate([{$group : {_id : "$by_user", num_tutorial : {$sum : 1}}}])

Incorrect document about find_one

function colmethods:find_one(query, returnfields)
    local id, results, t = self:query(query, returnfields, 0, 1)
    if id == "\0\0\0\0\0\0\0\0" and results[1] then
        return results[1]
    end
    return nil
end

This API returns a document not a single element array when document found.

请问如果想分页查询需要使用哪个api呢

我想实现分页查询的功能
目前使用
cursor = col:find({JOB_NAME="JOB_NAME"},{},10)
可以返回我想要的数据量，但是不知道使用哪个api来区分页码，请提供一个示例程序，多谢。

Do not use ngx.var inside mongol's code

Hey guys!

There is a string: https://github.com/bigplum/lua-resty-mongol/blob/master/lib/resty/mongol/object_id.lua#L56

ngx.var api is disabled inside ssl_certificate_by_lua* derectives. Is it really necessary to use ngx.var? Mb we can use something instead?

support batch read & write chunks in gridfs_file read & write method

use batch operation to decrease number of requests to mongodb

findAndModify

How to implement 'findAndModify' ?

Bson convert $or not correct

in mycode
sel = {};
sel["$or"]={{fuid='51a8e9d639f85f4caa5a789f'},{tuid='51a8e9d639f85f4caa5a789f'}};

and in mongodb log
assertion 13263 $or array must contain objects ns:icredit.transactions query:{ $or: [ null, { fuid: "51a8e9d639f85f4caa5a789f" }, { tuid: "51a8e9d639f85f4caa5a789f" } ] }

I don't know why mongol send null for first element

impossible to create empty table in documents

trying to define empty table:

local tab = {}

and then add it to data:
local data = {}
table.insert(data,{tab = tab})

(then insert it as new document)

will result as empty object (instead of empty array):
{
tab = {
}
}

GridFS "chunkSize" is stored as a BSON float

We got a report that lua-resty-mongol stores "chunkSize" as a BSON float instead of an int. This causes PyMongo to throw an exception trying to read GridFS files stored by lua-resty-mongol, and it will likely cause other MongoDB drivers to throw exceptions as well:

https://jira.mongodb.org/browse/PYTHON-749

The symptom will be that when you query the "files" collection using the Mongo shell, you can find documents where chunkSize is a float type:

db.fs.files.find({chunkSize: {$type: 1}})

... but not int32 type:

db.fs.files.find({chunkSize: {$type: 16}})

If the driver is storing files in GridFS correctly, then chunkSize will be int type: $type: 16.

Is this the right place to file this, or shall I file it on the upstream "mongol" project?

Uncatchable error occured: '/usr/local/openresty/lualib/resty/mongol/colmt.lua:78: assertion failed!'

Uncatchable error occured: '/usr/local/openresty/lualib/resty/mongol/colmt.lua:78: assertion failed!'

It seems to happen randomly so it's hard to reproduce the error.

Any ideas?

Way to do read preference?

Hi, Is there way I can use read preference?

I like to set reading from slave preferred. I am not sure whether current driver has support or any workaround existing..

Query command

I try to test query command

id, results, t = col:query(sel,{_id=1},0,1);

ngx.say(#results);

i got 1

id, results, t = col:query(sel,{_id=1},5,5);

ngx.say(#results);

i got 10. should be 5
What's wrong ?

关于查询分页和排序

1. 使用find的接口没有分页的偏移查询功能，也没有排序功能，sort的是查出来再排序而不是在mongo中先排序再查询
2. 使用query的接口没有排序查询接口先sort再limit
   local id, results, t = col:query(query,returnfields,toskip,limit)
   返回的都是mongo默认的_id排序

Query with $or condition

How to use mongol for query like this
db.inventory.find( { $or: [ { qty: { $gt: 100 } },
{ price: { $lt: 9.95 } } ]
} )

Support ssl when connecting to mongodb?

How to call server side function ?

how to call javascript on server ? like this
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Sarah C."
}
)

sort + limit使用问题

cursor:limit(6)
local sort_res = cursor:sort({startTime=-1}}, 1000)

当最大条目小余limit值，最开始的几条数据不会返回~~这个是怎么回事?

比如最大条目是7，limit为10，则全部返回，limit为5，则最大的两条不会返回

the driver can not use in context ngx.timer

when use the lua-resty-mongol in ngx.timer context, a error occured as follows:resty/mongol/object_id.lua:56: API disabled in the current context, context: ngx.timer,
the reason is the ngx.var.pid can not use in ngx.timer context, and at last I use ngx.worker.pid() instead.

Dot Notation not working

As per this post querying mongo on partial sub array via dot notation doesnt work with this module (at least not that ive been able to get working). If it is available its not documented anywhere?

http://stackoverflow.com/questions/11414300/query-for-partial-object-in-array-mongodb

文档似乎有很多处和实际代码使用不一样的地方

比如文档中说的conn:settimeout(msec), 实际上代码内写的是conn:set_timeout(msec).

文档内说的conn:cmd这个mothod没有，引用会报nil错误.

update md5 in gridfs_file write method

convert objectID from string and back

Hi, I have a problem with working with object ID.

I like to use object ID to be a unique id for a document. And, when I need to serialize that to string so tht I can send as JSON doc, it works with objectID's tostring() function.

Now, when I receive the string, I tried to convert that to object ID and it asserts. It's because object ID was 12 byte, and when the object ID converted to string (hex string %02x), it becomes 24 bytes. Now, when I receive the data, your library check whether the side is 12 bytes so it won't work.

Even though I removed the assert to see whether it will convert back, it does not.

So, there seem to be no way that I can convert to object ID again from the string so that I can run a query based on that.. I also use PyMongo in my python cron job, but, it works fine in that library.

any thought? or any good idea to work around? I tried many different method, but, i just cannot get back to the obj id.

auth 失败

你好，我在安装的openresty，添加mongol的库文件到resty，在连接mongol成功后，鉴权总是失败，检查账号密码没有问题，鉴权这里是不是代码有bug，希望可以改进。

crashed

I got

[error] 67343#0: *24 lua entry thread aborted: runtime error: /usr/local/openresty/lualib/resty/mongol/ll.lua:36: assertion failed!

How can i get an object in array ?

Hi,

I want to get the below object.

{
        "_id" : "hoge",
         "sInfo" : [
                {
                        "sToken" : "testToken",
                        "sTokenExpirationDate" : ISODate("2015-12-15T02:01:06.802Z")
                }
        ]
}

So, i built a query to get sInfo.sToken.
The query is :

local query = {
   sInfo = {
      {
         sToken = "testToken"
      }
   }
}
local r = col:find_one(query)
ngx.log(ngx.ERR, r["_id"])

But, the variable of 'r' is nil.
How can i get 'sInfo.sToken' ?

I use mongodb 3.x，db:auth(user,pwd) can't work.Is this program need upgrade?

local mongo = require "resty.mongol"
local host_info = {
host="192.168.1.2",
port=27017,
username="ttt",
password="ttt"
}
conn = mongo:new()
conn:set_timeout(2000)
ok, err = conn:connect(host_info["host"], host_info["port"])
ngx.say(ok,err)
-- 1 nil
if not ok then
ngx.log("connect failed: "..err)
end
local db = conn:new_db_handle("ttt")
ok, err = db:auth(host_info["username"], host_info["password"])
ngx.say(ok, err)
-- nil lauth failed
col = db:get_col("tt")
r = col:find_one({a=1})
if r == nil then
return false
else
ngx.say(r["_id"])
return true
end

if field value is -1 , and datatype=int32 ,then get value will be 4294967295

in mongodb database, if a record has field value -1 , and its datatype=int32 ,
then use lua-resty-mongol will get value 4294967295 。
I guess it is a overflow problem ,but how to solve it ? thank you very much!

inserting table of tables fails,and $push results to $set data

local versions = {}
local t = {}
table.insert(t,{a = "aa"})
table.insert(t,{b = "bb"})
versions={{name="dog",container="mycontainer",tab= t}}

mongos> db.files.find({container:'mycontainer'}).pretty()
{
    "_id" : ObjectId("4fbb710018b42f60c829ae87"),
    "container" : "mycontainer",
    "tab" : [
        undefined,
        {
            "a" : "aa"
        },
        {
            "b" : "bb"
        }
    ],
    "name" : "dog"
}



Also, pushing new info to "tab" results in resetting "tab" into new array of:
[
    undefined,
    {
        "c" : "cc"
    }
]

naming inconsistencies

Hi there,

There's some inconsistency in the naming on this library.

• Some methods are under_scored, others are camelCased, and others are allwordslowercase. For example, in dbmt.lua, there's listcollections, dropDatabase & add_user. I recommend sticking to a single policy there. Since mongo's interface is camelCased, I recommend camelCase.
• Some words are used in both compressed and uncompressed form. Sometimes it's 'database' and others is 'db'. Sometimes is 'collection' and others is 'col'. On the public interface, these names should be consistent. I recommend using the long version on the public interface.

object stored in ngx.ctx is not reusable

Here is how to reproduce:

in access :

local ok, object = DB.query('collection',search)
ngx.ctx.env.object = object

in content:

local object = ngx.ctx.env.object
local search = {}
search.something       = object._id
search.container    = ngx.ctx.env.containerName
DB.query('collection',search)

error in logs:

[main] Uncatchable error occured: '/usr/local/openresty/lualib/resty/mongol/bson.lua:151: Failure converting function: function: 0x40868ce0', ...

-> I think there the object is not properly kept in ngx.ctx, and then cannot be passed to another
query

Dumping the object used in content:

search contains:
container, and object as object._id

 Logger.debug('--->')
for k,v in pairs(search) do
    if type(v) ~= 'table' then
        Logger.debug( k..' -> '..v)
    else
        for i,j in pairs(v) do
            Logger.warn(i..'->'..type(j))
        end
    end
end
 Logger.debug('<---')

--->
get_inc->function, ....
id->string, ...
get_hostname->function, ...
get_pid->function, ...
tostring->function, client: ...
get_ts->function, client...
container -> 'xxxx' # (this is real data, existing)
<---

nb: the same technic, reloading the object in 'content', works perfectly.

got Can't assign requested address

is there a way to reuse the connection socket? I got this error when I ab it with big amount of connection.

[crit] 1207#0: *10119 connect() to 127.0.0.1:27017 failed (49: Can't assign requested address), client: 127.0.0.1,

openresty 添加 resty.mongol 库报错API disabled in the current context

2018/12/22 14:19:20 [error] 12243#0: init_worker_by_lua_file error: /usr/local/openresty/lualib/resty/mongol/object_id.lua:56: API disabled in the current context
stack traceback:
[C]: in function '__index'
/usr/local/openresty/lualib/resty/mongol/object_id.lua:56: in main chunk
[C]: in function 'require'
/usr/local/openresty/lualib/resty/mongol/bson.lua:25: in main chunk
[C]: in function 'require'
/usr/local/openresty/lualib/resty/mongol/colmt.lua:20: in main chunk
[C]: in function 'require'
/usr/local/openresty/lualib/resty/mongol/dbmt.lua:9: in main chunk
[C]: in function 'require'
/usr/local/openresty/lualib/resty/mongol/init.lua:14: in main chunk
[C]: in function 'require'
/home/proj/gamebin//logic/banksafer.lua:3: in main chunk
[C]: in function 'require'
/home/proj/gamebin//handler/centerhandler.lua:9: in main chunk
[C]: in function 'require'
/home/proj/gamebin//libs/center.lua:11: in main chunk
[C]: in function 'require'
/home/proj/gamebin/libs/worker_init.lua:2: in main chunk

Getting "attempt to call method 'new' (a nil value)" when running the test code.

Running into this error even with the simplest code.

mongol = require "resty.mongol"
conn = mongol:new() 

Get the "attempt to call method 'new' (a nil value)" error on line 2.

Running off of github clone. Basic Lua hello world is working in Openresty. Any clues?

how to create a collection if it does not exist!

Hi,
In MongoDB this command created a collection if it does not exist: db.collection.insert()

I see no way of creating a collection like this in mongol.

Am i missing something here?

Thanks for your help.

Samar

Mongo Date type supported?

It seems there is a issue inserting Date type.
For instance, I have a code to create new user with created_at field.

local mongo = require "resty.mongol"
local bson = require "resty.mongol.bson"

local now = bson.get_utc_date(ngx.time() * 1000)
local data = {
    user = 1,
    created_at = now
}
local db, err = mongo:new(XXXXX)
local user = db:get_col("user")
local res, err = user:insert({data}, nil, true)

Data stored in created_at is actually {"v"=>1469128796000} and the value 1469128796000 is stored as int64 type, not mongo's Date type.
Does anybody has same issue or a suggestion to fix it?

Thanks

count统计方法在3.6版本调用出错了，暂时不知道怎么解决