neopallium/lua-llthreads

Low-Level threads(pthreads or WIN32 threads) for Lua.

About

A simple Lua wrapper for pthreads & WIN32 threads.

Each thread gets it's own lua_State and there is no shared global state. The parent thread can pass data to a child thread only as parameters when creating the child thread. The child threads can return data back to the parent thread only when it return (i.e. ends). The parent needs to call child:join() to get the return values from a child thread, this call will block until the child thread ends.

The design goals of this module is only provide support for creating new lua_State and running them in a different thread. This module will not provide any methods of thread-to-thread data passing between running threads (i.e. no locks, no shared state).

Thread to Thread communication methods

• The recommend method of passing data between threads is to use ZeroMQ.

• Another method is to use sockets library like LuaSocket or Nixio.

Installation

Release 1.2

lua-llthread 1.2 release:

$ sudo luarocks install lua-llthreads

Lastest Git Revision

With LuaRocks 2.0.4.1:

$ sudo luarocks install https://github.com/Neopallium/lua-llthreads/raw/master/rockspecs/lua-llthreads-scm-0.rockspec

With CMake:

$ git clone git://github.com/Neopallium/lua-llthreads.git
$ cd lua-llthreads ; mkdir build ; cd build
$ cmake ..
$ make
$ sudo make install

Example usage

local llthreads = require"llthreads"

local thread_code = [[
	-- print thread's parameter.
	print("CHILD: received params:", ...)
	-- return all thread's parameters back to the parent thread.
	return ...
]]

-- create detached child thread.
local thread = llthreads.new(thread_code, "number:", 1234, "nil:", nil, "bool:", true)
-- start non-joinable detached child thread.
assert(thread:start(true))
-- Use a detatched child thread when you don't care when the child finishes.

-- create child thread.
local thread = llthreads.new(thread_code, "number:", 1234, "nil:", nil, "bool:", true)
-- start joinable child thread.
assert(thread:start())
-- Warning: If you don't call thread:join() on a joinable child thread, it will be called
-- by the garbage collector, which may cause random pauses/freeze of the parent thread.
print("PARENT: child returned: ", thread:join())

local socket = require"socket"
socket.sleep(2) -- give detached thread some time to run.

lua-llthreads's Issues

Calling local function in thread.new

Hello,
I am trying to invoke a function local to the lua script using llthread but it doesn't find in the scope.

I get following error: What am I missing?

Error from thread: [string " client_main(host, port); "]:1: attempt to call global 'client_main' (a nil value)
stack traceback:
[string " client_main(host, port); "]:1: in main chunk

example code:

local function client_main(host, port)
print("host:port")
end

local host = "127.0.0.1"
local port = 10000
local thread_code = [[ client_main(host, port); ]]
local thread = llthreads.new(thread_code, host, port)
thread:start()
thread:join()

Build failing on Win10, LuaJIT, MinGW

Installing through luarocks gives this error:

C:\Users\Alex\Desktop>luarocks install lua-llthreads
Installing https://luarocks.org/lua-llthreads-1.2-1.src.rock...
Using https://luarocks.org/lua-llthreads-1.2-1.src.rock... switching to 'build' mode
mingw32-gcc -O2 -c -o src/pre_generated-llthreads.nobj.o -IC:/My/LuaJIT/Install/Directory// src/pre_generated-llthreads.nobj.c
src/pre_generated-llthreads.nobj.c: In function 'Lua_LLThread__start__meth':
src/pre_generated-llthreads.nobj.c:1136:3: error: unknown type name 'bool'
   bool start_detached_idx2 = lua_toboolean(L,2);
   ^
src/pre_generated-llthreads.nobj.c:1137:3: error: unknown type name 'bool'
   bool res_idx1 = 0;
   ^
src/pre_generated-llthreads.nobj.c:1152:13: error: 'true' undeclared (first use in this function)
  res_idx1 = true;
             ^
src/pre_generated-llthreads.nobj.c:1152:13: note: each undeclared identifier is reported only once for each function it appears in
src/pre_generated-llthreads.nobj.c: In function 'Lua_LLThread__join__meth':
src/pre_generated-llthreads.nobj.c:1161:3: error: unknown type name 'bool'
   bool res_idx1 = 0;
   ^
src/pre_generated-llthreads.nobj.c:1202:14: error: 'false' undeclared (first use in this function)
   res_idx1 = false;
              ^

Error: Build error: Failed compiling object src/pre_generated-llthreads.nobj.o

Installing manually with
luarocks install https://github.com/Neopallium/lua-llthreads/raw/master/rockspecs/lua-llthreads-scm-0.rockspec
works as expected.
Update please!

llthread_join do not set TSTATE_JOINED on windows.

in this example __gc method do not call.

local llthreads = require "llthreads"

CODE=[[
local function gc(fn)
  local p = assert(newproxy())
  assert(debug.setmetatable(p, { __gc = fn }))
  return p
end 

p = gc(function() print("hello from gc") end)
]]

local thread = llthreads.new(CODE)
thread:start()
thread:join()

Add `alive` method to thread object.

On windows it could be done by WaitSingleObject and on pthread i think it could be done by pthread_kill

Add timeout to join method and add method kill

help

how to use camke build xcode for ios?
I do this,like cmke -GXcode ,it is xcode for mac
then i user toolchan iOS.make,but,I failure....

llthread_join should close child lua state

tihs code cause deadlock

CODE = [[
local zmq      = require "zmq"
local zthreads = require "zmq.threads"
local ctx = zthreads.get_parent_ctx()
local skt = ctx:socket(zmq.REQ)
]]

local zthreads = require "zmq.threads"
local ctx = require "zmq".init(1)
local thread = zthreads.runstring(ctx, CODE)
thread:start()
thread:join()
-- __gc method is not called yet and skt still open
-- ctx.term wait this socket and thread object can not be destroyed
ctx:term() 

Possible NULL pointer dereference on Lua_LLThread__delete__meth??

Dear developers:
Our static analysis tool reports an NPD bug here, since this_idx1 may be null. It may be a false positive. Thank yo for your confirmation.

lua-llthreads/src/pre_generated-llthreads.nobj.c

Line 1788 in 8dcf387

if((this_idx1->state & TSTATE_STARTED) == TSTATE_STARTED &&

static int Lua_LLThread__delete__meth(lua_State *L) {
  int this_flags_idx1 = 0;
  Lua_LLThread * this_idx1;
	Lua_LLThread_child *child;

  this_idx1 = obj_type_Lua_LLThread_delete(L,1,&(this_flags_idx1)); //execute obj_udata_luadelete_weak which may return null.
  if(!(this_flags_idx1 & OBJ_UDATA_FLAG_OWN)) { return 0; }
	/* if the thread has been started and has not been detached/joined. */
	if((this_idx1->state & TSTATE_STARTED) == TSTATE_STARTED &&
			(this_idx1->state & (TSTATE_DETACHED|TSTATE_JOINED)) == 0) {
		...;
	}
	llthread_destroy(this_idx1);

  return 0;
}

This method may return NULL.

static FUNC_UNUSED void *obj_udata_luadelete_weak(lua_State *L, int _index, obj_type *type, int *flags) {
	void *obj;
	obj_udata *ud = obj_udata_luacheck_internal(L, _index, &(obj), type, 0);
	if(ud == NULL) return NULL;
	*flags = ud->flags;
	/* null userdata. */
	ud->obj = NULL;
	ud->flags = 0;
	/* clear the metatable in invalidate userdata. */
	lua_pushnil(L);
	lua_setmetatable(L, _index);
	/* get objects weak table. */
	lua_pushlightuserdata(L, obj_udata_weak_ref_key);
	lua_rawget(L, LUA_REGISTRYINDEX); /* weak ref table. */
	/* remove object from weak table. */
	lua_pushlightuserdata(L, obj);
	lua_pushnil(L);
	lua_rawset(L, -3);
	return obj;
}

implement lua wrapper like in zthreads

I find useful this module:
(note: new thread call lua_init from env.)

local llthreads = require"llthreads"

local setmetatable = setmetatable
local tonumber = tonumber
local assert = assert

local thread_mt = {}
thread_mt.__index = thread_mt

function thread_mt:start(detached)
    return self.thread:start(detached)
end

function thread_mt:join(...)
    return self.thread:join(...)
end

function thread_mt:kill()
    return self.thread:kill()
end

local bootstrap_pre = [[
local action, action_arg = ...
local lua_init = os.getenv("lua_init")
if lua_init and #lua_init > 0 then
    if lua_init:sub(1,1) == '@' then
        dofile(lua_init:sub(2))
    else
        assert(loadstring(lua_init))()
    end
end

-- create global 'arg'
arg = { select(3, ...) }
]]

local bootstrap_post = [[
local func
-- load Lua code.
if action == 'runfile' then
    func = assert(loadfile(action_arg))
    -- script name
    arg[0] = action_arg
elseif action == 'runstring' then
    func = assert(loadstring(action_arg))
    -- fake script name
    arg[0] = '=(loadstring)'
end

-- run loaded code.
return func(unpack(arg))
]]

local bootstrap_code = bootstrap_pre..bootstrap_post

local function new_thread(bootstrap_code, action, action_arg, ...)
    local thread = llthreads.new(bootstrap_code, action, action_arg, ...)
    return setmetatable({
        thread = thread,
    }, thread_mt)
end

local M = {}

M.set_bootstrap_prelude = function (code)
    bootstrap_code = bootstrap_pre .. code .. bootstrap_post
end;

M.runfile = function (file, ...)
    return new_thread(bootstrap_code, 'runfile', file, ...)
end;

M.runstring = function (code, ...)
    return new_thread(bootstrap_code, 'runstring', code, ...)
end;

M.runfile_ex = function (prelude, file, ...)
    local bootstrap_code = bootstrap_pre .. prelude .. bootstrap_post
    return new_thread(bootstrap_code, 'runfile', file, ...)
end;

M.runstring_ex = function (prelude, code, ...)
    local bootstrap_code = bootstrap_pre .. prelude .. bootstrap_post
    return new_thread(bootstrap_code, 'runstring', code, ...)
end;

return M

And zthreads implement:

local zmq = require"zmq"
local Threads = require"llthreads.ex"

local zthreads_prelude = [[
local zmq = require"zmq"
local zthreads = require"zmq.threads"
local parent_ctx = arg[1]
if parent_ctx then zthreads.set_parent_ctx(zmq.init_ctx(parent_ctx)) end
arg = { select(2, unpack(arg)) }
]]

module(...)

function runfile(ctx, file, ...)
    -- convert ZMQ_Ctx to lightuserdata.
    if ctx then ctx = ctx:lightuserdata() end
    return Threads.runfile_ex(zthreads_prelude, file, ctx, ...)
end

function runstring(ctx, code, ...)
    -- convert ZMQ_Ctx to lightuserdata.
    if ctx then ctx = ctx:lightuserdata() end

    return Threads.runstring_ex(zthreads_prelude, code, ctx, ...)
end

local parent_ctx = nil
function set_parent_ctx(ctx)
    parent_ctx = ctx
end

function get_parent_ctx(ctx)
    return parent_ctx
end

when pass table to another thread, truely pass referrence or serialized string?

hi,
when pass table to another thread, truely pass referrence or serialized string?

Thank you.

Update rockspec

1. This rockspec work on my windows system.

build = {
    type = "builtin",
    modules = {
        llthreads = {
            sources = {"src/pre_generated-llthreads.nobj.c"},
            libraries = {"pthread"},
        }
    },
    platforms = { windows = { modules = { llthreads = { libraries = {'kernel32'} } } } }
}

1. Release version compatible with Lua 5.2.
2. May be change version to Lua >= 5.1, < 5.3.

i can't require file in thread

Error from thread: [string " local _http = require("socket.http")..."]:1: module 'socket.http' not found:
no field package.preload['socket.http']
no file './socket/http.lua'
no file '/usr/local/share/lua/5.1/socket/http.lua'
no file '/usr/local/share/lua/5.1/socket/http/init.lua'
no file '/usr/local/lib/lua/5.1/socket/http.lua'
no file '/usr/local/lib/lua/5.1/socket/http/init.lua'
no file './socket/http.so'
no file '/usr/local/lib/lua/5.1/socket/http.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file './socket.so'
no file '/usr/local/lib/lua/5.1/socket.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:

i got this error when require("socket.http") in child thread.but it's all right in master thread. how to fix it; thanks

Randomly freezes with infinite loop in child thread

I was writing an application using llthreads and it just seems to randomly freeze at certain points. I found out that llthreads is the cause of this, and removed everything but what is necessary to cause the error:

stuff.lua

function selectat(index, ...)
    local result = select(index, ...)
    return result
end

function string.split(str, delim, skipempty)
    local parts, prev = {}, 1
    for i, j in string.gmatch(str, '()'..delim..'()') do
        local part = str:sub(prev, i-1)
        prev = j
        if not (skipempty and #part == 0) then
            table.insert(parts, part)
        end
    end
    local part = str:sub(prev, -1)
    if not (skipempty and #part == 0) then
        table.insert(parts, part)
    end
    return parts
end

test.lua

local llthreads = require 'llthreads'

function asdf()
    local thread = llthreads.new[[
        while true do end
    ]]
    thread:start()
end

function dfasd()
    require 'stuff'
    print 'success'
end

asdf()
dfasd()

On my comuter, using lua 5.1.5 and llthreads 1.2, it doesn't get to the "print 'success'" line. But it does if I make the thread code return and not work forever.