fnuecke/eris

Heavy Duty Persistence for Lua 5.2 and 5.3

eris's Issues

upvalues in global functions to local variables not restored properly

When investigating a bug report of OpenComputers I found that some lua code can behave incorrectly between loads (restoring the world after closing it)

MightyPirates/OpenComputers#2598

The summary of the problem is:

The issue appears to be that local values captured (upvalue) in global functions do not restore the same reference, the local scope gets a new object, and the global function has its own. It's like its own global upvalue. In fact, all global functions share the same upvalue, and all local functions share the same upvalue, those being 2 separate upvalues.

Here is a simple repro

local value = 0
local sync = true

function g_fp()
  value = value + 1
  if not sync then
    print("out of sync")
  end
end

function g2(given_value)
  if given_value ~= value then
    sync = false
  end
end

while true do
  -- read here just to delay/slow the loop for observation
  io.read()
  local before_call = value
  g_fp()
  if before_call < value then
    print("value increased", value)
  else
    g2(value)
  end
end

Please update the project description

From what I see in the release page, you are targeting Lua 5.3, but the short description and the main Readme.md state that you are targeting Lua 5.2. Could you please update this?

Thanks, Mt

some lua unit test failures when using `make mingw` with latest eris release (1.1.0-5.3)

I have been trying to get all the lua unit tests to pass for my project whether I build for linux native or cross-compile with mingw. I have had some problems with the mingw version -- most likely there are some problems in my projects configuration.

However, in trying to pinpoint it I also found that when I do a default build of eris (1.1.0-5.3) I don't pass all of the lua-5.3.0 unit tests either, when I run through wine.

I am testing using the given make file with make mingw and running the executable through wine. My test script looks like this, if you care:

I am using i686-w64-mingw32-gcc.

#!/bin/bash
#rm -rf eris-1.1.0-5.3/
#tar -xzf eris-1.1.0-5.3.tar.gz
# Must override provided makefile to use the mingw compiler!

cd eris-1.1.0-5.3/
make clean
make mingw
cd src
cp lua.exe ../../
cp lua53.dll ../../
cd ../../lua-5.3.0-tests/
wine ./../lua.exe -e "_port=true" all.lua

The specific tests that I fail are:
Some test in strings.lua, apparently related to format %a:

***** FILE 'calls.lua'*****
testing functions and calls
+
+
+
+
.............................................................................................................................................................................................................................................................................................................................................................testing binary chunks
OK
.testing strings and string library
testing 'format %a %A'
Z:\home\chris\eris-5.3-master-tests\lua.exe: strings.lua:231: assertion failed!
stack traceback:
    [C]: in function 'assert'
    strings.lua:231: in main chunk
    [C]: in function 'dofile'
    all.lua:153: in main chunk
    [C]: in ?
.>>> closing state <<<

And I fail some verybig.lua test (but this looks OS-related):

***** FILE 'verybig.lua'*****
testing RK
testing large programs (>64k)
Z:\home\chris\eris-5.3-master-tests\lua.exe: cannot open file '\s8.' (Permission denied)
stack traceback:
    [C]: in function 'io.output'
    ?: in main chunk
    (...tail calls...)
    all.lua:180: in main chunk
    [C]: in ?
.>>> closing state <<<

Anyways I thought maybe it's worth reporting. Do you think I should be worried about any of this?

---

Edit: For what it's worth, I fiddled around with this some more, the '%a' issue seems to be fixed if I disable it in luaconf.h

//#if !defined(LUA_USE_C89) || defined(LUA_USE_WINDOWS)
//#define LUA_USE_AFORMAT
//#endif

It might just be poor support for the feature or something in my version of mingw?
Possibly related SO post

Internals of eris

Thanks for such library. Is there any good articles about internals of eris? I want to learn internals to understand eris. I just want to find any article for quick start (it seems like there is no one).

LuaJIT support

Is there a chance this can work with LuaJIT?

Memory corruption was detected in ERIS

Hi. I learned internals of eris little bit during debugging undump of lua script. As I understood, there is memory corruption which occurs during undumping script (script creates not many objects (like closures etc). This is happened due inproper (I guess) working with marks of gco objects. Here is brief example of corruption:

1. Undumping closure (which leads to creation of new proto)
2. Creation of proto
3. Filling proto
4. Running of garbage collector which removes newly created proto
5. Working with dead pointer to proto (in debug leads to access violation, in release it depends on actual state of memory)

I suppose that some work with garbage collector barriers is needed because following steps fix problem:

1. Stop garbage collector (before undumping/unpresisting)
2. Undump/Unpersist
3. Restart garbage collector (after undumping/unpresisting)

Dynamic options

I'd like to make some of the settings dynamic, these include:

• Whether to pass along the writer + ud / zio when persisting / unpersisting to the special persistence callbacks.
• The name of the key in table and userdata metatables used for special persistence.
• Whether to persist debug information (upvalue and local variable names, line numbers).
• Whether to display the "path" in the object where an error occurred.

Considerations: should all these options also be available when calling the library vom Lua? In particular the first one? How to pass the options along? I think a third parameter that can be a table with these options would be nice, since this would avoid any static variables (which could cause issues in a multi-threaded environment where each thread has its own Lua state). Downside: the permanent value table parameter would become obligatory in that case, since persist(perms, table) and persist(table, options) would be ambiguous otherwise.

Any feedback on this would be welcome.

Linking issue when compiling eris as C++ caused by "typedef int bool"

In eris.c there are some compatibility declarations for MSVC I guess containing

/* Not using stdbool because Visual Studio lives in the past... */
typedef int bool;
#define false 0
#define true 1

Later we get extern declarations so we can see the eris_permXlib functions:

/* Functions in Lua libraries used to access C functions we need to add to the
 * permanents table to fully support yielded coroutines. */
extern void eris_permbaselib(lua_State *L, bool forUnpersist);
extern void eris_permcorolib(lua_State *L, bool forUnpersist);
extern void eris_permloadlib(lua_State *L, bool forUnpersist);
extern void eris_permiolib(lua_State *L, bool forUnpersist);
extern void eris_permstrlib(lua_State *L, bool forUnpersist);

However the definitions of these functions in each of the actual library .c files are like this

void eris_permbaselib(lua_State *L, int forUnpersist) {

In C I guess this works, but modern C++ compilers are basically configured to ignore things like typedef int bool and #define true 1, because although this is common in C headers, bool and true are part of the C++ language, unlike how they are handled in C.

In C++ bool and int are not the same type so the declarations and definitions don't match, which I think causes the linker error. In C they are the same so linking succeeds.

It took me a while to figure out why I could not link when compiling eris as c++, if you would like a patch that changes the bool declarations to int or vice versa I would be happy to provide. It seems to fix the linking for me, and passes eris' internal tests as well as the internal tests of my project.

Eris for Lua 5.3

I have ported Eris to Lua 5.3 (in the Lua5.3 branch).

All tests pass, so things are looking good, but it'll need some more long-term testing to verify that nothing sneakily breaks in unexpected places.

Bug

set kWriteDebugInformation to false;. Then persist and unpersist a lclosure. In the function u_proto the function will return earlier because of the missing debug information.

/* Read debug information if any is present. */
if (!READ_VALUE(uint8_t)) {
return;
}

Like expected there is a proto on top of the stack. So far so good. But when the function call returns to u_closure it pops the proto but also the function from the stack (lua_pop(info->L, 2); /* ... lcl */). With activated debug information u_proto would push 2 protos (lua_pushvalue(info->L, -1); /* ... proto proto */). I guess to make the assertion eris_assert(top + 1 == lua_gettop(info->L)); happy. Solution: push also a second proto on the stack when leaving earlier with deactivated debug information.

/* Read debug information if any is present. */
if (!READ_VALUE(uint8_t)) {
lua_pushvalue(info->L, -1);
return;
}

Problems with conditionally loaded libraries

Widelands contains the following conditional code:
http://bazaar.launchpad.net/~widelands-dev/widelands/trunk/view/7367/src/scripting/scripting.cc#L162

The idea behind this is debug builds of the game contain functionality that is potential unsafe, but useful for debugging. Now I have the issues though that when a debug game is saved and it should be loaded in release mode, I see the following error:

PANIC: unprotected error in call to Lua API (bad permanent value (no value))

I do not think this is a bug per se, more a search for an explanation and a suggested way forward. What can I do to make my savegames agnostic of debug/release?

Crossplatform compatibility

For one, the persisted data should be compatible between 32bit and 64bit platforms. The question here is whether to always store the data in 64bit, always in 32bit or whether to prefix it with some information on the value sizes. The advantage of always storing it in 64bit mode is that there is no information loss for 64bit systems, but data may be truncated when loading it in 32bit systems and will be larger. When storing in 32bit mode we possibly truncate data when saving on 64bit systems, but can be sure the behavior is the same when loading on either. I'm currently tending towards the second, since the only really relevant cases are the stack size and offsets within it, as far as I can tell.

Second, the persisted data should be compatible on platforms with different Endianness. Most relevant data already goes through the WRITE_VALUE macro, but for some things such as the Proto.code array we may have to switch to writing it value-by-value instead of as one block.

What is the purpose of persisting iolib and loadlib?

I am embedding Eris in a stripped-down version of Lua that does not have iolib or loadlib. It appears to work fine.

However I noticed that even in the master Eris branch, when removing eris_permloadlib and eris_permiolib, the testsuite still passes. Could you maybe explain in what case these functions are required?

What's meaning of invalid reference error?

I am trying to use eris to dump/undump global table. But eris occasionally fails during undumping lua closure with error : invalid reference #4210936. this usually means a special persistence callback of a table referenced said table. What's meaning of such error? Maybe there is common pitfalls which signals about such error?

No issue, but seemingly the only way to reach you :)

Thanks for eris! Just fyi, we just merged it as our main Lua persistence code into Widelands: http://bazaar.launchpad.net/~widelands-dev/widelands/trunk/revision/6829

Thanks for all your hard work on this project!

Memory limit

I studied OpenComputers, and I want to know how to limit memory?

Potential crash when persisting threads

There was a chance that the pointer used for iteration over a thread's stack got invalid, if the stack of the thread being persisted got re-allocated as a result of a call inside the loop.

This has no influence on save-state compatibility. It could lead to a crash when persisting, though. This bug had gone unnoticed so far because it happens very rarely.

Literal persistance of userdata

The docs state the a userdata with a metatable which has a key of __persist and value of true:
marks the object for literal persistence. This is the default for tables. Trying to literally persist userdata without this will result in an error. If set, however, the userdata's memory block will be written as-is, and read back as-is.

Yet it seems the code also tries to read the metatable, rather than just the userdata block.

How to navigate invalid reference errors?

Not sure if this is an issue with eris, or more likely an issue with my understanding of how eris works ;). In any case, I'm having some trouble figuring out what causes the "invalid reference" errors. I seem to get them after making seemingly harmless changes, so it's difficult for me to resolve them. After looking at the code, it appears that this error should only happen if eris attempts to load an object that should have already been loaded into a reference table but wasn't.

Here is an example:
Scripts/gui.lua:116: invalid reference #33 (root.windows[1].upvalues[attrs].content_view.upvalues[attrs]._subviews[1].upvalues[attrs].delegate.selectedSliceIndex.upvalues[camera_state].onChange.upvalues[observers][1].upvalues[view])

The last value upvalues[view] is the same view as _subviews[1]. The code is attempting to construct some views which reference a state object to figure out how to render themselves. The state object maintains a list of observers. Each of the views also register themselves as observers of the state object so they can re-render when another view changes the state object. The invalid reference comes and goes as I add and remove views that follow the same pattern.

Two upvalue related bugs

There were two upvalue bugs in the library, both concerning open upvalues, one when persisting, one when unpersisting. Both should be fixed now. This is just for the record, and as a notification to those using the library.

Persisting threads and the global environment

Hello, I have been running into some issues while attempting to persist a thread. Threads share their global environment with their parent, so I had expected that persisting a thread would pass over the globals, and that unpersisting would cause the thread to share the parent's globals at that time. However, persisting a thread does cause it to try to save the globals; I realized this because having any libraries loaded, even coroutine itself so my script could yield, causes Eris to complain about being passed light C functions (which in this case were the library functions. Adding _G to the perms table does not stop this from happening.

I hope this is actually a bug and not some misunderstanding on my part.

For reference, here is the code that produces this issue:

lua_State* L;
lua_State* thread;
const char* script = "yieldprint.lua";
int result;

L = luaL_newstate();
luaL_openlibs(L);
thread = lua_newthread(L);
lua_setglobal(L, "thread"); // Store the thread to get it off the stack and save it from being garbage collected
luaL_loadfile(thread, script); // Push the script as a function onto the thread's stack
result = lua_resume(thread, L, 0); // Start the thread

if (result == LUA_YIELD)
{
lua_newtable(L); // Perms table

// Adding the global table to the perms doesn't make a difference
lua_getglobal(L, "_G");
lua_pushstring(L, "global");
lua_settable(L, -3);

lua_getglobal(L, "thread"); // Thread to persist
eris_persist(L, 1, 2); // Kablammo

}

And here's the script:

local function yieldPrint(str)
print(str)
coroutine.yield()
end

yieldPrint("First")
yieldPrint("Second")
yieldPrint("Third")
yieldPrint("Fourth")
yieldPrint("Fifth")
yieldPrint("Sixth")

Output:

First
PANIC: unprotected error in call to Lua API (attempt to persist a light C function (6CBF4E60))

attempt to persist a light C function?

Thank you for all your hard work on this library. I'm really excited about using it, I can see it saving me TONS of time. However, I'm having some issues understanding how I'm supposed to work around the "attempt to persist a light C function".

I'm not sure I understand what is meant by "light C function" anymore.

For example, why does this fail? I don't see any use of a C function:

a = 3
eris:persist(function() return a end)

while this works:

local a = 3
eris:persist(function() return a end)

It's also worth noting that I expected this to work as well:

eris:perist({[cfunc]: 1}, function() return cfunc end)

But it didn't. This did work as expected though:

eris:perist({[cfunc]: 1}, cfunc)