lseek() return value overflows on large files.

[btimby]

lseek() return value overflows on large files.

[gvvaughan]

Lua 5.1 only has 53 bits to storeintegers, so this is a luaposix issue that affects any wrapped API that accepts or returns a long int that doesn't fit in 53 bits.

The solution is to add support for a userdata that holds a long, or a pid_t etc and adjust all affected functions to use that. I expect this would be a non-backwards compatible change, but only for < Lua 5.3 where a true 64 bit integer type was added to the language.

Would you like to work on a patch?

[btimby]

Makes sense, thanks for the reply. I would be happy to look into providing a patch but I must say that I am fairly new to lua. I do have a background in C/C++ so that will probably help.

IOW, I will have to research your comments above as I 50% understand them :-).

[btimby]

So essentially:

• Conditional compile (ifdef) to enable support when compiling for lua < 5.3.
• Register a new userdata with lua runtime: https://www.lua.org/pil/28.1.html
• Provide helper like checkint(), maybe checklonglong() (or modify checklong()) or similar that utilizes that data type.
• Provide helper like pushresult(), maybe pushresultlong() or pushresultlonglong().
• Modify any luaposix functions such as seek() to use checklonglong() / pushresultlonglong().

[gvvaughan]

That looks about right. Thanks!

I'm not sure whether this all works already with Lua >=5.3 by virtue of 64bit support, so it would be good to add test cases so that github CI will tell us if things work correctly on each Lua version in the test matrix when >53bit integers come back from or are passed into luaposix APIs.

Feel free to post your progress in a branch of your fork if you get stuck, and I'll try to help out.

[btimby]

For context, here is how I fixed my issue.

btimby/nginx-lua-upload-module@0d0c86b

In my case I was trying to determine the size of a file I was writing to. Seeking to the end should (and does) return the file size for smaller files. The stat.st_size member however seems to work for larger files.

[gvvaughan]

Actually, thinking about it a bit more, maybe it would be better for backwards compatibility to always return at least 53bits from eg lseek, just as we do now, but for numbers that are too big to fit on versions of Lua without 64bit support to also return an optional second value which is the 64bit userdata I described above.

That way existing working code will keep working as before (bugs and all) but new code can access the additional bits that are otherwise lost.

There might be more work in accepting either a 53bit number, or a 64bit userdata as parameters where that makes sense.

[btimby]

Another backwards compat option would be to add bigseek() on lua < 5.3.

[gvvaughan]

I think that would be more difficult to use in client code that wants to work on multiple Lua versions than this:

r, big = seek(blah)
if r and big then r = big end

...and we could even add the wrapper boilerplate to init.lua!

[RhodiumToad]

Note that supporting values up to 2^52 is sufficient for essentially all practical purposes - only people doing extremely weird things with very sparse files or very large raw devices would need to handle more than 4096 TB in one file.

And to answer your question above: no, it does not work on 5.3 as is, because the code is using int everywhere relevant and not lua_Integer.

The code for stat.st_size is using macros that use the lua_Integer type and not int, which is why it works (for values representable in a double) when lseek does not.