mpeterv/hererocks

Python script for installing Lua/LuaJIT and LuaRocks into a local directory

hererocks's Issues

Support Bash on Windows

Bash can run on Windows, and comes bundled with Git. However, Hererocks doesn't generated activation scripts for Bash when running on Windows, only cmd and Powershell scripts.

It would be useful to let what activation scripts get generated be configurable by a flag, or even enable Bash by default on Windows.

Option to just download

I'd like it if hererocks just had an option to download (and pre-fill the cache) and skip the build/install step.

lua_objlen with Lua 5.3 using -DLUA_COMPAT_5_1

I'm trying to install lua-cjson in a Lua 5.3 environment with hererocks. While installing lua-cjson it failed with "warning: implicit declaration of function 'lua_objlen'." I found this issue suggesting it could be resolved by compiling Lua with the compatibility flag for 5.1, however, it doesn't seem to work. If someone could point me in the right direction to resolve this I would be grateful.

This is the Dockerfile that I'm building:

FROM python:3.6.5

ENV LUA_VERSION 5.3
ENV LUA_DIR “/lua_install”
ENV PATH “$PATH:$LUA_DIR/bin”

RUN apt-get -y update \
 && apt-get install -y -qq --no-install-recommends unzip \
 && pip install hererocks \
 && hererocks $LUA_DIR -r^ --lua=$LUA_VERSION --compat=all --verbose \
 && luarocks install lua-cjson

Hererocks seems to compile Lua with the -DLUA_COMPAT_5_1 flag:

Building Lua 5.3.4 (compat: all)
Skipping 4 patches, use --patch to apply them
Running gcc -std=gnu99 -O2 -Wall -Wextra -DLUA_USE_POSIX -DLUA_USE_DLOPEN -DLUA_USE_READLINE -DLUA_COMPAT_5_1 -DLUA_COMPAT_5_2 -c -o lapi.o lapi.c
...

However, the installation of lua-cjson still throws this warning and is unusable:

Installing https://luarocks.org/lua-cjson-2.1.0.6-1.src.rock
lua_cjson.c: In function ‘json_append_data’:
lua_cjson.c:743:19: warning: implicit declaration of function ‘lua_objlen’ [-Wimplicit-function-declaration]
             len = lua_objlen(l, -1);
                   ^~~~~~~~~~
gcc -O2 -fPIC -I/lua_install/include -c lua_cjson.c -o lua_cjson.o

The full Docker build log: docker-build-log.txt

LuaRocks download path changed in upstream cause fetching LuaRocks fail

LuaRocks download path changed in upstream cause fetching LuaRocks fail.

Using cl.exe found in PATH.
Fetching LuaJIT 2.1.0-beta3 (target: vs) (cached)
Verifying SHA256 checksum
Building LuaJIT 2.1.0-beta3 (target: vs)
Installing LuaJIT 2.1.0-beta3 (target: vs)
Fetching LuaRocks 2.4.4 from http://luarocks.github.io/luarocks/releases/luarocks-2.4.4-win32.zip
Download failed: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Use of lua/lua

In a month or two the contents/structure of the https://github.com/lua/lua repository will be changing to closer reflect how development is done by the lua team.
I thought I'd let you know, as I saw hererocks uses it (

hererocks/hererocks.py

Line 426 in 055dd7c

default_repo = "https://github.com/lua/lua"

)

Compile CLUA5* with SO lib target

Hello, I'm using hererocks to test the embedded runtime integration in a program, and it's lacking the .so creation. Is there a way to make hererocks generate it? Or would it be possible to patch building of clua so it builds the shared library? As it's building the .so for luajit, it'd be a good idea to have it consistent across flavours.

luarocks don't use --cflags

When Lua 5.3 is built with a specific number type.
The value of --cflags is not added in etc/luarocks/config.lua.
So luarocks can not build a valid native extension (lfs in the following example).

$ hererocks Lua-5.3-longdouble --lua 5.3 --cflags="-DLUA_FLOAT_TYPE=3" --luarocks latest
Fetching Lua from http://www.lua.org/ftp/lua-5.3.4.tar.gz (cached)
Verifying SHA256 checksum
Building Lua 5.3.4 (cflags: -DLUA_FLOAT_TYPE=3)
No patches available for Lua 5.3.4
Installing Lua 5.3.4 (cflags: -DLUA_FLOAT_TYPE=3)
Fetching LuaRocks from http://luarocks.github.io/luarocks/releases/luarocks-2.4.2.tar.gz
Verifying SHA256 checksum
Building LuaRocks 2.4.2
Installing LuaRocks 2.4.2
Done.
$ source Lua-5.3-longdouble/bin/activate
$ luarocks install luafilesystem
Installing https://luarocks.org/luafilesystem-1.6.3-2.src.rock
gcc -O2 -fPIC -I/home/dev/checkout/LUA/hererocks/Lua-5.3-longdouble/include -c src/lfs.c -o src/lfs.o
gcc -shared -o lfs.so -L/home/dev/checkout/LUA/hererocks/Lua-5.3-longdouble/lib src/lfs.o
No existing manifest. Attempting to rebuild...
luafilesystem 1.6.3-2 is now installed in /home/dev/checkout/LUA/hererocks/Lua-5.3-longdouble (license: MIT/X11)

$ lua -l lfs
lua: core and library have incompatible numeric types
stack traceback:
	[C]: in ?
	[C]: in function 'require'
	[C]: in ?

Fails to build luajit-2.1

python hererocks.py .env -j 2.1
Cloning LuaJIT from https://github.com/luajit/luajit @v2.1
error: pathspec 'v2.1' did not match any file(s) known to git.
Error: got exitcode 1 from command git checkout 'v2.1'

hererocks doesn't work after `unset HOME`

$ unset HOME
$ python hererocks.py --lua 5.1 --luarocks 2.2.2
Traceback (most recent call last):
  File "hererocks.py", line 1689, in <module>
    main()
  File "hererocks.py", line 1561, in main
    default=get_default_cache())
  File "hererocks.py", line 85, in get_default_cache
    return os.path.join(os.getenv("HOME"), ".cache", "hererocks")
  File "/usr/lib/python2.7/posixpath.py", line 77, in join
    elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'

The current master is broken for LuaJIT

Something went wrong. When I use hererocks.py from the current master, then nginx build with nginx-lua-module fails https://travis-ci.org/jirutka/ngx-oauth/jobs/94990859.

I’ll try to find which commit is responsible for this and provide more details later.

THIS REPO IS NO LONGER MAINTAINED: go to https://github.com/luarocks/hererocks

This repository is no longer maintained. Sadly @mpeterv passed away last year. Please submit issues and PRs to https://github.com/luarocks/hererocks, and let us know if you would consider helping to maintain the project! Thank you!

Lua installation from Git fails due to overhaul of lua/lua repo

The https://github.com/lua/lua repository was overhauled a while back, and no longer contains a directly buildable copy of lua, but rather shows the repository as it is seen by the Lua team. This results in any attempt to install Lua via Git failing:

jonathan@JONATHAN-HP:~/lua$ hererocks . -l@
Cloning Lua from https://github.com/lua/lua @master
Traceback (most recent call last):
  File "/home/jonathan/.local/bin/hererocks", line 11, in <module>
    load_entry_point('hererocks==0.15.0', 'console_scripts', 'hererocks')()
  File "/home/jonathan/.local/lib/python3.6/site-packages/hererocks.py", line 1892, in main
    if RioLua(opts.lua).update_identifiers(identifiers):
  File "/home/jonathan/.local/lib/python3.6/site-packages/hererocks.py", line 1034, in __init__
    super(RioLua, self).__init__(version)
  File "/home/jonathan/.local/lib/python3.6/site-packages/hererocks.py", line 590, in __init__
    self.major_version = self.major_version_from_source()
  File "/home/jonathan/.local/lib/python3.6/site-packages/hererocks.py", line 606, in major_version_from_source
    with open(os.path.join("src", "lua.h")) as lua_h:
FileNotFoundError: [Errno 2] No such file or directory: 'src/lua.h'

Two possible solutions here: either just disable installing Lua via Git, or add code to assemble the files from the repo into proper build directories. I'm not sure if the latter is possible (as I haven't studied the new repo closely or looked at the hererocks code lately, but it would be nice to continue to have a Git option for installing Lua.

Improve tests

• Port tests to Python (unittest/nose)
• Add test coverage
• Add tests until coverage is OK
• Tests pass on Travis (Linux)
• Tests pass on Travis (OS X)
• Tests pass on Appveyor (Windows)

Option to disable readline on Linux

I'm building with the following command on Linux machine without readline package installed on it:

$ hererocks -l 5.3.1 .
...
gcc -std=gnu99 -O2 -Wall -Wextra -DLUA_COMPAT_5_2 -DLUA_USE_LINUX    -c -o lua.o lua.c
lua.c:80:31: fatal error: readline/readline.h: No such file or directory
compilation terminated.
make[2]: *** [lua.o] Error 1
make[2]: Leaving directory `/tmp/tmp123/lua-5.3.1/src'
make[1]: *** [linux] Error 2
make[1]: Leaving directory `/tmp/tmp123/lua-5.3.1/src'
make: *** [linux] Error 2
Error: got exitcode 2 from command make linux

I can use --target generic to disable this, but then I also disable dynamic module loading, which is non-acceptable. Can you provide an option to patch Lua sources to disable readline, please?

an option to apply patches against known bugs

Could you add an option (probably enabled by default) to apply known patches before building. It takes several months before patches are incorporated into new bugfix release. At the same time, new release (even bugfix) often brings new bugs. Moreover, Lua 5.1 is end-of-life, so some bugs are just not fixed.

Error install lua 5.1.5 use on OS X.

Building Lua 5.1.5
sed: 1: "src/Makefile": unterminated substitute pattern
Error: got exitcode 1 from command sed -i s/MYCFLAGS/SYSCFLAGS/g src/Makefile

see https://travis-ci.org/xpol/linter-luacheck/jobs/99927264

Fallback to mirrors

e.g. if lua.org is down, hererocks will fail.
Instead, fallback to mirrors.

reproduce LUA_PATH and LUA_CPATH order in Lua 5.1 and LuaJIT

hererocks adds its paths to the beginning of LUA_PATH_DEFAULT and LUA_CPATH_DEFAULT.

It is similar to how Lua 5.3 does. I'm not certain about Lua 5.2.

Lua 5.1 and LuaJIT use local path "./?.lua" as first member.

I think, changing LUA_LDIR and LUA_CDIR is more reliable than adding new component to the beginning. Moreover, current behaviour promotes code inside hererocks to load system-wide code, which breaks the isolation and may result in hidden dependencies.

See http://lua.2524044.n2.nabble.com/Virtualenv-Lua-liketool-td7672360.html

Update for luarocks 3.0.0 release

WIndows: configure LuaRocks to use correct CMake generator

By default CMake uses generator for the latest Visual Studio version installed. LuaRocks must be configured to instruct CMake to use appropriate generator. See how it's done in lua-appveyor-example: https://github.com/ignacio/lua-appveyor-example/blob/7f8e5e57289ba7698a5d2b11503cf1d835e504fd/.appveyor/install.bat#L172-L187

Unable to install LuaJIT on windows

Maybe I'm missing something obvious, but whenever i run python -m hererocks lua_install -j ^ -r ^, I get this error:

user@DESKTOP-AGI9U5M MINGW64 ~/Documents/Code
$ python -m hererocks lua_install -j ^ -r ^ --target mingw
Fetching LuaJIT from https://github.com/LuaJIT/LuaJIT/archive/v2.0.4.tar.gz (cached)
Verifying SHA256 checksum
Building LuaJIT 2.0.4
Installing LuaJIT 2.0.4
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python35-32\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Program Files (x86)\Python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\hererocks.py", line 995, in <module>
    main()
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\hererocks.py", line 979, in main
    identifiers_changed = LuaJIT(opts.luajit).update_identifiers(identifiers)
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\hererocks.py", line 334, in update_identifiers
    self.install()
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\hererocks.py", line 470, in install
    self.make_install()
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\hererocks.py", line 774, in make_install
    shutil.copy(arch_file, os.path.join(opts.location, "lib", target_arch_file))
  File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 235, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 114, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: 'lua51.lib'

The contents of lua_install are:

├───bin
│       lua.exe
│       lua51.dll
│
├───include
│       lauxlib.h
│       lua.h
│       lua.hpp
│       luaconf.h
│       luajit.h
│       lualib.h
│
└───lib

Problems with luarock 3.0.2 under Windows

see https://ci.appveyor.com/project/osch/experimental/build/0.3.2.37

luarocks 2.4.4 output:

Configuration files:
         System: c:/hererocks/luarocks/config-5.3.lua (ok)
         User  : C:/Users/appveyor/AppData/Roaming/luarocks/config-5.3.lua (not found)

luarocks 3.0.2 output:

Configuration files:
         System  : C:/Program Files/luarocks/config-5.3.lua (not found)
         User    : C:/Users/appveyor/AppData/Roaming/luarocks/config-5.3.lua (not found)

So somehow the system configuration file cannot be found.

Workaround:

    set LUAROCKS_CONFIG=c:\hererocks\luarocks\config-%LUA%.lua

error message is not printed in Python 2 due to encoding errors

Example: https://travis-ci.org/mpeterv/hererocks/jobs/167906475

Windows: add generic Visual Studio target

Specifying VS version and arch in a target (e.g. vs15_64) is good for testing on Appveyor, but for other uses there should be a generic vs target that doesn't do anything if cl.exe is already in PATH and detects installed VS version and host arch and uses that otherwise. See https://github.com/keplerproject/luarocks/blob/0ddc39af72f608f8d386f5e79a86417da16c4b21/install.bat#L427-L536

Support Windows & Appveyor

Great work! I think, i'll replace bash scripts with hererocks in my projects. (They are based on lua-travis-example.) However I have not only Bash scripts but also Batch scripts to automate exactly the same thing on Windows (and on AppVeyor). Batch scripts are based on lua-appveyor-example. Can I use hererocks in AppVeyor?

Can it install multiple versions of both 32bit and 64bit Lua?

Can it install multiple versions of both 32bit and 64bit Lua?

How to build Lua 5.3 with support `bit32` library

I try this on Travis

1.13s$ pip install --user hererocks
You are using pip version 6.0.8, however version 8.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting hererocks
  Downloading hererocks-0.6.0.tar.gz
Installing collected packages: hererocks
  Running setup.py install for hererocks
    Installing hererocks script to /home/travis/.local/bin
Successfully installed hererocks
before_install.3
6.19s$ hererocks here -r^ --$LUA
Fetching Lua from https://www.lua.org/ftp/lua-5.3.2.tar.gz
Verifying SHA256 checksum
Building Lua 5.3.2
Installing Lua 5.3.2
Fetching LuaRocks from http://keplerproject.github.io/luarocks/releases/luarocks-2.3.0.tar.gz
Verifying SHA256 checksum
Building LuaRocks 2.3.0
Installing LuaRocks 2.3.0
Done.

But since hererocks 6.3 Lua 5.3 does not have builtin bit32 library by default

Add config file

I'd like to be able to set the download directory for future hererocks invocations. ==> I'd like to be able to set the default download location in a config file.
It should probably look in .config/hererocks first, followed by /etc/hererocks if that doesn't exist.

Cannot install using Visual Studio 2017

Output when opening a prompt, running the VS script for variable setup, then calling hererocks

(here) C:\DELETEME>"C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Auxiliary\Build\vcvars64.bat"
**********************************************************************
** Visual Studio 2017 RC Developer Command Prompt v15.0.26206.0
** Copyright (c) 2016 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'

(here) C:\DELETEME>hererocks -l 5.1 lua51
Using cl.exe found in PATH.
Traceback (most recent call last):
  File "C:\DELETEME\here\Scripts\hererocks-script.py", line 9, in <module>
    load_entry_point('hererocks==0.15.0', 'console_scripts', 'hererocks')()
  File "c:\deleteme\here\lib\site-packages\hererocks.py", line 1892, in main
    if RioLua(opts.lua).update_identifiers(identifiers):
  File "c:\deleteme\here\lib\site-packages\hererocks.py", line 571, in update_identifiers
    self.set_identifiers()
  File "c:\deleteme\here\lib\site-packages\hererocks.py", line 1050, in set_identifiers
    super(RioLua, self).set_identifiers()
  File "c:\deleteme\here\lib\site-packages\hererocks.py", line 625, in set_identifiers
    cl_help = get_output("cl")
  File "c:\deleteme\here\lib\site-packages\hererocks.py", line 275, in get_output
    return run(get_output=True, *args)
  File "c:\deleteme\here\lib\site-packages\hererocks.py", line 272, in run
    return capture and output.decode("UTF-8").strip()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc6 in position 44: invalid continuation byte```

There's also no `vs17` option in the `--target` option.

LuaJIT targets: luajit.h header is missing, lua.hpp doesn't work

Normally LuaJIT installs headers luajit.h lua.hpp.

user@host:/tmp/here$ cat include/lua.hpp 
// C++ wrapper for LuaJIT header files.

extern "C" {
#include "lua.h"
#include "lauxlib.h"
#include "lualib.h"
#include "luajit.h"
}

hererocks installs the same lua.hpp, but no luajit.h.

Example of failed build: https://travis-ci.org/npge/lua-npge/jobs/112376649#L1455

Installing LuaJIT sometimes fails with SHA256 mismatch

For instance, when installing LuaJIT 2.0.4 this can happen:

Fetching LuaJIT from https://github.com/LuaJIT/LuaJIT/archive/v2.0.4.tar.gz
Verifying SHA256 checksum
Error: SHA256 checksum mismatch for /home/mpeterv/.cache/hererocks/LuaJIT-2.0.4.tar.gz
Expected: d2abdf16bd3556c41c0aaedad76b6c227ca667be8350111d037a4c54fd43abad
Observed: 324f191dcf771310be0fc6496f7282387be9cf8d38b105566226c6cb079e90eb

Happens with LuaJIT 2.1.0-beta2, too. SHA256 for other downloads seemingly don't mismatch. Observed invalid checksums are the same every time.

Comparing normal and invalid files (at least for LuaJIT 2.0.4) shows that the invalid one is almost correct but has extra 7 bytes prepended and appended to it. Prefix is d0a4f\r\n and suffix is \r\n0\r\n\r\n.

The failures happen on Travis/Appveyor fairly often but I've only managed to reproduce it locally once.

Why is luaX.dll installed into bin instead of lib?

Isn’t that a mistake? hererocks.py:1388

I’m trying to build Lua and Luarocks on Windows under MSYS2 and it’s not very straightforward (and not just b/c I don’t know Windows). 😕 One of the problems is that Luarocks expects Lua library inside lib directory, not in bin.

Problem with Visual Studio and HereRocks

Hello there.

I downloaded MinGW, and tried to install LuaCheck using LuaRocks, but, it doesn't work.
After that I have decided to remove MinGW.

How can I solve the solution?

Luarocks build failure

Not sure if it is actually hererocks related but I guess you will know: https://ci.appveyor.com/project/hishamhm/luarocks/build/3.0.0.807-test/job/id5sy0myosu953n2

Should in some way support travis-CI's caching.

Currently, there's no simple way to not download and build something if it already exists using hererocks without a bash script. I propose having a command line option that makes hererocks do nothing if the selected lua runtime is already installed to the specified folder.

Building lua 5.1 fails to find ncurses on debian stretch

# python /hererocks.py -r^ --lua=5.1 /lua5.1
Fetching Lua 5.1.5 (cached)
Verifying SHA256 checksum
Building Lua 5.1.5
Skipping 1 patch, use --patch to apply it
/usr/bin/ld: cannot find -lncurses
collect2: error: ld returned 1 exit status
Error: got exitcode 1 from command gcc -o luac luac.o print.o liblua51.a -Wl,-E -ldl -lreadline -lhistory -lncurses -lm