imodpasteur/lutorpy

Use torch in python for deep learning.

lutorpy's Issues

Support Lua5.2

We received users using lua5.2 instead of luajit with torch, and that will cause errors as #31 and #30 etc.

Lua to Python Method Syntax

Hi,

I am working on writing the lua code from DenseCap (https://github.com/jcjohnson/densecap#evaluation) into Python code (I prefer to work on python). I have one question regarding the syntax in lutorpy:

In the file "run_model.lua" the torch model is loaded:

--Load the model, and cast to the right type
local dtype, use_cudnn = utils.setup_gpus(opt.gpu, opt.use_cudnn)
local checkpoint = torch.load(opt.checkpoint)
local model = checkpoint.model
model:convert(dtype, use_cudnn)

And then some parameters are passed to the model:

model:setTestArgs{ rpn_nms_thresh = opt.rpn_nms_thresh,
  final_nms_thresh = opt.final_nms_thresh,
  num_proposals = opt.num_proposals,
}

How should I change the last command into python code? I've see that python and lua have a similar syntax for lists and tables, but I am not sure about the previous example. Is this line calling the "setTestArgs" method with a list as input argument (sorry if this is too basic, I am new to lua)?

Is the following syntax the corresponding line in python (using lutorpy)?

model._setTestArgs(rpn_nms_thresh=opt.rpn_nms_thresh, 
       final_nms_thresh=opt.final_nms_thresh,
       num_proposals=opt.num_proposals)

Thanks a lot

PS: If anyone is interested I can upload the final code (once it is up and running) as an extra example of lua - python conversion using lutorpy

Detect where torch is intalled

Now the installation script assume torch is installed in the default home folder, but it's not always true.

Problems with Python 2.7x and Mac OSX

I am not sure if I should post this in #25 or in #13.

I get the message below, wheter I install Lutorpy with pip or cloning repo.

Failed building wheel for lutorpy
  Running setup.py clean for lutorpy
Failed to build lutorpy
Installing collected packages: lutorpy
  Running setup.py install for lutorpy ... error
    Complete output from command /.../anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/32/mwq0jhwd3dx7vjqmm8hkljp80000gn/T/pip-build-ZW2IBR/lutorpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/32/mwq0jhwd3dx7vjqmm8hkljp80000gn/T/pip-eFxCfL-record/install-record.txt --single-version-externally-managed --compile:
    ('torch_install:', '/.../torch/install')
    ('os family', 'Darwin')
    building without Cython
    torch install:/.../torch/install
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.5-x86_64-2.7
    creating build/lib.macosx-10.5-x86_64-2.7/lutorpy
    copying lutorpy/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/lutorpy
    copying lutorpy/torch_path.py -> build/lib.macosx-10.5-x86_64-2.7/lutorpy
    copying lutorpy/version.py -> build/lib.macosx-10.5-x86_64-2.7/lutorpy
    running build_ext
    building 'lutorpy._lupa' extension
    creating build/temp.macosx-10.5-x86_64-2.7
    creating build/temp.macosx-10.5-x86_64-2.7/lutorpy
    gcc -fno-strict-aliasing -I/.../anaconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/.../torch/install/include -I/.../torch/install/include/TH -I/.../anaconda2/lib/python2.7/site-packages/numpy/core/include -I/.../anaconda2/include/python2.7 -c lutorpy/_lupa.c -o build/temp.macosx-10.5-x86_64-2.7/lutorpy/_lupa.o -std=c++0x -Wno-unused-function -Wno-unreachable-code -Wno-strict-prototypes
    error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
    error: command 'gcc' failed with exit status 1

I tried also your suggestion to

reinstall torch with Lua 5.2 instead of LuaJIT

but that did not change the result.
I have Xcode installed with Devel tools. I tried several reinstalls of gcc, all to no avail. Always same result.
Any suggestions?

how to define a torch function within python?

I want to use the 'optim' package for the optimisation process. I follow the example here to define the feval() function within python:

require('optim')
for epoch in range(50):
   function feval(params)
      gradParams:zero()
      local outputs = model:forward(batchInputs)
      local loss = criterion:forward(outputs, batchLabels)
      local dloss_doutputs = criterion:backward(outputs, batchLabels)
      model:backward(batchInputs, dloss_doutputs)
      return loss, gradParams
   end
   optim.sgd(feval, params, optimState)
end 

But python claims it is a invalid syntax. I cannot put the function definition in a separate .lua file then import it to python because it nees to access many shared variables such as gradParams, model, criterion,...

Is there a way to overcome it?

run CNN torch model in python

Hi,
i have a CNN model in torch, but I am trying to transfer it to python using lutorpy library this is the original torch model

unpack = table.unpack
require 'nn'
require("component")
require('pl.stringx').import()
local file = require('pl.file')
local List = require('pl.List')
-- Parse nonlinearity
local nltbl = {
['tanh'] = nn.Tanh,
['relu'] = nn.ReLU,
['prelu'] = nn.PReLU,
}
local function create_common_model(seq)
local model = seq
local aadict = file.read(path.join(hashdir, "aa1.lst"))
local aaxsize = #(aadict:splitlines())
model:add(nn.LookupTable(aaxsize, embedsize))
model:add(nn.Dropout(0.1))
return model
end
-- Convolution Models
local function create_conv_base_model(seq)
local model = seq
model:add(nn.UpDim())
local padding = math.floor(kernelsize/2)
if padding > 0 then
local concat = nn.ConcatTable()
for j = 0,poolingsize-1 do
--add zeros to (left,right,top,bottum)
concat:add(nn.SpatialZeroPadding(0,0,padding-j,kernelsize-padding-1+j+poolingsize))
end
model:add(concat)
model:add(nn.JoinTable(1))
end
-- nn.TemporalConvolution(inputFrameSize, outputFrameSize, kernal Widith, [dW]=1)
model:add(nn.TemporalConvolution(embedsize, hiddenunit, kernelsize))
model:add(nonlinearity())
if poolingsize > 1 then
model:add(nn.TemporalMaxPooling(poolingsize))
end
model:add(nn.Dropout(0.3))
-- end
model:add(nn.TemporalZip())
return model
end
local function create_conv_model()
local model = nn.Sequential()
create_common_model(model)
create_conv_base_model(model)
model:add(nn.TemporalConvolution(hiddenunit, nclass, 1))
return model
end

and i transform it to this

`import lutorpy as lua
require("nn")
#require("component")
#require('pl.stringx').import()
#local file = require('pl.file')
#local List = require('pl.List')
def reportAcc(acc,score,bestaccfile):
print('Hyperas:valid accuracy:', acc,'valid loss',score)
if not exists(bestaccfile):
current = float("inf")
else:
with open(bestaccfile) as f:
current = float(f.readline().strip())
if score < current:
with open(bestaccfile,'w') as f:
f.write('%f\n' % score)
f.write('%f\n' % acc)
def model(X_train, Y_train, X_test, Y_test):
W_maxnorm = 3
DROPOUT = {{choice([0.3,0.1])}}
poolingsize = 2
embedsize = 15
kernalsize = 5
hiddenunit = 1024
aaxsize=0
aadict = X_train
print(aadict)
aaxsize += len(aadict)
#model = Sequential()
model = nn.Sequential()
#model = nn.Sequential()
model._add(nn.LookupTable(aaxsize, embedsize))
model._add(nn.Dropout(0.1))
#Convolution Models
model._add(nn.UpDim())
padding=int(kernalsize/2)
if padding>0:
concat = nn.ConcatTable()
for j in poolingsize-1 :
concat._add(nn.SpatialZeroPadding(0,0,padding-j,kernalsize-padding-1+j+poolingsize))
model._add(concat)
model._add(nn.JoinTable(1))
model._add(nn.TemporalConvolution(embedsize, hiddenunit, kernalsize))
model._add(nonlinearity())
if poolingsize > 1 :
model._add(nn.TemporalMaxPooling(poolingsize))
model._add(nn.Dropout(0.3))
model._add(nn.TemporalZip())
#create_common_model(model)
#create_conv_base_model(model)
model._add(nn.TemporalConvolution(hiddenunit, nclass, 1))
myoptimizer = RMSprop(lr={{choice([0.01,0.001,0.0001])}}, rho=0.9, epsilon=1e-06)
mylossfunc = 'binary_crossentropy'
model.compile(loss=mylossfunc, optimizer=myoptimizer,metrics=['accuracy'])
model.fit(X_train, Y_train, batch_size=100, nb_epoch=5,validation_split=0.1)
score, acc = model.evaluate(X_test,Y_test)
model_arch = 'MODEL_ARCH'
bestaccfile = join('TOPDIR',model_arch,model_arch+'_hyperbestacc')
reportAcc(acc,score,bestaccfile)
return {'loss': score, 'status': STATUS_OK,'model':(model.to_json(),myoptimizer,mylossfunc)}

`
can i mix some of torch model and python fit to gether???

Common behaviour ?

I ran following code for testing and it showed a Torch prompt.

import lutorpy as lua
import numpy as np

## use require("MODULE") to import lua modules
require("nn")

## run lua code in python with minimal modification:  replace ":" to "._"
t = torch.DoubleTensor(10,3)
print(t._size()) # the corresponding lua version is t:size()

## or, you can use numpy array
xn = np.random.randn(100)
## convert the numpy array into torch tensor
xt = torch.fromNumpyArray(xn)

## convert torch tensor to numpy array
### Note: the underlying object are sharing the same memory, so the conversion is instant
arr = xt.asNumpyArray()
print(arr.shape)

After i quit, Torch the result showed up, is it normal behaviour or is there anything wrong with my setup

TypeError

Running into this when trying snippets from the Readme:

In [11]: t = torch.DoubleTensor(10, 3)

In [12]: print(t._size)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-ad8044caef9e> in <module>()
----> 1 print(t._size)

lutorpy/_lupa.pyx in lutorpy._lupa._LuaObject.__getattr__ (lutorpy/_lupa.c:11078)()

TypeError: startswith first arg must be bytes or a tuple of bytes, not str

On Ubuntu 14.04, same error with Python 3.5, 3.4, and 2.7

precision problem

I saved a tensor in torch.
When I load it in torch, everything is fine.
but When I load it with lutorpy, I lose some precision.
For example the original tensor contains this number: -1.9356058835983
but when I load the tensor in python I get this: -1.9325
I don't know why!
Here is my code:

`import lutorpy as lua

import numpy as np

inputs=torch.load('inp.t7')

print inputs[1][1][1]
`

cannot import name torch_path

I get the following error message when i try to run import lutorpy as lua -

---> 12 from lutorpy import torch_path
13
14 TorchInstallPath = torch_path.torch_path

ImportError: cannot import name torch_path

using python2, installed using sudo pip.

lutorpy, importing Error:

I've installed torch 7 and then I fire the command
sudo ~/torch/install/bin/luarocks install image
sudo ~/torch/install/bin/luarocks install nn
sudo pip install lutorpy
But now when I am in terminal python:
`aquib@javed:~$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import lutorpy
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/lutorpy/init.py", line 17, in
lualib = ctypes.CDLL(os.path.join(TorchInstallPath, "lib/libluajit.so"), mode=ctypes.RTLD_GLOBAL)
File "/usr/lib/python2.7/ctypes/init.py", line 362, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/aquib/torch/install/lib/libluajit.so: cannot open shared object file: No such file or directory

import lutorpy as lua
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/lutorpy/init.py", line 12, in
from lutorpy import torch_path
ImportError: cannot import name torch_path
`
how can I fix it?

Uninitialize a torch network

I have some code which loads a torch model using lutorpy, every initialization consumes GPU memory around 250-300mb. Is there a way to free this memory?

As I tried to dereference from this initialized model variable and run garbage collector in python, and that didn't help.

Thanks.

Having issue with calling torch autograd

Running into some issues but I am new to lua , torch, and to some extent python.. I have pasted a small test pgm here to show the issue(s) I am having. I have attached a file since code insert does not seem to work. Any help you can give or if there is another forum for lutorpy if this is not real issue.
test.txt

require function not importing "image" or "nn" properly

Thank you for this binding it looks good. However I'm having issues on Ubuntu 15.10 with both the "nn" library and "image". They seem to import without error yet the interpreter doesn't recognize their names, so I can't use any of the functions.

~$ python
Python 2.7.10 (default, Oct 14 2015, 16:09:02) 
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import lutorpy as lua
>>> require("image")
<Lua table at 0x41b509c8>
>>> image.lena()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'image' is not defined

Different values under Python and Torch environment

Hi, I found a weired thing, the Lena image seems to have different values under Torch and Python environment.

In torch I use the following command to load lena()

local im = image.rgb2y( image.lena() )
im[{},{1},{1}]
-- Output: "(1,.,.) = \n 0.1182\n[torch.DoubleTensor of size 1x1x1]\n"

In Python, here is the command with lutorpy:

im = image.rgb2y( image.lena() )
im = im.asNumpyArray()
print im[0,0,0]

Output: 0.117321568627

What happens ? There is around 1% error when reading an image.

ImportError: No module named 'torch_path'

Hi,

I've installed lutorpy using

$ pip install luatorpy

and tried to import it in python. It gives me an import error.

$ python
Python 3.5.2 |Anaconda custom (64-bit)| (default, Jul  2 2016, 17:53:06) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lutorpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/users/sasank/anaconda3/lib/python3.5/site-packages/lutorpy/__init__.py", line 9, in <module>
    import torch_path
ImportError: No module named 'torch_path'
>>> 

Additional info:
Ubuntu 14.04.
torch installed in /home/users/sasank/torch

I also tried installing from git repo as

$ pip install git+https://github.com/imodpasteur/lutorpy.git

But that gives the same error.

Support transparent conversion between numpy array and torch tensor

Would be great if we can use numpy array with torch without converting manually using torch.fromNumpyArray and tensor.asNumpyArray.

Error raised when I import lutorpy

...   
import lutorpy as lua
  File "/usr/local/lib/python2.7/dist-packages/lutorpy-1.3.6-py2.7-linux-x86_64.egg/lutorpy/__init__.py", line 18, in <module>
    THlib = ctypes.CDLL(os.path.join(TorchInstallPath, "lib/libTH.so"), mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /root/torch/install/lib/libTH.so: undefined symbol: spotrs_

Any idea for resolving this ?

NameError: name 'torch' is not defined in python 3.5

Hi there! 😄 So I installed lutorpy python 3.5 and 2.7 using pip3 install --user lutorpy and pip2 install --user lutorpy respectively, under ~/.local/lib/. It appears the version for Python 3.5 is not working properly.

Python 2.7

$ python2
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import lutorpy as lua
>>> t = torch.DoubleTensor(10,3)
>>> print(t._size())
 10
  3
[torch.LongStorage of size 2]

Python 3.5

~$ python3
Python 3.5.2 (default, Jul  5 2016, 12:43:10) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import lutorpy as lua
>>> t = torch.DoubleTensor(10,3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'torch' is not defined
>>> 

..

how to get torch pretrained model's parameters

the python codes:
model = torch.load('pretrained_model.t7')
model = model._float()
print(model) # have SpatialConvolution,SpatialBatchNormalization,SpatialFullConvolution layers
conv_nodes = model._findModules('nn.SpatialConvolution')
for i in range(2):
print conv_nodes[i].weight
print conv_nodes[i].bias

I can get correct model structure through "print(model)" but can't get pretrained model's every layer's parameters like the above described method, only got none through "print conv_nodes[i].weight" , Is the method wrong?

Importing custom class

Hello there,

How would one import a custom torch class through lutorpy? Does this package include paths functionality or would this be done with require?

custom.lua:

local custom = torch.class('custom')

function custom:__init()
    self.a = 'hello'
    self.b = 'world'
end

function custom:getA()
    return self.a
end

py.py:

import os
import lutorpy as lua
lua.LuaRuntime(zero_based_index=True)

path = os.path.join('/path/to/cutom.lua')

# Initialize custom.lua

Appreciated,

T

demo in python code

Hi,
I used "qlua" to demo the torch net in lua, but I want to convert lua into python, how do I demo in python code?
Thanks!!

issuse in loading pre-trained model with torch

Hello, everayone. I'd like to load a pre-trained torch model using lutorpy, but I failed. Your help will be appreciated.
The codes are as follows:

Save the torch model

require 'nn';
mlp = nn.Sequential()
mlp:add(nn.Linear(10, 10))
mlp:add(nn.Tanh())
mlp:add(nn.Linear(10, 10))
mlp:add(nn.Tanh())
mlp:add(nn.Linear(10, 2))
mlp:add(nn.LogSoftMax())
print(mlp)

input = torch.Tensor(1,10):copy(torch.range(1, 10))
print(input)
out = mlp:forward(input)
print(out)

torch.save('test.t7', mlp)

Load the troch model and apply it with lutorpy

import numpy as np
import lutorpy as lua

# load your torch file(for example xx.t7 containing the model and weights)
model = torch.load('test.t7')

arr = np.array(range(10))
x = torch.fromNumpyArray(arr)

yt = model._forward(x)
yn = yt.asNumpyArray()
print yn

Errors

Traceback (most recent call last):
  File "test.lutorpy.py", line 5, in <module>
    model = torch.load('test.t7')
  File "lutorpy/_lupa.pyx", line 562, in lutorpy._lupa._LuaObject.__call__ (lutorpy/_lupa.c:9459)
  File "lutorpy/_lupa.pyx", line 1658, in lutorpy._lupa.call_lua (lutorpy/_lupa.c:25877)
  File "lutorpy/_lupa.pyx", line 1667, in lutorpy._lupa.execute_lua_call (lutorpy/_lupa.c:25987)
  File "lutorpy/_lupa.pyx", line 1620, in lutorpy._lupa.raise_lua_error (lutorpy/_lupa.c:25297)
lutorpy._lupa.LuaError: ...niel/software/torch/install/share/lua/5.1/torch/File.lua:343: unknown Torch class <nn.Sequential>

Add more examples

We received several questions regarding the usage of lutorpy, would be better if we have an example folder contains different type of examples.

NameError: name 'cutorch' is not defined

Hello!
I have some problems with cutorch.

I'm trying to import cutorch through require("cutorch"), but I get the NameError: name 'cutorch' is not defined.
The same problem with optim and sys by the way. But require("nn") works.

Although when I print require "cutorch" in th, everything works perfectly so I think the problem is not in PATH.

I use Python 3.5 and Lua 5.2.4 running on Amazon Linux AMI.

Installing lutorpy on OSX with Python 3.5

I know it's not fully supported, so just wanted to document my experience so far.

In short:

1. I couldn't install with pip
2. I could install from source, but a few things weren't compatible with python3, but patching them still couldn't get the runtime to work
3. I installed lupa separately and forced lutorpy to call the external runtime, but that segfaulted

---

Attempt 1: Install with pip.

pip3 install lutorpy
Collecting lutorpy
  Using cached lutorpy-1.3.3.tar.gz
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/local/lib/python3.5/site-packages (from lutorpy)
Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python3.5/site-packages (from lutorpy)
Building wheels for collected packages: lutorpy
  Running setup.py bdist_wheel for lutorpy ... error
  Complete output from command /usr/local/opt/python3/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/tmpcycbonnppip-wheel- --python-tag cp35:
  torch_install: /Users/ahm/local/torch/install
  os family Darwin
  building without Cython
  torch install:/Users/ahm/local/torch/install
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.10-x86_64-3.5
  creating build/lib.macosx-10.10-x86_64-3.5/lutorpy
  copying lutorpy/__init__.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
  copying lutorpy/torch_path.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
  copying lutorpy/version.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
  running build_ext
  building 'lutorpy._lupa' extension
  creating build/temp.macosx-10.10-x86_64-3.5
  creating build/temp.macosx-10.10-x86_64-3.5/lutorpy
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/Users/ahm/local/torch/install/include -I/Users/ahm/local/torch/install/include/TH -I/usr/local/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c lutorpy/_lupa.c -o build/temp.macosx-10.10-x86_64-3.5/lutorpy/_lupa.o -std=c++0x -Wno-unused-function -Wno-unreachable-code -Wno-strict-prototypes
  error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
  error: command 'clang' failed with exit status 1

  ----------------------------------------
  Failed building wheel for lutorpy
  Running setup.py clean for lutorpy
Failed to build lutorpy
Installing collected packages: lutorpy
  Running setup.py install for lutorpy ... error
    Complete output from command /usr/local/opt/python3/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-2tw3qvot-record/install-record.txt --single-version-externally-managed --compile:
    torch_install: /Users/ahm/local/torch/install
    os family Darwin
    building without Cython
    torch install:/Users/ahm/local/torch/install
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.10-x86_64-3.5
    creating build/lib.macosx-10.10-x86_64-3.5/lutorpy
    copying lutorpy/__init__.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
    copying lutorpy/torch_path.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
    copying lutorpy/version.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
    running build_ext
    building 'lutorpy._lupa' extension
    creating build/temp.macosx-10.10-x86_64-3.5
    creating build/temp.macosx-10.10-x86_64-3.5/lutorpy
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/Users/ahm/local/torch/install/include -I/Users/ahm/local/torch/install/include/TH -I/usr/local/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c lutorpy/_lupa.c -o build/temp.macosx-10.10-x86_64-3.5/lutorpy/_lupa.o -std=c++0x -Wno-unused-function -Wno-unreachable-code -Wno-strict-prototypes
    error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
    error: command 'clang' failed with exit status 1

    ----------------------------------------
Command "/usr/local/opt/python3/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-2tw3qvot-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/

The primary error in question here appears to be error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'. I'm not very familiar with C/C++ and had trouble debugging this; in short, I switched to a new strategy...

---

Attempt 2: Install from git source.

git clone lutorpy repo, then python3 setup.py install

Successfully installs! (Though this may have required some of the steps I did trying to debug attempt 1, which included installing a few things--hard to recall now.)

Open a new tab => import lutorpy gives a few errors related to python3 issues. This I'm able to fix by modifying init.py (in git repo + reinstall or in site-packages directly) with the following edits:

• replace import torch_path with import lutorpy.torch_path
• replace two has_keys calls with an equivalent in expression

However, now I get the following error:

>>> import lutorpy as lua
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/site-packages/lutorpy-1.3.3-py3.5-macosx-10.10-x86_64.egg/lutorpy/__init__.py", line 71, in <module>
    LuaRuntime()
  File "/usr/local/lib/python3.5/site-packages/lutorpy-1.3.3-py3.5-macosx-10.10-x86_64.egg/lutorpy/__init__.py", line 68, in LuaRuntime
    luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
  File "lutorpy/_lupa.pyx", line 231, in lutorpy._lupa.LuaRuntime.__cinit__ (lutorpy/_lupa.c:4642)
lutorpy._lupa.LuaError: Failed to initialise Lua runtime

After seeing a few posts about getting lupa properly set up elsewhere, I switched to that.

---

Attempt 3: Install Lupa first

I was able to install lupa through the following steps:

1. install xcode
2. xcode-select -s '/Applications/Xcode.app/Contents/Developer'
3. brew install macvim --with-lua
4. git clone lupa repo
5. cd lupa && python3 setup-py install

Note that the first two steps were necessary for step 3, and step 3 was necessary to solve the "lua.h missing" error that others have experienced as well.

Lupa is successfully installed, and I can run the test examples in lupa like...

>>> import lupa
>>> from lupa import LuaRuntime
>>> lua = LuaRuntime(unpack_returned_tuples=True)
>>> lua.eval('1+1')

Going back to lutorpy, I try modifying the LuaRuntime function as follows:

import lutorpy

def LuaRuntime(*args, **kwargs):
    global luaRuntime
    if not ('zero_based_index' in kwargs):
        kwargs['zero_based_index']=True
    luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
    return luaRuntime

to

import lupa

def LuaRuntime(*args, **kwargs):
    global luaRuntime
    luaRuntime = lupa.LuaRuntime(*args, **kwargs)
    return luaRuntime

(with the zero_based_index code removed since the external version of lupa does not recognize it)

This actually successfully launches the runtime, but unfortunately I now get segfaults from the python terminal when I try to use the lutorpy commands (such as lua.require('nn')--just calling require('nn') wasn't recognized as a function), despite lupa working fine on its own.

I'm not sure if there's a different way to get the lupa runtime working within lutorpy. I'd be happy to submit a pull request on the python3 compatibility changes--I believe they should work in python2 as well, but I didn't test it.

I'll continue trying a few things tomorrow, but I wanted to document this for now.

misspelling of intereted on readme

This project is great. I starred it, but please check the spelling in the sentence:

Let us know by giving lutorpy project a star, so we know how many people are intereted, thank you.

will be more "professional" :)

forward table to the model

I have alread known how to load pre-trained torch model and apply it using lutropy. It is able to forward a tensor to the model, but I'd like to forward a table with two tensors to the model.

As the model defined in torch with table as input, do forward in torch is
model:forward({x, y})
I wonder how to forward in the similiar way with lutropy.

Thanks in advance.

maybe something useful to add to the install instructions

It's possible I'm just a little thick, but I spent the last 10 minutes scratching my head as to why setup.py was not picking up my TORCH_INSTALL env variable. Turns out I had forgotten that 'sudo' (using Arch, it might be aliased by default in Ubuntu) causes you to lose all of your environment variables and pick up root's.

Installing with 'sudo -E' solves this problem. Maybe worth including a one-liner in README.md, to help folks like me who are too lazy to create a virtualenv :)

Also, I've been searching for exactly this library, so can't wait to try it out!

undefined symbol: PyFPE_jbuf

I successfully installed lutorpy but get such error during import.

Traceback (most recent call last): File "demo1.py", line 16, in <module> import lutorpy File "/root/anaconda2/lib/python2.7/site-packages/lutorpy/__init__.py", line 58, in <module> from lutorpy._lupa import * ImportError: /root/anaconda2/lib/python2.7/site-packages/lutorpy/_lupa.so: undefined symbol: PyFPE_jbuf

no module named builtins

[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lutorpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "lutorpy/__init__.py", line 7, in <module>
    import builtins
ImportError: No module named builtins

Need to run:
sudo pip install future

TypeError: 'NoneType' object is not callable

Hi,
I want to port z = nn.Sampler():forward({mean, log_var}) from torch to the following in python:

sampler = nn.Sampler()
z = sampler._forward(mean, log_var)

and I get the following error:

sampler = nn.Sampler()
TypeError: 'NoneType' object is not callable

How can I solve it?

Using optim module with lutorpy

Hello. I'm trying to replicate the simple optim example given in https://github.com/torch/optim/blob/master/doc/intro.md, but using lutorpy:

require('nn')
require('optim')

global gradParams, model, batch_inputs, batch_labels

nr_inputs = 2
nr_hidden = 20
model = nn.Sequential()
model._add(nn.Linear(nr_inputs, nr_hidden))
model._add(nn.Tanh())
model._add(nn.Linear(nr_hidden, 1))

criterion = nn.MSECriterion()

batch_size = 128
batch_inputs = torch.DoubleTensor(batch_size, nr_inputs)
batch_labels = torch.DoubleTensor(batch_size)

for i in range(0, batch_size):
	input_ = torch.randn(2)
	label = None
	if input_[0] * input_[1] > 0:
		label = -1
	else:
		label = +1
	batch_inputs[i] = input_
	batch_labels[i] = label

params, gradParams = model._getParameters()

for epoch in range(0, 50):

	def feval(params):
		
		global gradParams, model, batch_inputs, batch_labels
		
		gradParams._zero()
		
		outputs = model._forward(batch_inputs)
		loss = criterion._forward(outputs, batch_labels)
		dloss_doutputs = criterion._backward(outputs, batch_labels)
		model._backward(batch_inputs, dloss_doutputs)
		params_2, gradParams = model._getParameters()

		return loss, gradParams
	
	optim.sgd(feval, params)

Now, this returns error "invalid arguments: DoubleTensor number nil" at .../lua/5.1/optim/sgd.lua:84. The reason is actually in line 44 of sgd.lua:

local fx,dfdx = opfunc(x)

This line means that, when feval() is called, its two return values loss and gradParams should go to fx and dfdx, respectively. However, what happens is that both values go to fx packed as userdata and nothing goes to dfdx! Is there anything I can do on the Python side to ensure that, when feval() is called from Lua, it will return the two values separately as expected?

The reason I would like this to work is that the optim module allows using things like momentum and weight decay. Many thanks for your time!

anaconda lutorpy

Can some one help me in installing lutorpy in anaconda framwork. Working on AWS , which has inbuilt anaconda.

install lutorpy without torch

Hi there,

got a quick question. Can I install lutorpy without installing torch?
I am using centos 7, which fails to install torch, it fails in "bash install-deps".
Thank you!

Best

Bing

cuda run time error(2): out of memory

When Im using NN in my program with single had no problem.
While enable 4 processor, this error had generated and luajit will parrallelly run in another core and leads memory full.

Please suggest solution to remove luajit backend process

Not working on OSX

Installation

pip install lutorpy

won't install unless you

export CC=gcc-6 
export CXX=g++-6

This is because

 clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/Users/atcold/torch/install/include -I/Users/atcold/torch/install/include/TH -I/usr/local/lib/python2.7/site-packages/numpy/core/include -I/usr/lo
cal/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.12_1/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lutorpy/_lupa.c -o build/temp.macosx-10.11-x86_64-2.7/lutorpy/_lupa.o -std=c++0x -Wno-unused-func
tion -Wno-unreachable-code -Wno-strict-prototypes
  error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
  error: command 'clang' failed with exit status 1

Importing

After having it installed, it won't import in python, and produces the following error

In [1]: import lutorpy as lua
---------------------------------------------------------------------------
LuaError                                  Traceback (most recent call last)
<ipython-input-1-ab626d710fb4> in <module>()
----> 1 import lutorpy as lua

/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in <module>()
     72     return luaRuntime
     73
---> 74 LuaRuntime()
     75
     76 class global_injector:

/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in LuaRuntime(*args, **kwargs)
     69     if not 'zero_based_index' in kwargs:
     70         kwargs['zero_based_index']=True
---> 71     luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
     72     return luaRuntime
     73

lutorpy/_lupa.pyx in lutorpy._lupa.LuaRuntime.__cinit__ (lutorpy/_lupa.c:4642)()

LuaError: Failed to initialise Lua runtime

Alright, tried to import lupa and I get an ImportError: No module named lupa. So, I guess there are missing dependencies in the pip installation procedure.

pip install lupa

Now lupa imports just fine, but importing lutorpy as lua gives Segmentation fault: 11.
More fun. Different bug while doing the same thing.

In [1]: import lupa

In [2]: import lutorpy
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
Abort trap: 6

And even more variety

In [1]: import lutorpy
---------------------------------------------------------------------------
LuaError                                  Traceback (most recent call last)
<ipython-input-1-db94a7dcb3df> in <module>()
----> 1 import lutorpy

/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in <module>()
     72     return luaRuntime
     73
---> 74 LuaRuntime()
     75
     76 class global_injector:

/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in LuaRuntime(*args, **kwargs)
     69     if not 'zero_based_index' in kwargs:
     70         kwargs['zero_based_index']=True
---> 71     luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
     72     return luaRuntime
     73

lutorpy/_lupa.pyx in lutorpy._lupa.LuaRuntime.__cinit__ (lutorpy/_lupa.c:4642)()

LuaError: Failed to initialise Lua runtime

In [2]: import lutorpy
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-db94a7dcb3df> in <module>()
----> 1 import lutorpy

/usr/local/lib/python2.7/site-packages/lutorpy/__init__.py in <module>()
     10 assert not os.path.exists('./lutorpy/_lupa.pyx'), 'PLEASE DO NOT IMPORT LUTORPY FROM SOURCE CODE FOLDER.'
     11
---> 12 from lutorpy import torch_path
     13
     14 TorchInstallPath = torch_path.__torch_path__

ImportError: cannot import name torch_path

ImportError: No module named _lupa

Hi...
I am getting following error when I am doing import lutorpy
Traceback (most recent call last):
File "", line 1, in
File "lutorpy/init.py", line 56, in
from lutorpy._lupa import *
ImportError: No module named _lupa

How to fix it?Thanks in advances

lua -> python translation failed

lua file works fine

require "torch"
require "nn"
require "nngraph"
require "optim"
package.path = '/root/runtime/SROM/srom_torch_model.lua;' .. package.path
require "srom_torch_model"

model = Torch_model.new('./')
print(model)

python3 file doesn't work

import lutorpy as lua
require("torch")
require("nn")
require("nngraph")
require("optim")
package.path = '/root/runtime/SROM/srom_torch_model.lua;' + package.path
require("srom_torch_model")

model = Torch_model.new("./")
print(model)

Throws the error:
File "test.py", line 9, in
model = Torch_model.new("./")
NameError: name 'Torch_model' is not defined

Any idea about this? The srom_torch_model is a customized lua model. Thanks

core dumpt for MaxPool2d in pytorch

I tried to use lutorypy with pytorch together, but there is coredump for MaxPool2d layer.
Would you like to fix it, please?
1 import lutorpy as lua
2 import torch as pt
3 print 'start test'
4 m = pt.nn.MaxPool2d(3, stride=2)
5 input = pt.autograd.Variable(pt.randn(20, 16, 50, 32))
6 pt_output = m(input)
7 print 'finish test'

---

successfully installed lutorpy but Unable to import in python

Hi,
I have installed using pip and everything went smooth. But when i try to import the package in python it fails. PFB logs
I am using Ubuntu 16.04 and torch luajit.

(py35bkJune) jarvis@jarvis-HP-ENVY-m6-Notebook-PC:PycharmProjects/thesis/beauty-torch-master$ sudo pip install lutorpy
The directory '/home/jarvis/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/jarvis/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lutorpy
Downloading lutorpy-1.3.7.tar.gz (210kB)
100% |████████████████████████████████| 215kB 3.1MB/s
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/lib/python3/dist-packages (from lutorpy)
Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python3.5/dist-packages (from lutorpy)
Installing collected packages: lutorpy
Running setup.py install for lutorpy ... done
Successfully installed lutorpy-1.3.7
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(py35bkJune) jarvis@jarvis-HP-ENVY-m6-Notebook-PC:PycharmProjects/thesis/beauty-torch-master$ python
Python 3.5.2 |Anaconda custom (64-bit)| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.

import lutorpy
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named 'lutorpy'

lutorpy + memoize

Hello

I am using lutorpy and it's great. One issue is that each "require" take a long time (since I run the same script many times). I would like to cache it, i.e., load once and use for all runs.

0. Have you tried to cache with lutorpy?
1. is it possible to speed up "require" somehow?
2. have you worked with loadcaffe (it also takes too long) with lutorpy
3. can lutorpy work with memoize? e.g., https://wiki.python.org/moin/PythonDecoratorLibrary

Thanks!

lua.h not found

I'm following the installation instruction and get this error:
lutorpy/_lupa.c:299:17: fatal error: lua.h: No such file or directory

Was I supposed to update the include, path?

fatal error: lua.h: No such file or directory

when i run sudo python setup.py install

I get the following error:

lutorpy/_lupa.c:299:17: fatal error: lua.h: No such file or directory
#include "lua.h"
^

I do have lua.h and it's in the PATH.

Installation

It took me
sudo TORCH_INSTALL=/home/username/torch/install/ pip install lutorpy
instead of
sudo pip install lutorpy
to install.
I'm guessing, it's not just me?

able to import lutorpy but none of the attributes are availabe

how should I handle the case like this

How should I handle below case?
output_image[{{i},{},{}}]:add(-meanStd.mean[i])

what should be the syntax to be used for [{{i},{},{}}]

Need help creating table using lutoropy [question]

My Requirement is simple. To be able to create a lua table which I can pass to the trainer

criterion = nn.MSECriterion()
self.trainer = nn.StochasticGradient(self.net, criterion)
self.trainer.learningRate = 0.01
self.trainer._train(dataset)

Now the dataset is a lua table, each row containing two lua tensors (input, and output)
How do I create this dataset variable if I'm starting from a X, y with sizes (50,10) and (50,1) respectively.

I'm aware of torch.fromNumpyArray functions. But it's unclear how to create the dataset variable in my case.

Issue with liblapack and openblas

I get the following error on importing lutorpy.
I am using ubuntu 16.04 and python2.7.

$ python -c "import lutorpy"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/users/sasank/anaconda3/envs/py2/lib/python2.7/site-packages/lutorpy/__init__.py", line 15, in <module>
    THlib = ctypes.CDLL(os.path.join(TorchInstallPath, "lib/libTH.so"), mode=ctypes.RTLD_GLOBAL)
  File "/home/users/sasank/anaconda3/envs/py2/lib/python2.7/ctypes/__init__.py", line 362, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/liblapack.so.3: undefined symbol: gotoblas

Weirdly, this error is solved if I uninstall openblas-base.

$ sudo apt-get remove libopenblas-base

But usual torch scripts stop working on uninstall and surprisingly, lutorpy works fine.

If I install openblas-dev,

$ sudo apt-get install libopenblas-dev

torch works fine but lutorpy starts giving above error.