imodpasteur / lutorpy Goto Github PK
View Code? Open in Web Editor NEWThis project forked from scoder/lupa
Use torch in python for deep learning.
License: MIT License
This project forked from scoder/lupa
Use torch in python for deep learning.
License: MIT License
Hi,
I am working on writing the lua code from DenseCap (https://github.com/jcjohnson/densecap#evaluation) into Python code (I prefer to work on python). I have one question regarding the syntax in lutorpy:
In the file "run_model.lua" the torch model is loaded:
--Load the model, and cast to the right type
local dtype, use_cudnn = utils.setup_gpus(opt.gpu, opt.use_cudnn)
local checkpoint = torch.load(opt.checkpoint)
local model = checkpoint.model
model:convert(dtype, use_cudnn)
And then some parameters are passed to the model:
model:setTestArgs{ rpn_nms_thresh = opt.rpn_nms_thresh,
final_nms_thresh = opt.final_nms_thresh,
num_proposals = opt.num_proposals,
}
How should I change the last command into python code? I've see that python and lua have a similar syntax for lists and tables, but I am not sure about the previous example. Is this line calling the "setTestArgs" method with a list as input argument (sorry if this is too basic, I am new to lua)?
Is the following syntax the corresponding line in python (using lutorpy)?
model._setTestArgs(rpn_nms_thresh=opt.rpn_nms_thresh,
final_nms_thresh=opt.final_nms_thresh,
num_proposals=opt.num_proposals)
Thanks a lot
PS: If anyone is interested I can upload the final code (once it is up and running) as an extra example of lua - python conversion using lutorpy
Now the installation script assume torch is installed in the default home folder, but it's not always true.
I am not sure if I should post this in #25 or in #13.
I get the message below, wheter I install Lutorpy with pip or cloning repo.
Failed building wheel for lutorpy
Running setup.py clean for lutorpy
Failed to build lutorpy
Installing collected packages: lutorpy
Running setup.py install for lutorpy ... error
Complete output from command /.../anaconda2/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/32/mwq0jhwd3dx7vjqmm8hkljp80000gn/T/pip-build-ZW2IBR/lutorpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/32/mwq0jhwd3dx7vjqmm8hkljp80000gn/T/pip-eFxCfL-record/install-record.txt --single-version-externally-managed --compile:
('torch_install:', '/.../torch/install')
('os family', 'Darwin')
building without Cython
torch install:/.../torch/install
running install
running build
running build_py
creating build
creating build/lib.macosx-10.5-x86_64-2.7
creating build/lib.macosx-10.5-x86_64-2.7/lutorpy
copying lutorpy/__init__.py -> build/lib.macosx-10.5-x86_64-2.7/lutorpy
copying lutorpy/torch_path.py -> build/lib.macosx-10.5-x86_64-2.7/lutorpy
copying lutorpy/version.py -> build/lib.macosx-10.5-x86_64-2.7/lutorpy
running build_ext
building 'lutorpy._lupa' extension
creating build/temp.macosx-10.5-x86_64-2.7
creating build/temp.macosx-10.5-x86_64-2.7/lutorpy
gcc -fno-strict-aliasing -I/.../anaconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/.../torch/install/include -I/.../torch/install/include/TH -I/.../anaconda2/lib/python2.7/site-packages/numpy/core/include -I/.../anaconda2/include/python2.7 -c lutorpy/_lupa.c -o build/temp.macosx-10.5-x86_64-2.7/lutorpy/_lupa.o -std=c++0x -Wno-unused-function -Wno-unreachable-code -Wno-strict-prototypes
error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
error: command 'gcc' failed with exit status 1
I tried also your suggestion to
reinstall torch with Lua 5.2 instead of LuaJIT
but that did not change the result.
I have Xcode installed with Devel tools. I tried several reinstalls of gcc, all to no avail. Always same result.
Any suggestions?
I want to use the 'optim' package for the optimisation process. I follow the example here to define the feval() function within python:
require('optim')
for epoch in range(50):
function feval(params)
gradParams:zero()
local outputs = model:forward(batchInputs)
local loss = criterion:forward(outputs, batchLabels)
local dloss_doutputs = criterion:backward(outputs, batchLabels)
model:backward(batchInputs, dloss_doutputs)
return loss, gradParams
end
optim.sgd(feval, params, optimState)
end
But python claims it is a invalid syntax. I cannot put the function definition in a separate .lua file then import it to python because it nees to access many shared variables such as gradParams, model, criterion,...
Is there a way to overcome it?
Hi,
i have a CNN model in torch, but I am trying to transfer it to python using lutorpy library this is the original torch model
unpack = table.unpack
require 'nn'
require("component")
require('pl.stringx').import()
local file = require('pl.file')
local List = require('pl.List')
-- Parse nonlinearity
local nltbl = {
['tanh'] = nn.Tanh,
['relu'] = nn.ReLU,
['prelu'] = nn.PReLU,
}
local function create_common_model(seq)
local model = seq
local aadict = file.read(path.join(hashdir, "aa1.lst"))
local aaxsize = #(aadict:splitlines())
model:add(nn.LookupTable(aaxsize, embedsize))
model:add(nn.Dropout(0.1))
return model
end
-- Convolution Models
local function create_conv_base_model(seq)
local model = seq
model:add(nn.UpDim())
local padding = math.floor(kernelsize/2)
if padding > 0 then
local concat = nn.ConcatTable()
for j = 0,poolingsize-1 do
--add zeros to (left,right,top,bottum)
concat:add(nn.SpatialZeroPadding(0,0,padding-j,kernelsize-padding-1+j+poolingsize))
end
model:add(concat)
model:add(nn.JoinTable(1))
end
-- nn.TemporalConvolution(inputFrameSize, outputFrameSize, kernal Widith, [dW]=1)
model:add(nn.TemporalConvolution(embedsize, hiddenunit, kernelsize))
model:add(nonlinearity())
if poolingsize > 1 then
model:add(nn.TemporalMaxPooling(poolingsize))
end
model:add(nn.Dropout(0.3))
-- end
model:add(nn.TemporalZip())
return model
end
local function create_conv_model()
local model = nn.Sequential()
create_common_model(model)
create_conv_base_model(model)
model:add(nn.TemporalConvolution(hiddenunit, nclass, 1))
return model
end
and i transform it to this
`import lutorpy as lua
require("nn")
#require("component")
#require('pl.stringx').import()
#local file = require('pl.file')
#local List = require('pl.List')
def reportAcc(acc,score,bestaccfile):
print('Hyperas:valid accuracy:', acc,'valid loss',score)
if not exists(bestaccfile):
current = float("inf")
else:
with open(bestaccfile) as f:
current = float(f.readline().strip())
if score < current:
with open(bestaccfile,'w') as f:
f.write('%f\n' % score)
f.write('%f\n' % acc)
def model(X_train, Y_train, X_test, Y_test):
W_maxnorm = 3
DROPOUT = {{choice([0.3,0.1])}}
poolingsize = 2
embedsize = 15
kernalsize = 5
hiddenunit = 1024
aaxsize=0
aadict = X_train
print(aadict)
aaxsize += len(aadict)
#model = Sequential()
model = nn.Sequential()
#model = nn.Sequential()
model._add(nn.LookupTable(aaxsize, embedsize))
model._add(nn.Dropout(0.1))
#Convolution Models
model._add(nn.UpDim())
padding=int(kernalsize/2)
if padding>0:
concat = nn.ConcatTable()
for j in poolingsize-1 :
concat._add(nn.SpatialZeroPadding(0,0,padding-j,kernalsize-padding-1+j+poolingsize))
model._add(concat)
model._add(nn.JoinTable(1))
model._add(nn.TemporalConvolution(embedsize, hiddenunit, kernalsize))
model._add(nonlinearity())
if poolingsize > 1 :
model._add(nn.TemporalMaxPooling(poolingsize))
model._add(nn.Dropout(0.3))
model._add(nn.TemporalZip())
#create_common_model(model)
#create_conv_base_model(model)
model._add(nn.TemporalConvolution(hiddenunit, nclass, 1))
myoptimizer = RMSprop(lr={{choice([0.01,0.001,0.0001])}}, rho=0.9, epsilon=1e-06)
mylossfunc = 'binary_crossentropy'
model.compile(loss=mylossfunc, optimizer=myoptimizer,metrics=['accuracy'])
model.fit(X_train, Y_train, batch_size=100, nb_epoch=5,validation_split=0.1)
score, acc = model.evaluate(X_test,Y_test)
model_arch = 'MODEL_ARCH'
bestaccfile = join('TOPDIR',model_arch,model_arch+'_hyperbestacc')
reportAcc(acc,score,bestaccfile)
return {'loss': score, 'status': STATUS_OK,'model':(model.to_json(),myoptimizer,mylossfunc)}
`
can i mix some of torch model and python fit to gether???
I ran following code for testing and it showed a Torch prompt.
import lutorpy as lua
import numpy as np
## use require("MODULE") to import lua modules
require("nn")
## run lua code in python with minimal modification: replace ":" to "._"
t = torch.DoubleTensor(10,3)
print(t._size()) # the corresponding lua version is t:size()
## or, you can use numpy array
xn = np.random.randn(100)
## convert the numpy array into torch tensor
xt = torch.fromNumpyArray(xn)
## convert torch tensor to numpy array
### Note: the underlying object are sharing the same memory, so the conversion is instant
arr = xt.asNumpyArray()
print(arr.shape)
After i quit, Torch the result showed up, is it normal behaviour or is there anything wrong with my setup
Running into this when trying snippets from the Readme:
In [11]: t = torch.DoubleTensor(10, 3)
In [12]: print(t._size)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-12-ad8044caef9e> in <module>()
----> 1 print(t._size)
lutorpy/_lupa.pyx in lutorpy._lupa._LuaObject.__getattr__ (lutorpy/_lupa.c:11078)()
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
On Ubuntu 14.04, same error with Python 3.5, 3.4, and 2.7
I saved a tensor in torch.
When I load it in torch, everything is fine.
but When I load it with lutorpy, I lose some precision.
For example the original tensor contains this number: -1.9356058835983
but when I load the tensor in python I get this: -1.9325
I don't know why!
Here is my code:
`import lutorpy as lua
import numpy as np
inputs=torch.load('inp.t7')
print inputs[1][1][1]
`
I get the following error message when i try to run import lutorpy as lua -
---> 12 from lutorpy import torch_path
13
14 TorchInstallPath = torch_path.torch_path
ImportError: cannot import name torch_path
using python2, installed using sudo pip.
I've installed torch 7 and then I fire the command
sudo ~/torch/install/bin/luarocks install image
sudo ~/torch/install/bin/luarocks install nn
sudo pip install lutorpy
But now when I am in terminal python:
`aquib@javed:~$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import lutorpy
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/lutorpy/init.py", line 17, in
lualib = ctypes.CDLL(os.path.join(TorchInstallPath, "lib/libluajit.so"), mode=ctypes.RTLD_GLOBAL)
File "/usr/lib/python2.7/ctypes/init.py", line 362, in init
self._handle = _dlopen(self._name, mode)
OSError: /home/aquib/torch/install/lib/libluajit.so: cannot open shared object file: No such file or directory
import lutorpy as lua
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/lutorpy/init.py", line 12, in
from lutorpy import torch_path
ImportError: cannot import name torch_path
`
how can I fix it?
I have some code which loads a torch model using lutorpy, every initialization consumes GPU memory around 250-300mb. Is there a way to free this memory?
As I tried to dereference from this initialized model variable and run garbage collector in python, and that didn't help.
Thanks.
Running into some issues but I am new to lua , torch, and to some extent python.. I have pasted a small test pgm here to show the issue(s) I am having. I have attached a file since code insert does not seem to work. Any help you can give or if there is another forum for lutorpy if this is not real issue.
test.txt
Thank you for this binding it looks good. However I'm having issues on Ubuntu 15.10 with both the "nn" library and "image". They seem to import without error yet the interpreter doesn't recognize their names, so I can't use any of the functions.
~$ python
Python 2.7.10 (default, Oct 14 2015, 16:09:02)
[GCC 5.2.1 20151010] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lutorpy as lua
>>> require("image")
<Lua table at 0x41b509c8>
>>> image.lena()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'image' is not defined
Hi, I found a weired thing, the Lena image seems to have different values under Torch and Python environment.
In torch I use the following command to load lena()
local im = image.rgb2y( image.lena() )
im[{},{1},{1}]
-- Output: "(1,.,.) = \n 0.1182\n[torch.DoubleTensor of size 1x1x1]\n"
In Python, here is the command with lutorpy:
im = image.rgb2y( image.lena() )
im = im.asNumpyArray()
print im[0,0,0]
What happens ? There is around 1% error when reading an image.
Hi,
I've installed lutorpy using
$ pip install luatorpy
and tried to import it in python. It gives me an import error.
$ python
Python 3.5.2 |Anaconda custom (64-bit)| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lutorpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/users/sasank/anaconda3/lib/python3.5/site-packages/lutorpy/__init__.py", line 9, in <module>
import torch_path
ImportError: No module named 'torch_path'
>>>
Additional info:
Ubuntu 14.04.
torch installed in /home/users/sasank/torch
I also tried installing from git repo as
$ pip install git+https://github.com/imodpasteur/lutorpy.git
But that gives the same error.
Would be great if we can use numpy array with torch without converting manually using torch.fromNumpyArray
and tensor.asNumpyArray
.
...
import lutorpy as lua
File "/usr/local/lib/python2.7/dist-packages/lutorpy-1.3.6-py2.7-linux-x86_64.egg/lutorpy/__init__.py", line 18, in <module>
THlib = ctypes.CDLL(os.path.join(TorchInstallPath, "lib/libTH.so"), mode=ctypes.RTLD_GLOBAL)
File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /root/torch/install/lib/libTH.so: undefined symbol: spotrs_
Any idea for resolving this ?
Hi there! 😄 So I installed lutorpy python 3.5 and 2.7 using pip3 install --user lutorpy
and pip2 install --user lutorpy
respectively, under ~/.local/lib/
. It appears the version for Python 3.5 is not working properly.
$ python2
Python 2.7.12 (default, Jul 1 2016, 15:12:24)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lutorpy as lua
>>> t = torch.DoubleTensor(10,3)
>>> print(t._size())
10
3
[torch.LongStorage of size 2]
~$ python3
Python 3.5.2 (default, Jul 5 2016, 12:43:10)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lutorpy as lua
>>> t = torch.DoubleTensor(10,3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'torch' is not defined
>>>
the python codes:
model = torch.load('pretrained_model.t7')
model = model._float()
print(model) # have SpatialConvolution,SpatialBatchNormalization,SpatialFullConvolution layers
conv_nodes = model._findModules('nn.SpatialConvolution')
for i in range(2):
print conv_nodes[i].weight
print conv_nodes[i].bias
I can get correct model structure through "print(model)" but can't get pretrained model's every layer's parameters like the above described method, only got none through "print conv_nodes[i].weight" , Is the method wrong?
Hello there,
How would one import a custom torch class through lutorpy? Does this package include paths functionality or would this be done with require?
custom.lua:
local custom = torch.class('custom')
function custom:__init()
self.a = 'hello'
self.b = 'world'
end
function custom:getA()
return self.a
end
py.py:
import os
import lutorpy as lua
lua.LuaRuntime(zero_based_index=True)
path = os.path.join('/path/to/cutom.lua')
# Initialize custom.lua
Appreciated,
T
Hi,
I used "qlua" to demo the torch net in lua, but I want to convert lua into python, how do I demo in python code?
Thanks!!
Hello, everayone. I'd like to load a pre-trained torch model using lutorpy, but I failed. Your help will be appreciated.
The codes are as follows:
Save the torch model
require 'nn';
mlp = nn.Sequential()
mlp:add(nn.Linear(10, 10))
mlp:add(nn.Tanh())
mlp:add(nn.Linear(10, 10))
mlp:add(nn.Tanh())
mlp:add(nn.Linear(10, 2))
mlp:add(nn.LogSoftMax())
print(mlp)
input = torch.Tensor(1,10):copy(torch.range(1, 10))
print(input)
out = mlp:forward(input)
print(out)
torch.save('test.t7', mlp)
Load the troch model and apply it with lutorpy
import numpy as np
import lutorpy as lua
# load your torch file(for example xx.t7 containing the model and weights)
model = torch.load('test.t7')
arr = np.array(range(10))
x = torch.fromNumpyArray(arr)
yt = model._forward(x)
yn = yt.asNumpyArray()
print yn
Errors
Traceback (most recent call last):
File "test.lutorpy.py", line 5, in <module>
model = torch.load('test.t7')
File "lutorpy/_lupa.pyx", line 562, in lutorpy._lupa._LuaObject.__call__ (lutorpy/_lupa.c:9459)
File "lutorpy/_lupa.pyx", line 1658, in lutorpy._lupa.call_lua (lutorpy/_lupa.c:25877)
File "lutorpy/_lupa.pyx", line 1667, in lutorpy._lupa.execute_lua_call (lutorpy/_lupa.c:25987)
File "lutorpy/_lupa.pyx", line 1620, in lutorpy._lupa.raise_lua_error (lutorpy/_lupa.c:25297)
lutorpy._lupa.LuaError: ...niel/software/torch/install/share/lua/5.1/torch/File.lua:343: unknown Torch class <nn.Sequential>
We received several questions regarding the usage of lutorpy, would be better if we have an example folder contains different type of examples.
Hello!
I have some problems with cutorch
.
I'm trying to import cutorch
through require("cutorch")
, but I get the NameError: name 'cutorch' is not defined
.
The same problem with optim
and sys
by the way. But require("nn")
works.
Although when I print require "cutorch"
in th, everything works perfectly so I think the problem is not in PATH.
I use Python 3.5 and Lua 5.2.4 running on Amazon Linux AMI.
I know it's not fully supported, so just wanted to document my experience so far.
In short:
Attempt 1: Install with pip.
pip3 install lutorpy
Collecting lutorpy
Using cached lutorpy-1.3.3.tar.gz
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/local/lib/python3.5/site-packages (from lutorpy)
Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python3.5/site-packages (from lutorpy)
Building wheels for collected packages: lutorpy
Running setup.py bdist_wheel for lutorpy ... error
Complete output from command /usr/local/opt/python3/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/tmpcycbonnppip-wheel- --python-tag cp35:
torch_install: /Users/ahm/local/torch/install
os family Darwin
building without Cython
torch install:/Users/ahm/local/torch/install
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.10-x86_64-3.5
creating build/lib.macosx-10.10-x86_64-3.5/lutorpy
copying lutorpy/__init__.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
copying lutorpy/torch_path.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
copying lutorpy/version.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
running build_ext
building 'lutorpy._lupa' extension
creating build/temp.macosx-10.10-x86_64-3.5
creating build/temp.macosx-10.10-x86_64-3.5/lutorpy
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/Users/ahm/local/torch/install/include -I/Users/ahm/local/torch/install/include/TH -I/usr/local/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c lutorpy/_lupa.c -o build/temp.macosx-10.10-x86_64-3.5/lutorpy/_lupa.o -std=c++0x -Wno-unused-function -Wno-unreachable-code -Wno-strict-prototypes
error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
error: command 'clang' failed with exit status 1
----------------------------------------
Failed building wheel for lutorpy
Running setup.py clean for lutorpy
Failed to build lutorpy
Installing collected packages: lutorpy
Running setup.py install for lutorpy ... error
Complete output from command /usr/local/opt/python3/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-2tw3qvot-record/install-record.txt --single-version-externally-managed --compile:
torch_install: /Users/ahm/local/torch/install
os family Darwin
building without Cython
torch install:/Users/ahm/local/torch/install
running install
running build
running build_py
creating build
creating build/lib.macosx-10.10-x86_64-3.5
creating build/lib.macosx-10.10-x86_64-3.5/lutorpy
copying lutorpy/__init__.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
copying lutorpy/torch_path.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
copying lutorpy/version.py -> build/lib.macosx-10.10-x86_64-3.5/lutorpy
running build_ext
building 'lutorpy._lupa' extension
creating build/temp.macosx-10.10-x86_64-3.5
creating build/temp.macosx-10.10-x86_64-3.5/lutorpy
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/Users/ahm/local/torch/install/include -I/Users/ahm/local/torch/install/include/TH -I/usr/local/lib/python3.5/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c lutorpy/_lupa.c -o build/temp.macosx-10.10-x86_64-3.5/lutorpy/_lupa.o -std=c++0x -Wno-unused-function -Wno-unreachable-code -Wno-strict-prototypes
error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
error: command 'clang' failed with exit status 1
----------------------------------------
Command "/usr/local/opt/python3/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-2tw3qvot-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/t7/v4xk5xqj0s7fjfd9x1r92lk58zclp8/T/pip-build-0yyrutkq/lutorpy/
The primary error in question here appears to be error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
. I'm not very familiar with C/C++ and had trouble debugging this; in short, I switched to a new strategy...
Attempt 2: Install from git source.
git clone
lutorpy repo, then python3 setup.py install
Successfully installs! (Though this may have required some of the steps I did trying to debug attempt 1, which included installing a few things--hard to recall now.)
Open a new tab => import lutorpy gives a few errors related to python3 issues. This I'm able to fix by modifying init.py (in git repo + reinstall or in site-packages directly) with the following edits:
import torch_path
with import lutorpy.torch_path
has_keys
calls with an equivalent in
expressionHowever, now I get the following error:
>>> import lutorpy as lua
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.5/site-packages/lutorpy-1.3.3-py3.5-macosx-10.10-x86_64.egg/lutorpy/__init__.py", line 71, in <module>
LuaRuntime()
File "/usr/local/lib/python3.5/site-packages/lutorpy-1.3.3-py3.5-macosx-10.10-x86_64.egg/lutorpy/__init__.py", line 68, in LuaRuntime
luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
File "lutorpy/_lupa.pyx", line 231, in lutorpy._lupa.LuaRuntime.__cinit__ (lutorpy/_lupa.c:4642)
lutorpy._lupa.LuaError: Failed to initialise Lua runtime
After seeing a few posts about getting lupa properly set up elsewhere, I switched to that.
Attempt 3: Install Lupa first
I was able to install lupa through the following steps:
xcode-select -s '/Applications/Xcode.app/Contents/Developer'
brew install macvim --with-lua
git clone
lupa repoNote that the first two steps were necessary for step 3, and step 3 was necessary to solve the "lua.h missing" error that others have experienced as well.
Lupa is successfully installed, and I can run the test examples in lupa like...
>>> import lupa
>>> from lupa import LuaRuntime
>>> lua = LuaRuntime(unpack_returned_tuples=True)
>>> lua.eval('1+1')
Going back to lutorpy, I try modifying the LuaRuntime function as follows:
import lutorpy
def LuaRuntime(*args, **kwargs):
global luaRuntime
if not ('zero_based_index' in kwargs):
kwargs['zero_based_index']=True
luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
return luaRuntime
to
import lupa
def LuaRuntime(*args, **kwargs):
global luaRuntime
luaRuntime = lupa.LuaRuntime(*args, **kwargs)
return luaRuntime
(with the zero_based_index code removed since the external version of lupa does not recognize it)
This actually successfully launches the runtime, but unfortunately I now get segfaults from the python terminal when I try to use the lutorpy commands (such as lua.require('nn')--just calling require('nn') wasn't recognized as a function), despite lupa working fine on its own.
I'm not sure if there's a different way to get the lupa runtime working within lutorpy. I'd be happy to submit a pull request on the python3 compatibility changes--I believe they should work in python2 as well, but I didn't test it.
I'll continue trying a few things tomorrow, but I wanted to document this for now.
This project is great. I starred it, but please check the spelling in the sentence:
Let us know by giving lutorpy project a star, so we know how many people are intereted, thank you.
will be more "professional" :)
I have alread known how to load pre-trained torch model and apply it using lutropy. It is able to forward a tensor to the model, but I'd like to forward a table with two tensors to the model.
As the model defined in torch with table as input, do forward in torch is
model:forward({x, y})
I wonder how to forward in the similiar way with lutropy.
Thanks in advance.
It's possible I'm just a little thick, but I spent the last 10 minutes scratching my head as to why setup.py was not picking up my TORCH_INSTALL env variable. Turns out I had forgotten that 'sudo' (using Arch, it might be aliased by default in Ubuntu) causes you to lose all of your environment variables and pick up root's.
Installing with 'sudo -E' solves this problem. Maybe worth including a one-liner in README.md, to help folks like me who are too lazy to create a virtualenv :)
Also, I've been searching for exactly this library, so can't wait to try it out!
I successfully installed lutorpy but get such error during import.
Traceback (most recent call last): File "demo1.py", line 16, in <module> import lutorpy File "/root/anaconda2/lib/python2.7/site-packages/lutorpy/__init__.py", line 58, in <module> from lutorpy._lupa import * ImportError: /root/anaconda2/lib/python2.7/site-packages/lutorpy/_lupa.so: undefined symbol: PyFPE_jbuf
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import lutorpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "lutorpy/__init__.py", line 7, in <module>
import builtins
ImportError: No module named builtins
Need to run:
sudo pip install future
Hi,
I want to port z = nn.Sampler():forward({mean, log_var})
from torch to the following in python:
sampler = nn.Sampler()
z = sampler._forward(mean, log_var)
and I get the following error:
sampler = nn.Sampler()
TypeError: 'NoneType' object is not callable
How can I solve it?
Hello. I'm trying to replicate the simple optim example given in https://github.com/torch/optim/blob/master/doc/intro.md, but using lutorpy:
require('nn')
require('optim')
global gradParams, model, batch_inputs, batch_labels
nr_inputs = 2
nr_hidden = 20
model = nn.Sequential()
model._add(nn.Linear(nr_inputs, nr_hidden))
model._add(nn.Tanh())
model._add(nn.Linear(nr_hidden, 1))
criterion = nn.MSECriterion()
batch_size = 128
batch_inputs = torch.DoubleTensor(batch_size, nr_inputs)
batch_labels = torch.DoubleTensor(batch_size)
for i in range(0, batch_size):
input_ = torch.randn(2)
label = None
if input_[0] * input_[1] > 0:
label = -1
else:
label = +1
batch_inputs[i] = input_
batch_labels[i] = label
params, gradParams = model._getParameters()
for epoch in range(0, 50):
def feval(params):
global gradParams, model, batch_inputs, batch_labels
gradParams._zero()
outputs = model._forward(batch_inputs)
loss = criterion._forward(outputs, batch_labels)
dloss_doutputs = criterion._backward(outputs, batch_labels)
model._backward(batch_inputs, dloss_doutputs)
params_2, gradParams = model._getParameters()
return loss, gradParams
optim.sgd(feval, params)
Now, this returns error "invalid arguments: DoubleTensor number nil" at .../lua/5.1/optim/sgd.lua:84
. The reason is actually in line 44 of sgd.lua
:
local fx,dfdx = opfunc(x)
This line means that, when feval()
is called, its two return values loss
and gradParams
should go to fx
and dfdx
, respectively. However, what happens is that both values go to fx
packed as userdata and nothing goes to dfdx
! Is there anything I can do on the Python side to ensure that, when feval()
is called from Lua, it will return the two values separately as expected?
The reason I would like this to work is that the optim module allows using things like momentum and weight decay. Many thanks for your time!
Can some one help me in installing lutorpy in anaconda framwork. Working on AWS , which has inbuilt anaconda.
Hi there,
got a quick question. Can I install lutorpy without installing torch?
I am using centos 7, which fails to install torch, it fails in "bash install-deps".
Thank you!
Best
Bing
When Im using NN in my program with single had no problem.
While enable 4 processor, this error had generated and luajit will parrallelly run in another core and leads memory full.
Please suggest solution to remove luajit backend process
pip install lutorpy
won't install unless you
export CC=gcc-6
export CXX=g++-6
This is because
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DLUA_COMPAT_ALL -I/Users/atcold/torch/install/include -I/Users/atcold/torch/install/include/TH -I/usr/local/lib/python2.7/site-packages/numpy/core/include -I/usr/lo
cal/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.12_1/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c lutorpy/_lupa.c -o build/temp.macosx-10.11-x86_64-2.7/lutorpy/_lupa.o -std=c++0x -Wno-unused-func
tion -Wno-unreachable-code -Wno-strict-prototypes
error: invalid argument '-std=c++0x' not allowed with 'C/ObjC'
error: command 'clang' failed with exit status 1
After having it installed, it won't import in python
, and produces the following error
In [1]: import lutorpy as lua
---------------------------------------------------------------------------
LuaError Traceback (most recent call last)
<ipython-input-1-ab626d710fb4> in <module>()
----> 1 import lutorpy as lua
/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in <module>()
72 return luaRuntime
73
---> 74 LuaRuntime()
75
76 class global_injector:
/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in LuaRuntime(*args, **kwargs)
69 if not 'zero_based_index' in kwargs:
70 kwargs['zero_based_index']=True
---> 71 luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
72 return luaRuntime
73
lutorpy/_lupa.pyx in lutorpy._lupa.LuaRuntime.__cinit__ (lutorpy/_lupa.c:4642)()
LuaError: Failed to initialise Lua runtime
Alright, tried to import lupa
and I get an ImportError: No module named lupa
. So, I guess there are missing dependencies in the pip
installation procedure.
pip install lupa
Now lupa
import
s just fine, but import
ing lutorpy as lua
gives Segmentation fault: 11
.
More fun. Different bug while doing the same thing.
In [1]: import lupa
In [2]: import lutorpy
PANIC: unprotected error in call to Lua API (attempt to index a nil value)
Abort trap: 6
And even more variety
In [1]: import lutorpy
---------------------------------------------------------------------------
LuaError Traceback (most recent call last)
<ipython-input-1-db94a7dcb3df> in <module>()
----> 1 import lutorpy
/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in <module>()
72 return luaRuntime
73
---> 74 LuaRuntime()
75
76 class global_injector:
/usr/local/lib/python2.7/site-packages/lutorpy/__init__.pyc in LuaRuntime(*args, **kwargs)
69 if not 'zero_based_index' in kwargs:
70 kwargs['zero_based_index']=True
---> 71 luaRuntime = lutorpy._lupa.LuaRuntime(*args, **kwargs)
72 return luaRuntime
73
lutorpy/_lupa.pyx in lutorpy._lupa.LuaRuntime.__cinit__ (lutorpy/_lupa.c:4642)()
LuaError: Failed to initialise Lua runtime
In [2]: import lutorpy
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-2-db94a7dcb3df> in <module>()
----> 1 import lutorpy
/usr/local/lib/python2.7/site-packages/lutorpy/__init__.py in <module>()
10 assert not os.path.exists('./lutorpy/_lupa.pyx'), 'PLEASE DO NOT IMPORT LUTORPY FROM SOURCE CODE FOLDER.'
11
---> 12 from lutorpy import torch_path
13
14 TorchInstallPath = torch_path.__torch_path__
ImportError: cannot import name torch_path
Hi...
I am getting following error when I am doing import lutorpy
Traceback (most recent call last):
File "", line 1, in
File "lutorpy/init.py", line 56, in
from lutorpy._lupa import *
ImportError: No module named _lupa
How to fix it?Thanks in advances
lua file works fine
require "torch"
require "nn"
require "nngraph"
require "optim"
package.path = '/root/runtime/SROM/srom_torch_model.lua;' .. package.path
require "srom_torch_model"
model = Torch_model.new('./')
print(model)
python3 file doesn't work
import lutorpy as lua
require("torch")
require("nn")
require("nngraph")
require("optim")
package.path = '/root/runtime/SROM/srom_torch_model.lua;' + package.path
require("srom_torch_model")
model = Torch_model.new("./")
print(model)
Throws the error:
File "test.py", line 9, in
model = Torch_model.new("./")
NameError: name 'Torch_model' is not defined
Any idea about this? The srom_torch_model is a customized lua model. Thanks
I tried to use lutorypy with pytorch together, but there is coredump for MaxPool2d layer.
Would you like to fix it, please?
1 import lutorpy as lua
2 import torch as pt
3 print 'start test'
4 m = pt.nn.MaxPool2d(3, stride=2)
5 input = pt.autograd.Variable(pt.randn(20, 16, 50, 32))
6 pt_output = m(input)
7 print 'finish test'
Hi,
I have installed using pip and everything went smooth. But when i try to import the package in python it fails. PFB logs
I am using Ubuntu 16.04 and torch luajit.
(py35bkJune) jarvis@jarvis-HP-ENVY-m6-Notebook-PC:PycharmProjects/thesis/beauty-torch-master$ sudo pip install lutorpy
The directory '/home/jarvis/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/jarvis/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting lutorpy
Downloading lutorpy-1.3.7.tar.gz (210kB)
100% |████████████████████████████████| 215kB 3.1MB/s
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/lib/python3/dist-packages (from lutorpy)
Requirement already satisfied (use --upgrade to upgrade): future in /usr/local/lib/python3.5/dist-packages (from lutorpy)
Installing collected packages: lutorpy
Running setup.py install for lutorpy ... done
Successfully installed lutorpy-1.3.7
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(py35bkJune) jarvis@jarvis-HP-ENVY-m6-Notebook-PC:PycharmProjects/thesis/beauty-torch-master$ python
Python 3.5.2 |Anaconda custom (64-bit)| (default, Jul 2 2016, 17:53:06)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
import lutorpy
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named 'lutorpy'
Hello
I am using lutorpy and it's great. One issue is that each "require" take a long time (since I run the same script many times). I would like to cache it, i.e., load once and use for all runs.
Thanks!
I'm following the installation instruction and get this error:
lutorpy/_lupa.c:299:17: fatal error: lua.h: No such file or directory
Was I supposed to update the include, path?
when i run sudo python setup.py install
I get the following error:
lutorpy/_lupa.c:299:17: fatal error: lua.h: No such file or directory
#include "lua.h"
^
I do have lua.h and it's in the PATH.
It took me
sudo TORCH_INSTALL=/home/username/torch/install/ pip install lutorpy
instead of
sudo pip install lutorpy
to install.
I'm guessing, it's not just me?
How should I handle below case?
output_image[{{i},{},{}}]:add(-meanStd.mean[i])
what should be the syntax to be used for [{{i},{},{}}]
My Requirement is simple. To be able to create a lua table which I can pass to the trainer
criterion = nn.MSECriterion()
self.trainer = nn.StochasticGradient(self.net, criterion)
self.trainer.learningRate = 0.01
self.trainer._train(dataset)
Now the dataset is a lua table, each row containing two lua tensors (input, and output)
How do I create this dataset variable if I'm starting from a X, y with sizes (50,10) and (50,1) respectively.
I'm aware of torch.fromNumpyArray functions. But it's unclear how to create the dataset variable in my case.
I get the following error on importing lutorpy.
I am using ubuntu 16.04 and python2.7.
$ python -c "import lutorpy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/users/sasank/anaconda3/envs/py2/lib/python2.7/site-packages/lutorpy/__init__.py", line 15, in <module>
THlib = ctypes.CDLL(os.path.join(TorchInstallPath, "lib/libTH.so"), mode=ctypes.RTLD_GLOBAL)
File "/home/users/sasank/anaconda3/envs/py2/lib/python2.7/ctypes/__init__.py", line 362, in __init__
self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/liblapack.so.3: undefined symbol: gotoblas
Weirdly, this error is solved if I uninstall openblas-base
.
$ sudo apt-get remove libopenblas-base
But usual torch scripts stop working on uninstall and surprisingly, lutorpy works fine.
If I install openblas-dev
,
$ sudo apt-get install libopenblas-dev
torch works fine but lutorpy starts giving above error.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.