when using Multi-GPU training , get "all tensors must be on devices[0]"

[meijieru]

when using Multi-GPU training , get "all tensors must be on devices[0]"

[meijieru]

Could you provide details of the problem?
Thanks.

[Lzc6996]

  batch_index = random_start + torch.range(0, self.batch_size - 1)
Traceback (most recent call last):
  File "crnn_main.py", line 220, in <module>
    cost = trainBatch(crnn, criterion, optimizer)
  File "crnn_main.py", line 195, in trainBatch
    preds = crnn(image)
  File "/gruntdata/DL_dataset/steven.lzc/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "/gruntdata/DL_dataset/steven.lzc/workspace/crnn.pytorch.armor/models/crnn.py", line 86, in forward
    output = utils.data_parallel(self.rnn, conv, self.ngpu)
  File "/gruntdata/DL_dataset/steven.lzc/workspace/crnn.pytorch.armor/models/utils.py", line 10, in data_parallel
    output = nn.parallel.data_parallel(model, input, range(ngpu))
  File "/gruntdata/DL_dataset/steven.lzc/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 105, in data_parallel
    outputs = parallel_apply(replicas, inputs, module_kwargs)
  File "/gruntdata/DL_dataset/steven.lzc/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 46, in parallel_apply
    raise output
RuntimeError: all tensors must be on devices[0]

i got the error when i use --ngpu 2

[YoungMiao]

@Lzc6996 have you solve this problem? thanks

[XuLiangFRDC]

I also find this problem.
My OS is CentOS Linux release 7.2.1511 (Core), x86_64.
My server has four Tesla P40 GPU cards.

My command is:
python crnn_main.py --trainroot /home/xuliang/CRNN_org/crnn/tool/synth90k_test_sort --valroot /home/xuliang/CRNN_org/crnn/tool/synth90k_val_sort --cuda --ngpu 2 --keep_ratio --random_sample

And error message is:
Namespace(adadelta=False, adam=False, alphabet='0123456789abcdefghijklmnopqrstuvwxyz', batchSize=64, beta1=0.5, crnn='', cuda=True, displayInterval=500, experiment=None, imgH=32, imgW=100, keep_ratio=True, lr=0.01, n_test_disp=10, ngpu=2, nh=256, niter=25, random_sample=True, saveInterval=500, trainroot='/home/xuliang/CRNN_org/crnn/tool/synth90k_test_sort', valInterval=500, valroot='/home/xuliang/CRNN_org/crnn/tool/synth90k_val_sort', workers=2)
Random Seed: 7224
CRNN (
(cnn): Sequential (
(conv0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu0): ReLU (inplace)
(pooling0): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu1): ReLU (inplace)
(pooling1): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
(conv2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(batchnorm2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
(relu2): ReLU (inplace)
(conv3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu3): ReLU (inplace)
(pooling2): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
(conv4): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(batchnorm4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(relu4): ReLU (inplace)
(conv5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu5): ReLU (inplace)
(pooling3): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
(conv6): Conv2d(512, 512, kernel_size=(2, 2), stride=(1, 1))
(batchnorm6): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(relu6): ReLU (inplace)
)
(rnn): Sequential (
(0): BidirectionalLSTM (
(rnn): LSTM(512, 256, bidirectional=True)
(embedding): Linear (512 -> 256)
)
(1): BidirectionalLSTM (
(rnn): LSTM(256, 256, bidirectional=True)
(embedding): Linear (512 -> 37)
)
)
)
Traceback (most recent call last):
File "crnn_main.py", line 197, in
cost = trainBatch(crnn, criterion, optimizer)
File "crnn_main.py", line 180, in trainBatch
preds = crnn(image)
File "/root/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/home/xuliang/CRNN_pytorch/crnn.pytorch/models/crnn.py", line 85, in forward
output = utils.data_parallel(self.rnn, conv, self.ngpu)
File "/home/xuliang/CRNN_pytorch/crnn.pytorch/models/utils.py", line 10, in data_parallel
output = nn.parallel.data_parallel(model, input, range(ngpu))
File "/root/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 105, in data_parallel
outputs = parallel_apply(replicas, inputs, module_kwargs)
File "/root/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/parallel_apply.py", line 46, in parallel_apply
raise output
RuntimeError: all tensors must be on devices[0]

How to solve this problem?

[meijieru]

@XuLiangFRDC Have you try the updated version?

[XuLiangFRDC]

@meijieru I updated the code to the current version. Now a different problem appeared:
The command is:
python crnn_main.py --trainroot /home/xuliang/CRNN_org/crnn/tool/synth90k_train_sort --valroot /home/xuliang/CRNN_org/crnn/tool/synth90k_val_sort --cuda --ngpu 4 --adadelta --keep_ratio --random_sample

And error message is:
Namespace(adadelta=True, adam=False, alphabet='0123456789abcdefghijklmnopqrstuvwxyz', batchSize=64, beta1=0.5, crnn='', cuda=True, displayInterval=500, experiment=None, imgH=32, imgW=100, keep_ratio=True, lr=0.01, n_test_disp=10, ngpu=4, nh=256, niter=25, random_sample=True, saveInterval=500, trainroot='/home/xuliang/CRNN_org/crnn/tool/synth90k_train_sort', valInterval=500, valroot='/home/xuliang/CRNN_org/crnn/tool/synth90k_val_sort', workers=2)
Random Seed: 7716
CRNN (
(cnn): Sequential (
(conv0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu0): ReLU (inplace)
(pooling0): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu1): ReLU (inplace)
(pooling1): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
(conv2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(batchnorm2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
(relu2): ReLU (inplace)
(conv3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu3): ReLU (inplace)
(pooling2): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
(conv4): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(batchnorm4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(relu4): ReLU (inplace)
(conv5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu5): ReLU (inplace)
(pooling3): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
(conv6): Conv2d(512, 512, kernel_size=(2, 2), stride=(1, 1))
(batchnorm6): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(relu6): ReLU (inplace)
)
(rnn): Sequential (
(0): BidirectionalLSTM (
(rnn): LSTM(512, 256, bidirectional=True)
(embedding): Linear (512 -> 256)
)
(1): BidirectionalLSTM (
(rnn): LSTM(256, 256, bidirectional=True)
(embedding): Linear (512 -> 37)
)
)
)
[0/25][500/112885] Loss: 16.009935
Start val
Traceback (most recent call last):
File "crnn_main.py", line 207, in
val(crnn, test_dataset, criterion)
File "crnn_main.py", line 158, in val
sim_preds = converter.decode(preds.data, preds_size.data, raw=False)
File "/home/xuliang/CRNN_pytorch_v2/crnn.pytorch/utils.py", line 51, in decode
t[index:index + l], torch.IntTensor([l]), raw=raw))
ValueError: result of slicing is an empty tensor

Thanks for your help.

[Lzc6996]

I have solve it. change output = utils.data_parallel(self.rnn, conv, self.ngpu) to output = self.rnn(conv) makes it work.

[meijieru]

@XuLiangFRDC Please open another issue as it seems not to be this problem

[XuLiangFRDC]

@meijieru I have opened another issue. Please see:
#41
thanks!

[XuLiangFRDC]

@Lzc6996 Could you please give more details according to your solution: I have solve it. change output = utils.data_parallel(self.rnn, conv, self.ngpu) to output = self.rnn(conv) makes it work.

I modified the file (crnn.pytorch-master/models/crnn.py) as following:
import utils

replace code: output = self.rnn(conv)
as: output = utils.data_parallel(self.rnn, conv, self.ngpu)

And I added the file (utils.py) into dir ((crnn.pytorch-master/models):
the new file utils.py is defined as following:

#!/usr/bin/python
#encoding: utf-8

import torch.nn as nn
import torch.nn.parallel

def data_parallel(model, input, ngpu):
if isinstance(input.data, torch.cuda.FloatTensor) and ngpu > 1:
output = nn.parallel.data_parallel(model, input, range(ngpu))
else:
output = model(input)
return output

But it doesn't work. Why?
The command is:
python crnn_main.py --trainroot /home/xuliang/CRNN_org/crnn/tool/synth90k_train_sort --valroot /home/xuliang/CRNN_org/crnn/tool/synth90k_val_sort --cuda --adadelta --keep_ratio --random_sample

The following error message is:
Namespace(adadelta=True, adam=False, alphabet='0123456789abcdefghijklmnopqrstuvwxyz', batchSize=64, beta1=0.5, crnn='', cuda=True, displayInterval=500, experiment=None, imgH=32, imgW=100, keep_ratio=True, lr=0.01, n_test_disp=10, ngpu=1, nh=256, niter=25, random_sample=True, saveInterval=500, trainroot='/home/xuliang/CRNN_org/crnn/tool/synth90k_train_sort', valInterval=500, valroot='/home/xuliang/CRNN_org/crnn/tool/synth90k_val_sort', workers=2)
Random Seed: 5088
CRNN (
(cnn): Sequential (
(conv0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu0): ReLU (inplace)
(pooling0): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
(conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu1): ReLU (inplace)
(pooling1): MaxPool2d (size=(2, 2), stride=(2, 2), dilation=(1, 1))
(conv2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(batchnorm2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
(relu2): ReLU (inplace)
(conv3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu3): ReLU (inplace)
(pooling2): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
(conv4): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(batchnorm4): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(relu4): ReLU (inplace)
(conv5): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(relu5): ReLU (inplace)
(pooling3): MaxPool2d (size=(2, 2), stride=(2, 1), dilation=(1, 1))
(conv6): Conv2d(512, 512, kernel_size=(2, 2), stride=(1, 1))
(batchnorm6): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(relu6): ReLU (inplace)
)
(rnn): Sequential (
(0): BidirectionalLSTM (
(rnn): LSTM(512, 256, bidirectional=True)
(embedding): Linear (512 -> 256)
)
(1): BidirectionalLSTM (
(rnn): LSTM(256, 256, bidirectional=True)
(embedding): Linear (512 -> 37)
(rnn): LSTM(256, 256, bidirectional=True)
(embedding): Linear (512 -> 37)
)
)
)

Traceback (most recent call last):
File "crnn_main.py", line 197, in
cost = trainBatch(crnn, criterion, optimizer)
File "crnn_main.py", line 180, in trainBatch
preds = crnn(image)
File "/root/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/root/anaconda2/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 59, in forward
return self.module(*inputs[0], **kwargs[0])
File "/root/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in call
result = self.forward(*input, **kwargs)
File "/home/xuliang/CRNN_pytorch_v2/crnn.pytorch/models/crnn.py", line 83, in forward
output = utils.data_parallel(self.rnn, conv, self.ngpu)
File "/root/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 238, in getattr
type(self).name, name))
AttributeError: 'CRNN' object has no attribute 'ngpu'

By the way, I changed the code to include attribute 'ngpu' and this problem is solved.
But for Multi-GPU, there still exists problem,
Please see:
#41