etienne87/torch_object_rnn

detection network with conv-rnn

torch_object_rnn's Issues

Training problem

When I start to train like below, an error comes up.
python train_det.py ./logs --batchsize=4 --cuda

==> Preparing dataset..
==> Building model..

Epoch: 0 (train)
C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [37,0,0], thread: [5,0,0] Assertion `index

= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [37,0,0], thread: [7,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [37,0,0], thread: [9,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [37,0,0], thread: [21,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [37,0,0], thread: [23,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [37,0,0], thread: [25,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [36,0,0], thread: [117,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [36,0,0], thread: [119,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [36,0,0], thread: [121,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [82,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [84,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [119,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [121,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [79,0,0], thread: [3,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [79,0,0], thread: [5,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [79,0,0], thread: [7,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [79,0,0], thread: [9,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [79,0,0], thread: [21,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [79,0,0], thread: [23,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [74,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [82,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [84,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [3,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [5,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [7,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [9,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [21,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [23,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [122,0,0], thread: [74,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [122,0,0], thread: [76,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [122,0,0], thread: [82,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [5,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [7,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [9,0,0] Assertionindex
= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [11,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [21,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [23,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [15,0,0], thread: [25,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [101,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [103,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [105,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [117,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [119,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [121,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [5,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [7,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [9,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [23,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [74,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [76,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [82,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [164,0,0], thread: [84,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [103,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [117,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [119,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [58,0,0], thread: [121,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [143,0,0], thread: [74,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [143,0,0], thread: [76,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [143,0,0], thread: [82,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [100,0,0], thread: [74,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [100,0,0], thread: [82,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [121,0,0], thread: [101,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [121,0,0], thread: [103,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [121,0,0], thread: [105,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [121,0,0], thread: [117,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [121,0,0], thread: [119,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [121,0,0], thread: [121,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [100,0,0], thread: [5,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [100,0,0], thread: [7,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [100,0,0], thread: [9,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:/w/1/s/tmp_conda_3.7_183424/conda/conda-bld/pytorch_1570818936694/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: block: [100,0,0], thread: [23,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
File "c:\Users\sjy.vscode\extensions\ms-python.python-2019.10.44104\pythonFiles\ptvsd_launcher.py", line 43, in
main(ptvsdArgs)
File "c:\Users\sjy.vscode\extensions\ms-python.python-2019.10.44104\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py", line 432, in main
run()
File "c:\Users\sjy.vscode\extensions\ms-python.python-2019.10.44104\pythonFiles\lib\python\old_ptvsd\ptvsd_main_.py", line 316, in run_file
runpy.run_path(target, run_name='main')
File "C:\Users\sjy.conda\envs\pytorch_1.3.0\lib\runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "C:\Users\sjy.conda\envs\pytorch_1.3.0\lib\runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "C:\Users\sjy.conda\envs\pytorch_1.3.0\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "d:\Work\OpenSources\torch_object_rnn-master\train_det.py", line 133, in
main()
File "d:\Work\OpenSources\torch_object_rnn-master\train_det.py", line 120, in main
trainer.train(epoch, train_dataset, args)
File "d:\Work\OpenSources\torch_object_rnn-master\core\trainer.py", line 68, in train
loss_dict = self.net.compute_loss(inputs, targets)
File "d:\Work\OpenSources\torch_object_rnn-master\core\single_stage_detector.py", line 52, in compute_loss
loc_loss, cls_loss = self.criterion(loc_preds, loc_targets, cls_preds, cls_targets)
File "C:\Users\sjy.conda\envs\pytorch_1.3.0\lib\site-packages\torch\nn\modules\module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "d:\Work\OpenSources\torch_object_rnn-master\core\losses.py", line 110, in forward
cls_loss = self.cls_loss_func(cls_preds, cls_targets, 'sum') / num_pos
File "d:\Work\OpenSources\torch_object_rnn-master\core\losses.py", line 42, in softmax_focal_loss
loss[y < 0] = 0
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered