RuntimeError: cublas runtime error : an illegal memory access was encountered

[chengdazhi]

RuntimeError: cublas runtime error : an illegal memory access was encountered

[Jiaoma]

By the way, I couldn't find any information about SparseConv2d in Deformable ConvNets v2: More Deformable, Better Results . What is it?

[Jiaoma]

Complete error output:
Traceback (most recent call last):
File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
runfile('/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py', wdir='/home/molijuly/github/SwiftDPP/models/Nets')
File "/home/molijuly/pycharm-2018.3.2/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/home/molijuly/pycharm-2018.3.2/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py", line 103, in
re=net(a)
File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py", line 77, in forward
enc1 = self.enc1(x)
File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py", line 34, in forward
return self.encode(x)
File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward
input = module(input)
File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/home/molijuly/github/dcn/modules/modulated_deform_conv2d.py", line 104, in forward
self.im2col_step)
File "/home/molijuly/github/dcn/functions/modulated_deform_conv2d_func.py", line 34, in forward
ctx.im2col_step)
RuntimeError: cublas runtime error : resource allocation failed at /opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THC/THCGeneral.cpp:250
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=77 : an illegal memory access was encountered

[Lausannen]

@Jiaoma I met the same problem, have you solved the problem?

[Jiaoma]

@Jiaoma I met the same problem, have you solved the problem?

I use the Deformable ConvNets v2 in mmdetecion, and it works.

[tkkcc]

Finally found a mismatch between offset size and feature size