RuntimeError: Backward is not reentrant

[chengdazhi]

RuntimeError: Backward is not reentrant

[JiancongWang]

I checked the code. Numerically the code should be correct. I verified that the error is very small using CUDA 9.0 on a GTX 1080 ~ 1e-16. The error is raised when calculating the gradient for input. The code uses atomicadd to calculate that since one can not possibly know how many time a pixel get used. Then the problem is the sequence of atomicadd is uncertain. Different sequence of adding will result in different value up to numerical error. The gradient on the bias/weight/offset does not have the atomicadd and thus have no such issue.

[xvjiarui]

Sorry for the late reply.

Deformable-Convolution-V2-PyTorch/test.py

Lines 627 to 631 in 1b5851a

	# """
	# ****** Note: backward is not reentrant error may not be a serious problem,
	# ****** since the max error is less than 1e-7,
	# ****** Still looking for what trigger this problem
	# """

As in comment, it won't affect the performance. Feel free to ignore.
Please reopen this issue if you have a solution. Thx

[GreenTeaHua]

thx

[JiancongWang]

That's only one possible cause of the issue though. I will double check and see if there is any other problem. For now I don't see any.

[heartInsert]

I checked the code. Numerically the code should be correct. I verified that the error is very small using CUDA 9.0 on a GTX 1080 ~ 1e-16. The error is raised when calculating the gradient for input. The code uses atomicadd to calculate that since one can not possibly know how many time a pixel get used. Then the problem is the sequence of atomicadd is uncertain. Different sequence of adding will result in different value up to numerical error. The gradient on the bias/weight/offset does not have the atomicadd and thus have no such issue.

You mean I can ignore this exception?