About "Native Implemention"

[nvlabs]

About "Native Implemention"

[suhangpro]

The native implementation is using only standard pytorch layers/operations, while the other one uses the Function interface at places. The latter can have some memory advantage for larger input sizes.

[suhangpro]

There shouldn't be any noticeable differences in speed. The non-native implementation has advantages in terms of peak memory usage, but I have no statistics regarding the gap. The native version was there mostly for debugging purposes, and the non-native version should be the preferred option now.

[Jerrypiglet]

Hey Hang, after a second thought, the original version is actually correct to me. kernel will be used when you compute grad_input, and input will be used when you compute grad_kernel. See the highlights below.

In your updated version I suppose you will see errors in https://github.com/NVlabs/pacnet/blob/master/pac.py#L352 that input is None when ctx.needs_input_grad[0] == False and ctx.needs_input_grad[0] == True. Am I right?

[Jerrypiglet]

Hey Hang, thanks for the comment! Additionally I am wondering if you have any statistics on how big the performance gap is between native and functional implementations, especially in case of large input sizes, or any differences in backprop speed, because I did some basic benchmarking and did not find significant difference there. Thanks!

[Jerrypiglet]

Ah i see. Thanks! The native version would also be useful for prototyping because it is easier to build upon.

[suhangpro]

@Jerrypiglet That's right. I wasn't careful enough on this. The change is now reverted.