After FFT & Winograd, what next?

[ottolu]

After FFT & Winograd, what next?

[andravin]

Hi @nomi-wei , just a clarification: our fast convnet algorithms use Winograd's convolution algorithms. But the same Shmuel Winograd did co-author the Coppersmith-Winograd fast matrix multiplication algorithm, so the confusion is understandable (I probably should not even mention that Winograd also devised fast DFT algorithms ;-)

[ottolu]

@andravin Ha-ha, my bad. Thanks for your clarification. It's really really helpful. ;-)
I didn't got the book you referenced, so I thought you might use winograd's famous mm method for this. LoL. No wonder I still find it hard to understand your approach, after I learned these mm algorithms from scratch these few days.

Thanks again!
BTW, Andrew, I wonder if you're still working on improving this conv kernel stuff? If yes, that would be awesome.

[hughperkins]

I think in theory 8-bit is enough to carry the information with quantization.

googling for dp4a reaches a thread with Scott Gray in as first hit :-) https://devtalk.nvidia.com/default/topic/934562/cuda-programming-and-performance/nvidia-pascal-geforce-gtx-1080-amp-gtx-1070/post/4889687/ So I would say he's aware of it :-)

I was actually pondering dabbling with ints way back in 2014 http://computer-go.org/pipermail/computer-go/2014-December/007105.html ... but it's just one of many things that never survived contact with finite-hours-in-the-day :-)

Considering the effort involved in making gpus work, and work quickly, I would think the first thing to do might be to demonstrate using normal cpu code that you can get ok results? You could just fire up torch, and create torch.ByteTensors.

A few questions which occur:

• how will you deal with overflows? It's one thing to have multiplications truncated to some maximum value, it's another thing for them to overflow into the opposite sign...
• 8-bits means there are 256 different values. How will you deal with, well, gradients and stuff?

Hmmm, I'm simply reciting back to you the questions that were stated to me when I mentioned the idea myself :-) http://computer-go.org/pipermail/computer-go/2014-December/007106.html