Some fun experiments around different types of NLP attention mechanisms
Done:
Todo:
- Shaped Transformer is still a WIP,
CovarianceTracker
class still needs some more work, and also need to derive Theorem 4.2 on page 9 and 30 of the paper. - Shift Short Attention seems similar to Blockwise Causal Attention, BCA
- Intention requires some expensive matrix inversion operations on top of attention compute logic, this may be why it is not widely adopted until now.
Credit: Rishikesh Magar, @ZickZack