rotary embedding question

[jinmang2]

rotary embedding question

[lucidrains]

@jinmang2 rotary embeddings is the best type of relative positional encoding, if you do not need to extrapolate to sequence lengths longer than what was trained on. it was proven out in PaLM! enough said! :)

[lucidrains]

@jinmang2 oh hey! yes, sorry the logic wasn't quite correct, fixed in f2d2815

so this was actually a discovery by EleutherAI and the building of GPT-J by Wang & Komatsuzaki et al. They found out that only rotating a part of the head dimension (and leaving the rest unrotated) leads to even better performance. I believe Deepmind picked up this practice in their latest language model (Gopher or something else)

And yes, you are correct that rotary can be split two ways! But as long as they match up later, it doesn't matter (and I believe I have it matched up correctly, but do submit a PR if you find otherwise)

[jinmang2]

I thought that Shaw' RPE was simply used in the RETRO, but I didn't know there was such an insight!
Thanks for the feedback :) issue closing