help with understanding the intuition behind the "4096" attention window.

[allenai]

help with understanding the intuition behind the "4096" attention window.

[ibeltagy]

RoBERTa has a 512 limit. Our pretrained model has a 4096 limit. Both can be retrained for longer sequences if you want.

[ibeltagy]

LongFormer does it by pooling all the local attentions (512) together in global attention (512 x 8 = 4096).

This is not entirely true. There's no "pooling" of the 4096 tokens into 512. We keep all 4096 tokens. The only change is how attention is computed; instead of every token attending to every other token, we change it such that every token attends to a smaller number of surrounding tokens. This speeds up selfattention computation (which is the bottleneck) by assuming that the attention score between certain pairs of words is zero. This doesn't change the architecture or introduce any pooling.

We are working on some code that will make it easy to train your own long model, so you can try longer sequences. We know it is easy to get to 16K or even 32k with RoBERTa-base architecture (need base model, fp16, gradient checkpointing). For sequences longer than that, you will need to find ways to save memory depending on your application. For example, reducing window size, reducing size of the feed forward layers, implementing reversible transformers, use sinusoidal position embedding instead of learned position embedding.

[vr25]

Thank you for the explanation. I thought RoBERTa has an attention constraint of 512 tokens too [here], just clarifying.

[ibeltagy]

Closing. Please feel free to reopen or create a new one if you have more questions.