Comments (4)
RoBERTa has a 512 limit. Our pretrained model has a 4096 limit. Both can be retrained for longer sequences if you want.
from longformer.
LongFormer does it by pooling all the local attentions (512) together in global attention (512 x 8 = 4096).
This is not entirely true. There's no "pooling" of the 4096 tokens into 512. We keep all 4096 tokens. The only change is how attention is computed; instead of every token attending to every other token, we change it such that every token attends to a smaller number of surrounding tokens. This speeds up selfattention computation (which is the bottleneck) by assuming that the attention score between certain pairs of words is zero. This doesn't change the architecture or introduce any pooling.
We are working on some code that will make it easy to train your own long model, so you can try longer sequences. We know it is easy to get to 16K or even 32k with RoBERTa-base architecture (need base model, fp16, gradient checkpointing). For sequences longer than that, you will need to find ways to save memory depending on your application. For example, reducing window size, reducing size of the feed forward layers, implementing reversible transformers, use sinusoidal position embedding instead of learned position embedding.
from longformer.
Thank you for the explanation. I thought RoBERTa has an attention constraint of 512 tokens too [here], just clarifying.
from longformer.
Closing. Please feel free to reopen or create a new one if you have more questions.
from longformer.
Related Issues (20)
- CUDA error: device-side assert triggered in multi class text classification
- AttributeError: 'RobertaEmbeddings' object has no attribute 'position_ids' HOT 1
- AttributeError: module 'dill._dill' has no attribute 'stack'
- Can't find a valid checkpoint at tmp
- One hot encoding classes
- LED Training Time
- Pretraining longformer for NER on big pdf text
- Why the TVM impelmentation is memroy efficient
- Updated BART to Longformer-encoder-decoder (LED) converter
- Answering performance of Longformer-base on the HotpotQA dev set
- Number of tokens per batch mismatch - longformer vs roberta HOT 1
- On cheatsheet
- @ibeltagy I have similar issues with converting the model to ONNX, I converted the model to ONNX model, but when I tried to infer with onnxruntime I got ScatterND error while session run. I am guessing there are some operations not supported by onnx.
- Longformer embeddings for calculating similarity score between 2 documents using KNN
- Where is the global attention?
- Reproducibility Problem
- may you share link to somebody else latest development for long sentences pls?
- Can Longformer support adapter transformer?
- "requirements.txt" update (transformers==3.0.2) HOT 2
- Cosine similarity scores between random words are well above 0.9 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from longformer.