Comments (6)
If document is :
Sentence1. Sentence2. Sentence3.
Then it will be separated with [CLS]
token and [SEP]
token, like this :
[CLS] Sentence1. [SEP] [CLS] Sentence2. [SEP] [CLS] Sentence3. [SEP]
from presumm.
No. Sorry my answer was not clear.
In the dataset, there is a sentence every line. So the StanFord tokenizer use the newline
as a sentence separator :
PreSumm/src/prepro/data_builder.py
Lines 123 to 125 in 29a6b1a
And then, when the dataset is processed, the author put [CLS]
token and [SEP]
token as separator between sentences.
from presumm.
Thanks for your reply. So it means it is separated by ' . ' token?
from presumm.
Thanks.
Btw, I am also confused regarding the decoding phase. What I understand is that at the end of the encoding phase, we will have [CLS] [S1, T1][S1 T,2].... [SEP] [CLS] [S2 ,T1] [S2 T,2].... [SEP] [CLS] [S3 ,T1] [S3 .T2].... [SEP] .....
Here, S1, S2, S3 are the sentences whereas T1 , T2 .... are the tokens of the sentences.
So during decoding phase, is not the decoder focusing on all the tokens including [CLS] [Si, Tj] [SEP] etc. ? Or is it just focusing on [CLS] token of each sentence? I did not find detailed explanation regarding decoding phase in the paper.
from presumm.
I'm not sure what you mean by encoding and decoding phase...
But anyway only tokens of the sentences are kept for the model.
For Extractive summarization, only [CLS]
tokens are used, while for Abstractive summarization, all tokens representations are used.
from presumm.
Does the newline in the summary separated by token?
from presumm.
Related Issues (20)
- having [Errno 21] Is a directory: while running train for BertExtAbs
- step 4 converting to simpler json returning asci error
- TypeError: __init__() got an unexpected keyword argument 'temp_dir'
- example_add_guidance.py
- Error when testing BertAbs model HOT 2
- Acc is very low and does not converge during training
- Cannot load model via torch.load HOT 1
- xsum数据集 HOT 1
- data preprocessing: empty 'tgt' text HOT 1
- issue for converting to bert_data HOT 2
- Use pretrained model : train_from HOT 9
- Getting the same sequence for all input candidate in generation
- How to do inference using pretrained bertsum models?
- Training the BERT large extractive model
- How can i know if i download BERT successfully
- bert-base-uncased HOT 2
- error in step 3 HOT 1
- 在运行test模式,BertAbs模型时,遇到了RuntimeError: "index_select_out_cuda_impl" not implemented for 'Float' HOT 1
- How to save the best model? HOT 1
- RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from presumm.