Regarding Separating Sentences.

[tahmedge]

Regarding Separating Sentences.

[astariul]

If document is :

Sentence1. Sentence2. Sentence3.

Then it will be separated with [CLS] token and [SEP] token, like this :

[CLS] Sentence1. [SEP] [CLS] Sentence2. [SEP] [CLS] Sentence3. [SEP]

[astariul]

No. Sorry my answer was not clear.

In the dataset, there is a sentence every line. So the StanFord tokenizer use the newline as a sentence separator :

PreSumm/src/prepro/data_builder.py

Lines 123 to 125 in 29a6b1a

	command = ['java', 'edu.stanford.nlp.pipeline.StanfordCoreNLP', '-annotators', 'tokenize,ssplit',
	'-ssplit.newlineIsSentenceBreak', 'always', '-filelist', 'mapping_for_corenlp.txt', '-outputFormat',
	'json', '-outputDirectory', tokenized_stories_dir]

And then, when the dataset is processed, the author put [CLS] token and [SEP] token as separator between sentences.

[tahmedge]

Thanks for your reply. So it means it is separated by ' . ' token?

[tahmedge]

Thanks.
Btw, I am also confused regarding the decoding phase. What I understand is that at the end of the encoding phase, we will have [CLS] [S1, T1][S1 T,2].... [SEP] [CLS] [S2 ,T1] [S2 T,2].... [SEP] [CLS] [S3 ,T1] [S3 .T2].... [SEP] .....
Here, S1, S2, S3 are the sentences whereas T1 , T2 .... are the tokens of the sentences.
So during decoding phase, is not the decoder focusing on all the tokens including [CLS] [Si, Tj] [SEP] etc. ? Or is it just focusing on [CLS] token of each sentence? I did not find detailed explanation regarding decoding phase in the paper.

[astariul]

I'm not sure what you mean by encoding and decoding phase...

But anyway only tokens of the sentences are kept for the model.
For Extractive summarization, only [CLS] tokens are used, while for Abstractive summarization, all tokens representations are used.

[tahmedge]

Does the newline in the summary separated by token?