if the max is 512 for tokenization, do you tokenize in chunks?

[nlpyang]

if the max is 512 for tokenization, do you tokenize in chunks?

[astariul]

There is no chunks, article is truncated at 512 and remaining text is not used.

PreSumm/src/models/data_loader.py

Lines 204 to 209 in fa69433

	end_id = [src[-1]]
	src = src[:-1][:self.args.max_pos - 1] + end_id
	segs = segs[:self.args.max_pos]
	max_sent_id = bisect.bisect_left(clss, self.args.max_pos)
	src_sent_labels = src_sent_labels[:max_sent_id]
	clss = clss[:max_sent_id]

[nlpyang]

No chunking is used.

If you want to encoder text longer than 512 (like 800 for example), checkout the dev branch, set max_pos=800 in both preprocessing and training. The model will add additional positional embeddings.

This has been described in the paper.

[aretius]

Hey @nlpyang ,
Apologies for reopening the issue. But in the paper you did mention training on NYT dataset with input truncated to 800 tokens. Do you have a 800 tokens pre-trained model available usable in the same format as CNN ones?

[AyeshaSarwar]

Yes, it is mentioned in the paper but then if you are initializing them randomly, isn't it the same thing that we are only considering only the first 512 tokens of the original document?
@nlpyang @astariul