Regarding Decoding Phase

[tahmedge]

Regarding Decoding Phase

[astariul]

In the forward pass of Abstractive model :

PreSumm/src/models/model_builder.py

Lines 240 to 244 in 29a6b1a

	def forward(self, src, tgt, segs, clss, mask_src, mask_tgt, mask_cls):
	top_vec = self.bert(src, segs, mask_src)
	dec_state = self.decoder.init_decoder_state(src, top_vec)
	decoder_outputs, state = self.decoder(tgt[:, :-1], top_vec, dec_state)
	return decoder_outputs, None

Decoder uses all tokens representations from encoder, not only [CLS] token.

---

In the forward pass of Extractive model :

PreSumm/src/models/model_builder.py

Lines 170 to 175 in 29a6b1a

	def forward(self, src, segs, clss, mask_src, mask_cls):
	top_vec = self.bert(src, segs, mask_src)
	sents_vec = top_vec[torch.arange(top_vec.size(0)).unsqueeze(1), clss]
	sents_vec = sents_vec * mask_cls[:, :, None].float()
	sent_scores = self.ext_layer(sents_vec, mask_cls).squeeze(-1)
	return sent_scores, mask_cls

Extractive layers uses only [CLS] token representations from encoder.

[tahmedge]

Thanks for your reply. :)