Mismatch between pre-training and fine-tuning phase

[thangld201]

Mismatch between pre-training and fine-tuning phase

[thangld201]

@Wendy-Xiao Could you help me clarify this ?

[Wendy-Xiao]

Hi Tang,

It's a good point, we do not have any particular strategy to address the gap between pre-training and the downstream task. As it has been noticed, there are still some problems with our model in the zero-shot setting, e.g. the length of generated summaries can not really be controlled except for setting a hard stop with a certain length.

In our initial explorations, we tried some methods to address the gap, e.g. adding multiple tokens as prefix of the input document. The interesting thing we found was that the more we added, the longer summary it tended to generate.

Few-shot finetuning could address the problem to some extend, so we will suggest the best way to use PRIMERA is in a few-shot manner, i.e. finetuning the model on few examples, so that it can learn the real task without .