As you said, it need to remove duplicate documents in KP20k. Can i know source code or methods how to check same document between KP20k and test sets?
We also found out that the training set of KP20k contains a non-negligible number of documents from the test sets of other datasets. We removed those
documents prior to training.