drcknowledgeteam/drcd

A 30000+ Chinese MRC dataset - Delta Reading Comprehension Dataset

drcd's Issues

Evaluation problem

According to the paper:

F1 score and exact match from Rajpurkar et al. (2016) are used as the evaluation metrics. Both metrics ignore punctuations. In F1 score metric, we consider predictions and ground truth as bag of Chinese character.

Is the ignored punctuations including fullwidth? Is there a original evaluation script for this dataset?

Dev problem

The dev set answers are duplicate in the same question.
Also, in SQuAD dataset, it has 3 different answers in dev set and test set, so the human performance is much higher than your dataset in EM performance. Are you going to provide more answer?

想问一下怎么样才能获取全部的数据集呢

是要发邮件来申请全部的数据集嘛？

Test set

What's current F1 score or EM score for test set if Chinese MRC dataset here ??

Training set problem

In the paper, it said that "the training set contains 26,932 questions in 8,014 paragraphs".
However, after calculating I found that I got 26936 question's id in the json file.

Format conversion to SQuAD2.0

When you used Bert-Chinese model to do the DRCD tasks like your paper told us, is there anything such as format conversion that we need to do first, and then we can use Bert-Chinese model to do DRCD tasks ?

p.s. Format conversion means that convert DRCD format to SQuAD2.0 format.