taolei87/askubuntu

AskUbuntu Question Dataset

askubuntu's Issues

Number of Tokens in vectors_pruned.200.txt

The number of tokens in the file vectors_pruned.200.txt is 100406.
I wanted to know if some of the tokens from the askUbuntu Dataset have been removed to achieve the above number of 100406.
If not which tokenizer is used over the data.

BM25 clarification

Hi @taolei87 ,

Thanks for sharing these data! I would like to ask on which text was the BM25 calculated?

In the test.txt and dev.txt files there are the BM25 scores of the questions computed by the Lucene search engine. However, I didn't see anywhere whether the scores are based on the titles, the bodies of the questions, or both of them, and whether stopwords are removed for these scores. Could you please clarify?

Thanks in advance:)

duplicate instances in the negative set

Dear @taolei87 , first of all, thank you for compiling this interesting corpus.

While exploring it, I have noted that the negative instances in the train_random.txt set include duplicates sometimes. For instance:

query : negative (random) instances appearing more than once
163842 : 217493, 185573
393230 : 197044
...

This phenomenon occurs in 1136 out of 12,723 instances. I fully understand this issue is not crucial and fixing it would change nothing, as it simply implies that a negative instance is considered twice. Still you might want to check the random instances selector in case there are other issues or side effects.

Best regards
albarron