How to start

[azhe825]

How to start

[timm]

From my small-scale experiments, BM25 is really a good strategy for all scenarios. It requires no previous data, little domain knowledge (knowing the keywords to search for).

huh. i wonder how much it changes the prior results?

[azhe825]

@timm
Updated some results. BM25 really works well in most cases.
How do you interpret the results?

[timm]

is this the reverse of our prior conclusion? that we dont need prior knowledge to do a good job of lit review?

[timm]

and, regardless of that, how does this play with the stopping results?

[azhe825]

is this the reverse of our prior conclusion? that we dont need prior knowledge to do a good job of lit review?

Kind of. UPDATE performs quite good when it can be applied.

• For practical use of the tool, I would suggest BM25:
  • it is reliably better than RANDOM
  • can be applied to any scenario
  • may also be useful when the review goes into a corner and the reviewer has a better view of which direction to go (search when the learner fails to find more relevant studies)
• For paper writing, not sure what to do, mainly because a) BM25 is an existing technique suggested by Cormack. b) BM25 is not consistently the best treatment (there is no such treatment).

regardless of that, how does this play with the stopping results?
Still running. Results are expected by Wednesday or Thursday. Meanwhile, I am writing/modifying the introduction of the paper.

[timm]

paper due aug 25
you need to do all that you do, and get it into the paper (which will make it longer) by aug15 (then the editing starts...)

by aug15, if possible, would like to see an intro that summarizes your view oin the new results. prior work on fastread incomplete. did not study the effects of prior knowledge. we find prior knowledge useful, but optional. surprisigly good results can be acheived using simple key word frequencies.

now this is ICSE so the intro is all. short sharp, too the point. tells you what was wrong from before, and what is our BIG and IMPORTANT jump. list of RQs (with results). list on contributions. final para on "structure of this paper". all in 1.5 pages.

please acknowledge this comment

[azhe825]

Acknowledged

[timm]

one thing i would say from the above is that RUSE is quirky. BM25 looks all around more reliable

[azhe825]

Auto_Syn is also not a good option since:

• requires domain knowledge to handcraft a fake ``relevant'' study (harder than come up with some keywords for BM25)
• performance is not stable (sometimes good, sometimes really bad)

[timm]

i look forward to reading your conclusions.... by aug15