Comments (10)
From my small-scale experiments, BM25 is really a good strategy for all scenarios. It requires no previous data, little domain knowledge (knowing the keywords to search for).
huh. i wonder how much it changes the prior results?
from ml-assisted-slr.
@timm
Updated some results. BM25 really works well in most cases.
How do you interpret the results?
from ml-assisted-slr.
is this the reverse of our prior conclusion? that we dont need prior knowledge to do a good job of lit review?
from ml-assisted-slr.
and, regardless of that, how does this play with the stopping results?
from ml-assisted-slr.
is this the reverse of our prior conclusion? that we dont need prior knowledge to do a good job of lit review?
Kind of. UPDATE performs quite good when it can be applied.
- For practical use of the tool, I would suggest BM25:
- it is reliably better than RANDOM
- can be applied to any scenario
- may also be useful when the review goes into a corner and the reviewer has a better view of which direction to go (search when the learner fails to find more relevant studies)
- For paper writing, not sure what to do, mainly because a) BM25 is an existing technique suggested by Cormack. b) BM25 is not consistently the best treatment (there is no such treatment).
regardless of that, how does this play with the stopping results?
Still running. Results are expected by Wednesday or Thursday. Meanwhile, I am writing/modifying the introduction of the paper.
from ml-assisted-slr.
paper due aug 25
you need to do all that you do, and get it into the paper (which will make it longer) by aug15 (then the editing starts...)
by aug15, if possible, would like to see an intro that summarizes your view oin the new results. prior work on fastread incomplete. did not study the effects of prior knowledge. we find prior knowledge useful, but optional. surprisigly good results can be acheived using simple key word frequencies.
now this is ICSE so the intro is all. short sharp, too the point. tells you what was wrong from before, and what is our BIG and IMPORTANT jump. list of RQs (with results). list on contributions. final para on "structure of this paper". all in 1.5 pages.
please acknowledge this comment
from ml-assisted-slr.
Acknowledged
from ml-assisted-slr.
one thing i would say from the above is that RUSE is quirky. BM25 looks all around more reliable
from ml-assisted-slr.
Auto_Syn is also not a good option since:
- requires domain knowledge to handcraft a fake ``relevant'' study (harder than come up with some keywords for BM25)
- performance is not stable (sometimes good, sometimes really bad)
from ml-assisted-slr.
i look forward to reading your conclusions.... by aug15
from ml-assisted-slr.
Related Issues (20)
- Test FASTREAD on Kitchenham data HOT 3
- Estimate Curve
- Plot the estimated curve HOT 3
- Miwa2014
- New experiments for FASTREAD HOT 5
- New Progress on FASTREAD
- Impact of reduced recall on the ‘case study’ review HOT 1
- Try out Abstrackr
- First exp on Syn_Error HOT 2
- Cost for completeness HOT 5
- When to stop HOT 1
- On the pragmatic design of literature studies in software engineering: an experience-based guideline
- Simple random error results HOT 6
- Random Human Error HOT 1
- Characteristics of Lit review data HOT 1
- javascripts SVMs
- human+machine > human HOT 2
- how to start HOT 9
- Challenges of the problem
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ml-assisted-slr.