Comments (12)
50 topics:
data_Hall_Wahono: 0.857870
data_Hall_Abdellatif: 0.754895
data_Abdellatif_Wahono: 0.901016
target_Hall_Wahono: 0.994922
target_Hall_Abdellatif: 0.425869
target_Abdellatif_Wahono: 0.368009
from ml-assisted-slr.
Systematic literature review (SLR) is the primary method for
aggregating and synthesizing evidence in evidence-based software engineering.
Performing an initial SLR on a new area can be very slow and expensive due to the cost of the primary selection study (where researchers review thousands of papers to find the dozens of relevant papers). Further, all that effort has to be repeated if ever that SLR is to be updated.
We find that the effort required to update an initial primary selection study can be significantly reduced via a automatic text mining. For initial primary selections, FASTREAD uses support vector machines to build a model of relevant and irrelevant papers seen to do. By incrementally querying and updating the support vectors of that SVM it is possible to greatly reduce the number of papers that humans have to read (by as much as 97%) during that initial primary selection study.
This paper checks if FASTREAD can be used to reduce the effort required to update initial primary selection studies with a new set of papers. In this work, we assess a {\em naive update policy}; i.e. the new SVM is initialized with the support vectors found in the prior study. Its turns out that this naive update policy does not always work. Rather, we recommend (a)~two different update methods and (b)~deciding which method to use based on the distance of the old selection study to the new study. In the experiments of this paper, we found that the effort associated with these primary selection studies can be reduced by our two methods by up too XXXX.
============
i got confused by all the final charts. which i think are trying to say that
- you can look at the corpus and auto detect similarity and difference.
- which means, in turn, you can select which transfer method to apply
- and i think u are also saying that if you use the opposite of your recommendations that things get sub-optimizal
which sounds like 3 RQs:
-
RQ4: do all transfer methods work equally well on all transfers? (pint3, above)
-
RQ5: is it possible to decide which transfer method should be applied to which data set?
-
RQ6: finally, once our recommended method is applied to the actual data, how much is easier is updating an SLR versus doing one from scratch
Missing
research questions
- can you introduce the idea of similarity intuitively as part of some initial research questions?
related work:
- need to introduce snowballing since that is a term others use
- transfer learning: for a cheats guide to transfer learning, see http://unbox.org/things/var/timm/13/transfer/ (just read sections 2 and 4) user=password=guest. feel free to take from http://unbox.org/things/var/timm/13/transfer/transferlit-v10.tex. note that this paper never got accepted anywhere
future work
- paper 4: impact of mistakes
Why
good that you did not walk thur 16 different options for fast read.
How
never “he” but “they”
from ml-assisted-slr.
- I have not solved the problem of "decide which method to apply by assessing the corpus".
- Why: target similarity is not available before review.
- Now: the suggestion for now is
- to update an SLR, use UPDATE method.
- to start a new SLR, if same topic as previous one, use UPDATE, like Hall to Wahono; if similar topic, use REUSE, like Hall to Abdellatif.
- Found some shortcomings of both UPDATE and REUSE:
- UPDATE: works perfectly on updating SLR with no concept drift. However, even though Hall and Wahono have very high target similarity, using UPDATE tends to retrieve less on the latter one. This effect is more significant as the target similarity goes lower.
- REUSE: not as good as UPDATE in the early stage. Outperforms UPDATE later, because of the effect mentioned earlier.
- possible solution: time decaying model which can have both methods' merits. Motivation: UPDATE is better than REUSE in any scenario if we solve the concept drift problem. Time decaying model throws away older positive examples to tackle concept drift. This can be a journal extension.
from ml-assisted-slr.
Why: target similarity is not available before review.
cant you lda the corpuses and report delta between the topics?
from ml-assisted-slr.
Data similarity is available before review while target similarity is not.
Target similarity is data similarity of only the relevant docs. Before review, we don't know which are relevant.
from ml-assisted-slr.
Data similarity is available before review while target similarity is not. Target similarity is data similarity of only the relevant docs. Before review, we don't know which are relevant.
agreed.what are the data similarities between (say) the first 1000 docs (picked at random) from the queries of your 3 corpuses?
from ml-assisted-slr.
I compared the data similarity of all docs between the 3 corpuses:
Similarity measurement: 30 topics LDA, L2 normalization, cosine distance.
Data similarity
data_Hall_Wahono: 0.860254
data_Hall_Abdellatif: 0.726351
data_Abdellatif_Wahono: 0.809703
Don't think data similarity can accurately reflect target similarity.
from ml-assisted-slr.
why are you doing 30 topics? you doing amrit's DE thing to find stable topics? without that, order effects on input could muddle your findings
from ml-assisted-slr.
Don't want to vary topic number on different corpuses. Otherwise it is not possible to compare the cosine distance (of lda results with different topic numbers). But i will try tuning the alpha and beta of LDA.
from ml-assisted-slr.
been thinking a little about the paper. our main point is that reusing projects from review[i] greatly simplifies review[i+1].
there are some technical choices for review[i+1] that, as yet, we cannot pre-specify which is best.
so lets not confuse current results with future work. can we focus on one of those technical choices as a way to do review[i+1] and mention the other as part of future work?
from ml-assisted-slr.
I am afraid offering only one method will cause the content of this paper to be too few.
Therefore old plan:
-
This paper:
- dealing with two scenarios: update an SLR and initiate a new SLR with similar topic.
- offering two methods, UPDATE and REUSE. For update scenario, choose UPDATE; for the other scenario, UPDATE or REUSE can save effort depending on target similarity (yet need human decision to choose which method).
-
extension (future paper):
- use a single method called time decaying model to replace the two. Works well on either scenario (have some preliminary result on this). Also, this new method has value even in a single large SLR, effective to deal with concept drift problem. Therefore can replace FASTREAD possibly.
New plan?
- this paper: only update SLR scenario discussed, UPDATE method introduced.
- extension: time decaying model for both scenario and concept drift.
from ml-assisted-slr.
Anyway, I can draft a paper discussing UPDATE only and lets see how long it will be.
from ml-assisted-slr.
Related Issues (20)
- Test FASTREAD on Kitchenham data HOT 3
- Estimate Curve
- Plot the estimated curve HOT 3
- Miwa2014
- New experiments for FASTREAD HOT 5
- New Progress on FASTREAD
- Impact of reduced recall on the ‘case study’ review HOT 1
- Try out Abstrackr
- First exp on Syn_Error HOT 2
- Cost for completeness HOT 5
- When to stop HOT 1
- How to start HOT 10
- On the pragmatic design of literature studies in software engineering: an experience-based guideline
- Simple random error results HOT 6
- Random Human Error HOT 1
- Characteristics of Lit review data HOT 1
- javascripts SVMs
- human+machine > human HOT 2
- how to start HOT 9
- Challenges of the problem
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ml-assisted-slr.