Transfer knowledge in SLR

[azhe825]

Transfer knowledge in SLR

[azhe825]

50 topics:
data_Hall_Wahono: 0.857870
data_Hall_Abdellatif: 0.754895
data_Abdellatif_Wahono: 0.901016
target_Hall_Wahono: 0.994922
target_Hall_Abdellatif: 0.425869
target_Abdellatif_Wahono: 0.368009

[timm]

Systematic literature review (SLR) is the primary method for
aggregating and synthesizing evidence in evidence-based software engineering.

Performing an initial SLR on a new area can be very slow and expensive due to the cost of the primary selection study (where researchers review thousands of papers to find the dozens of relevant papers). Further, all that effort has to be repeated if ever that SLR is to be updated.

We find that the effort required to update an initial primary selection study can be significantly reduced via a automatic text mining. For initial primary selections, FASTREAD uses support vector machines to build a model of relevant and irrelevant papers seen to do. By incrementally querying and updating the support vectors of that SVM it is possible to greatly reduce the number of papers that humans have to read (by as much as 97%) during that initial primary selection study.

This paper checks if FASTREAD can be used to reduce the effort required to update initial primary selection studies with a new set of papers. In this work, we assess a {\em naive update policy}; i.e. the new SVM is initialized with the support vectors found in the prior study. Its turns out that this naive update policy does not always work. Rather, we recommend (a)~two different update methods and (b)~deciding which method to use based on the distance of the old selection study to the new study. In the experiments of this paper, we found that the effort associated with these primary selection studies can be reduced by our two methods by up too XXXX.

============

i got confused by all the final charts. which i think are trying to say that

1. you can look at the corpus and auto detect similarity and difference.
2. which means, in turn, you can select which transfer method to apply
3. and i think u are also saying that if you use the opposite of your recommendations that things get sub-optimizal

which sounds like 3 RQs:

• RQ4: do all transfer methods work equally well on all transfers? (pint3, above)

• RQ5: is it possible to decide which transfer method should be applied to which data set?

• RQ6: finally, once our recommended method is applied to the actual data, how much is easier is updating an SLR versus doing one from scratch

Missing

research questions

• can you introduce the idea of similarity intuitively as part of some initial research questions?

related work:

• need to introduce snowballing since that is a term others use
• transfer learning: for a cheats guide to transfer learning, see http://unbox.org/things/var/timm/13/transfer/ (just read sections 2 and 4) user=password=guest. feel free to take from http://unbox.org/things/var/timm/13/transfer/transferlit-v10.tex. note that this paper never got accepted anywhere

future work

• paper 4: impact of mistakes

Why

good that you did not walk thur 16 different options for fast read.

How

never “he” but “they”

[azhe825]

1. I have not solved the problem of "decide which method to apply by assessing the corpus".

• Why: target similarity is not available before review.
• Now: the suggestion for now is
  • to update an SLR, use UPDATE method.
  • to start a new SLR, if same topic as previous one, use UPDATE, like Hall to Wahono; if similar topic, use REUSE, like Hall to Abdellatif.

2. Found some shortcomings of both UPDATE and REUSE:

• UPDATE: works perfectly on updating SLR with no concept drift. However, even though Hall and Wahono have very high target similarity, using UPDATE tends to retrieve less on the latter one. This effect is more significant as the target similarity goes lower.
• REUSE: not as good as UPDATE in the early stage. Outperforms UPDATE later, because of the effect mentioned earlier.
• possible solution: time decaying model which can have both methods' merits. Motivation: UPDATE is better than REUSE in any scenario if we solve the concept drift problem. Time decaying model throws away older positive examples to tackle concept drift. This can be a journal extension.

[timm]

Why: target similarity is not available before review.

cant you lda the corpuses and report delta between the topics?

[azhe825]

Data similarity is available before review while target similarity is not.

Target similarity is data similarity of only the relevant docs. Before review, we don't know which are relevant.

[timm]

Data similarity is available before review while target similarity is not. Target similarity is data similarity of only the relevant docs. Before review, we don't know which are relevant.

agreed.what are the data similarities between (say) the first 1000 docs (picked at random) from the queries of your 3 corpuses?

[azhe825]

I compared the data similarity of all docs between the 3 corpuses:

Similarity measurement: 30 topics LDA, L2 normalization, cosine distance.

Data similarity

data_Hall_Wahono: 0.860254
data_Hall_Abdellatif: 0.726351
data_Abdellatif_Wahono: 0.809703

Don't think data similarity can accurately reflect target similarity.

[timm]

why are you doing 30 topics? you doing amrit's DE thing to find stable topics? without that, order effects on input could muddle your findings

[azhe825]

Don't want to vary topic number on different corpuses. Otherwise it is not possible to compare the cosine distance (of lda results with different topic numbers). But i will try tuning the alpha and beta of LDA.

[timm]

been thinking a little about the paper. our main point is that reusing projects from review[i] greatly simplifies review[i+1].

there are some technical choices for review[i+1] that, as yet, we cannot pre-specify which is best.

so lets not confuse current results with future work. can we focus on one of those technical choices as a way to do review[i+1] and mention the other as part of future work?

[azhe825]

I am afraid offering only one method will cause the content of this paper to be too few.

Therefore old plan:

• This paper:

  • dealing with two scenarios: update an SLR and initiate a new SLR with similar topic.
  • offering two methods, UPDATE and REUSE. For update scenario, choose UPDATE; for the other scenario, UPDATE or REUSE can save effort depending on target similarity (yet need human decision to choose which method).

• extension (future paper):

  • use a single method called time decaying model to replace the two. Works well on either scenario (have some preliminary result on this). Also, this new method has value even in a single large SLR, effective to deal with concept drift problem. Therefore can replace FASTREAD possibly.

New plan?

• this paper: only update SLR scenario discussed, UPDATE method introduced.
• extension: time decaying model for both scenario and concept drift.

[azhe825]

Anyway, I can draft a paper discussing UPDATE only and lets see how long it will be.