does phase delay increase bug costs?
ai-se / phasedelay Goto Github PK
View Code? Open in Web Editor NEWdoes phase delay increase bug costs?
does phase delay increase bug costs?
reviewer2:
reviewer1:
I do not understand how section 5 is structured. I would make three separate subsections, i.e. I would number 5.2.1 as 5.2 and 5.2.2 as 5.3. Moreover, for consistency, I would include H1 in the title of Subsection 5.1.
In 5.2.2 the numbers reported are not consistent with Figure 10, please check. Also, the caption refers to Left but I think you mean Right.
I am troubled by Fig. 12, how many defects have in total been fixed in the same phase in which they have been found? What is the proportion from the initial 47,376 defect logs that are concerned in the statistics in Fig. 13?
Reveiwer1:
Reviewer3:
Section 5.2.3: There is a minor discrepancy between Figure 7 and the text. In the running text you list system tests before acceptance tests, but the opposite order is depicted in the figure. Make sure the order is correct, also in Figure 13. To me acceptance testing normally happens as the final V&V step before release. Did all 171 projects really do system testing last? Maybe this is prescribed in TSP?
"The first thing to note in Figure 8, is that this data does not follow
the exponential increase feared by Beck’s Figure 1"
Figure 8 does show increase but it does not even show the cost of defect:
it shows the numbers of defects?!
DEAR AUTHORS-- THIS TEXT NOW MOVED TO PAPER
Dear authors,
Three knowledgable reviewers have scrutinized the manuscript
and found it interesting and relevant, although in need for
certain improvements. Therefore, the authors are
invited to submit a revised version for a new review, where
the reviewers' comments
are adressed. Particularly, this holds for:
- the connection to activities of theory building in ESE
(reviewers 1).
This new draft now maps our work into the SE literature on theory building.
- clarification about the data origin and
characteristics (reviewer 1).
This new draft now maps not contains extensive notes on the origin and characteristics of our data.
- use of statistical analysis (reviewers 2 and 3)
All our results are now augmented with statistical significance tests as well as effect size tests. Those tests greatly strength the overall message of this paper.
- writing style, that would benefit form being more
precise (reviewer 3)
Reviewer #1: PAPER SUMMARY:
One of the most widespread beliefs in software engineering
is that the sooner you identify and resolve an issue the
less effort it requires (the authors call it DIE). This is
also being taught in software engineering education; fix
something in the r equirements phase and it is a quick fix,
wait until testing and you will have a to debug source code,
modify design, update documentation, and then change the
requirement - a much more complex task. This rule-of-thumb
was expressed in the 80s, and much has changed in the
software business since, e.g., programming languages,
editors, agile development and stack overflow. To what
extent does DIE actually exist in modern software
engineering projects?
The authors investigate the DIE phenomenon quantitatively by
analyzing data from 171 completed projects. All included
projects used Team Software Process (TSP), a development
methodology developed by the Software Engineering Institute
at Carnegie Mellon University. A central component in TSP is
detailed time logging, i.e., a TSP developer reports how
much time he spends on various tasks including issue
resolution. Thanks to access to a large dataset of completed
TSP-driven projects, the authors have been able to analyze
effort spent on issues in different development phases.
The results show that DIE does not apply to all projects,
and the effect is not as strong as previous publications
suggest. The authors thus conclude that DIE cannot be
assumed to always hold. Furthermore, they present five
explanations of the observed lack-of-effect: i) DIE is a
historic relic, ii) DIE is intermittent, iii) DIE applies
only to very large systems, iv) DIE has been mitigated by
modern development methodologies, and v) DIE has been
mitigated by modern development tools. Finally, the authors
state the relevant question: how could DIE become such a
widespread idea in the literature, despite limited empirical
evidence?
REVIEW SUMMARY: + The paper investigates whether a widely
accepted "rule" in software engineering holds by looking at
a large set of empirical data. The research topic is highly
relevant and the results are interesting. + The discussion
on reassessing old truism fits the scope of the journal very
well. + The language in the paper is excellent, and the
structure is (although non-standard) easy to follow.
- It is not easy for the reader to understand what parts
of the "life of an issue" the authors claim are affected by
DIE, and it is also not too easy to understand what data the
authors have available. I believe adding another figure
could address both these aspects. ```
- The different projects studied are all grouped together. I
believe studying clusters of development contexts would be
highly interesting, but unfortunately the current
characterization of projects is inadequate.
- The authors do not relate their reassessment of an
"old truism" to the growing set of papers on theory building
in software engineering. With access to such large amounts
of empirical evidence, it should be possible to take some
steps toward improved DIE theories.
[ ] @fshull need words on this
DETAILED REVIEW: The authors should expand on what they
include in the "delayed issue effect" (DIE), in particular
in relation to standard milestones in the life of an issue
and the corresponding timespans. I would suggest adding a
figure to point out important issue milestones on a
timeline, e.g., 1) injected, 2) found/reported, 3) work on
resolution started, 4) work on resolution ended, 5) V&V
completed, 6) resolution deployed/integrated. I believe the
authors refer to an increase in time between 3) and 4) as
the DIE, but I'm not really sure. I'm particularly
interested in whether all "indirect" issue resolution steps
are covered, e.g., change impact analysis, updating
documentation, and regression testing. The effort involved
in these steps clearly depends on the development context of
the specific project. The short a-c) list on page 14:50
suggest that the issues studied are restricted to minor
fixes, i.e., no updates to architecture, changed hardware,
recertification, updated user manuals etc.
A figure with issue milestones could also help the reader
understand what data are actually available for the
analysis. Section 6.3 describes the data, but some aspects
should be further explained. Logging interruption time
sounds very useful, but I wonder how carefully the
developers actually did this -
it must be really hard to keep up the discipline required.
If you are interrupted as a developer (e.g., phone calls,
urgent mail, someone asks a question) I don't think the
first thing you do is to stop a timer on your computer.
Moreover, developers often work on several defects in
parallel, and might interweave bug fixing with new
development. I don't think the "interruption time" captures
all such multi-tasking, and it should at least be properly
discussed in Section 7 "Threats to validity".
The discussion on reassessing old truisms (Section 3) is
interesting, but it should be complemented by the
perspective of theory building in software engineering - An
active research topic lately, with a dedicated workshop
series (GTSE). I suggest looking into the following
references for a start: Sjøberg et al. (2008) "Building
Theories in Software Engineering", Smolander and Päivärinta
(2013) "Theorizing about software development practices",
and Stol and Fitzgerald (2015) "Theory-oriented software
engineering". Considering new empirical evidence is
obviously critical to theory building, and discussing your
new results in the light of an theory creation would be
valuable. The authors mention that DIE appears to occur
intermittently in certain kinds of projects - maybe the
authors could elaborate on this idea and present an improved
DIE theory based on what they now know? I believe the
authors have the best available data to do so, and I would
expect the paper to go beyond simply questioning the "old
truth".
The authors study three claims in this paper, and the
third claim is the central one: "delayed issues are not
harder to resolve". To study whether issues require longer
resolution times in later phases, the authors analyze a
large set of issue data from historical projects. Thus the
manuscript reports from an observational study rather than
an experiment with a controlled delivery of treatments.
While I believe the authors' approach is practical, I would
like to see a critical discussion on threats to validity of
observational studies (e.g., Madigan et al., A Systematic
Statistical Approach to Evaluating Evidence from
Observational Studies, Annu. Rev. Stat. Appl. 2014. 1:11-39
and Carlson and Morison, Study Design, Precision, and
Validity in Observational Studies, J Palliat Med. 2009 Jan;
12(1): 77-82). An alternative study design (although
difficult to realize in a system of industrial size) would
be to let different developers resolve the same issues for
the same software system, i.e., one group resolves an issue
during design, another during implementation, and a third
during testing - some discussion along these lines would
strengthen the validity section.
The first and second claims are studied with much less
rigor. "DIE is a commonly held belief" is studied using a
survey of software engineers, both practitioners and
researchers. According to Fig. 2 (actually a table), the
number of respondents is 16 and 30 for practitioners and
researchers, respectively. Sixteen respondents from industry
represent a tiny survey of a very general claim that any
software engineer could respond to. Why were not more
answers collected? The authors do not have much evidence to
defend the first claim, and there is no discussion of the
corresponding validity threats. The second claim, "DIE is
poorly documented", is studied using a literature review.
Unfortunately, the method underlying the literature review
is not presented. Although it doesn't need to be an SLR of
the most rigorous kind, the authors should report how the
papers were identified. I suspect the terminology used to
describe the DIE phenomenon is highly diverse, thus it would
strengthen the paper if the authors reported how they
reviewed the literature. According to Page 19:34 only eight
publications were identified.
The iterative fashion of modern software development,
with agile at the extreme end, is not fully discussed (the
short discussion section could be extended). The phases of
the linear development of the 80s, (such as in Fig. 1)
probably don't exist in many of the projects in the TSP
dataset, still the authors discuss the DIE-effect from the
perspective of the 80s. Page 16:31 states "DIE has been used
to justify many changes to early lifecycle software
engineering" - does this mean the agile movement
successfully mitigated DIE? This possibility is not fully
considered in the paper.
How many of the 171 projects were
(more or less) agile? This appears to be an important
characteristic of the included projects - very important to
describe!
Concerning characterization of the 171 projects, the
paper needs to report much more detail. I would expect to
see some descriptive statistics. On page 17:18 the authors
say "perhaps we should split our SE principles into two sets
/---/" - of course SE practices need to be adapted to the
development context, and also two sets of principles is a
too simple split. The paper does not report much
characterization of the 171 projects. I strongly suggest the
authors to dig deeper into the data, and analyzing for which
types of projects the findings hold. What patterns are there
to discover? I suggest Petersen and Wohlin, "Context in
industrial software engineering research" (ESEM2009) for
details on how to characterize development contexts.
Moreover, I would really like to see what practitioners from
the 171 projects think of your findings - an interesting
option for a qualitative angle on your study.
Fig 1: "Operation" dwarfs everything in Boehm's diagram.
Since you do not study anything post release in this paper,
I think this figure skews the reader's mental picture. If
you remove the rightmost bar and rescale accordingly, the
plot better matches the findings you report in Fig. 10. You
still identify an interesting result though, but the
presentation turns fairer.
MINOR COMMENTS: Keywords: I believe the keywords could be
improved to help readers find this work.
Several figures are copied from previous work. Are all copyrights properly
managed?
Fig 10: Black and red is not a good pair for
grayscale printouts. Please use black and gray instead.
- [ ] @timm
Some figures are actually tables, thus their captions should
be replaced accordingly.
- [ ] @llayman
Some figures should be resized to better match the page width.
- [ ] @llayman
Page 3:14 - "The above argument" Which argument? Could be precisely specified.
- [X]
Page 3:22 - Please provide the full link to the dataset.
- [ ] @timm
Page 4:8 - "More difficult" but "harder" is used in claim3. Why this inconsistency?
- [ ]
Page 18:26 - First sentence: "Unexpected results such as this one". Ambiguous reference,
please be specific.
- [ ]
Page 18:27 - "We also survey the state of SW dev. practice /---/": given the size of the survey,
this statement feels a bit bold.
- [ ]
TYPOS ETC.
Sec 1, §1: [3] and [30] have been swapped?
Page 14:46 - Spell out IV&V the first (and only) time it appears.
Page 3:24 - missing verb (is) Page 16:43 - "Did this study failed"
Page 18:44 - Appache
Page 19:19 - Al Ref 26 - oo --> OO
- [X]
_____________________________________________
Reviewer #2: The topic of this paper is the empirical
evaluation of a largely admitted claim in software
engineering, i.e. the exponential cost of correcting errors
according to the phase in which the errors are discovered.
The claim is first confirmed by surveying and interviewing
practitioners and experts. Next, the authors use a large set
of data from projects in SEI database. The results show
that, strictly speaking, the claim does not hold (although
some effects of delayed corrections can be noticed).
Globally, the paper is well written and the authors develop
a convincing demonstration. The topic of the paper is highly
relevant for both practitioners and people from academia.
Among many, I personally believed in this claim and had
regularly taught it in my courses. Thus, reading this paper
offers a refreshing perspective on our understandings of
software engineering background and theoretical knowledge. I
also like the suggested posture that insists on the
necessary skepticism that we should have against such
prevailing claims.
Beyond this positive global impression, I have some concerns
about the study reporting and analysis: - It seems to me
that more descriptive statistics about the sample would
contribute to a better understanding of the scope of the
study.
- [ ] @timm more stats
I think in particular about the size (lines of codes and/or
number of software components); a duration histogram would
also be relevant. Accordingly, the formula for calculating
variable "Total effort" in Fig. 7 could be explained, and
Figure 9 made more readable (bigger size).
- [X] @llayman We have added more description of the projects to Section 6.4. We have also added more precise definitions of our measures in Section 6.2. We have adjusted the size of figures throughout the paper to be more readable.
If certain attributes for the issues and errors are
available in the data (e.g. severity, priority, etc.), they
should also be brought to the readers knowledge.
- [ ] @WilliamNichols: got any stats on these
Figure 10 is central to the paper's demonstration; its
expressiveness could be enhanced: i) the reader tends to
think that the "right hand side bars" are different from the
BLACK and RED bars, ii) the formula for calculating the 50th
and 95 ratio could be provided, iii) the unit of column
"Percentile" could be mentioned.
- [ ] @timm split into two
I like the discussion section, in particular, the idea
of software architectures that could be a contributing
factor for enhancing software evolution and reducing the
cost of issues and errors fixing. Is there any available
knowledge in the data set concerning any architectural
design choices made in each project? If so, would it make
sense to seek any correlation between these architectural
choices and how expensive it was to solve issues?
- [ ] @WilliamNichols: anything on architectural style? or should
we jsut say "many and varied and hard to get a precise
picture of it all
In the same trend of ideas, I think what has
fundamentally changed since early days of SE is the
development of requirements engineering. In actual SE
practices, problem and the solution spaces are
systematically explored thanks to RE techniques; together
with architecture, this could also explain why things have
changed since the 70'. It could be, for example, that people
tend to make less severe errors in early software project
phases (thanks to RE techniques); a similar phenomenon was
observed when SE practices become more mature (see Harter et
al. 2012).
References Harter D. E., Kemerer C. F., Slaughter S. (2012).
Does Software Process Improvement Reduce the Severity of
Defects? A Longitudinal Field Study. IEEE Transactions on
Software Engineering, vol. 38, n° 4, p. 810-827.
- [ ] @timm reference harter
- [X] @llayman Absolutely - there are potentially many explanations for why the DIE was NOT observed in our dataset. We discuss some more of the potential causes in Section 8.
In the conclusion section, I feel uncomfortable with
the assertion "That data held no trace of the delayed issued
effect" (line 17). As mentioned elsewhere in the paper, the
delayed issue is not absent; it is much less significant and
systematic than what is usually claimed. Moreover, this
reduction has been demonstrated for medium sized projects;
we cannot generalize to larger projects.
- [ ] @timm
Minor remarks: - p.9, line 29 : " … reported by Shull [52], found that the cost to find certain non-critical classes of defects …" => is it the "cost to find" or the "cost to FIX"?
- [X] @llayman It should be "cost to fix". We have corrected this.
- [X]
_______________________
Reviewer #3: I like the idea of the paper and I certainly
would like to see more empirical studies that periodically
check if our beliefs about software engineering practices
still hold (or even if they have ever held). The authors
address one of the beliefs that are more entrenched in the
population of researchers and practitioners, and a very
important one too.
At the same time, I think that the paper needs to be
improved before it can be published.
The empirical analysis (Section 6) is a bit of a let down. I
would like to see some sort of more robust and rigorous
statistical analysis, based on statistical tests. Instead,
the paper only provides qualitative comparisons between the
results obtained at the 50-th and 95-percentiles. This is
not to say that the results or the discussions are
incorrect, but only that they need to be better supported.
The lack of this kind of statistical analysis should have
least be mentioned in the Threats to Validity. I understand
that you say that your "claims" (including Claim 3) should
not be considered "hypotheses" in the statistical sense and
I agree with you on Claim 1 and Claim 2. However, Claim 3
lends itself to a statistical analysis, without which your
results become a bit more anecdotal, and this is what you
rightly criticize in the previous claims that DIE actually
exists. In addition, you may want to analyze some additional
and somewhat unexpected results that can be derived from
your data (see Detail Comments below).
- [ ] do the stats
The paper seems to be written in a somewhat casual style,
which makes the paper more pleasant to read, but less clear
in some parts (see Detail Comments below).
DETAIL COMMENTS
Section 2
I'm not really sure what the authors mean by "We say that a
measure collected in phase 1, ., i, .. j is very much more
when that measure at phase j is larger than the sum of that
measure in earlier phases 1 <= i < j." You are defining a
property of a measure, but, the way this definition is
written, it seems as if you are defining the property of a
measure of "being much more," without any further
qualifications. I can guess that you mean that a measure
value of
(so, it's a property m has in a specific phase and not a
general property of the measure) if $\sum_{1 <= i < i} m(i)
< m(j)$. Then, it's up to you to make this a property of the
measure, for example by using an existential "policy" (
has this property if there exists one phase
$\sum_{1 <= i < i} m(i) < m(j)$) or a universal one (
this property if for all phases
m(i) < m(j)$).
- [ ] @timm just do that
The real problem with this definition, however, is that it
doesn't seem to be used anywhere in the paper. The only
point where it might be used is in the discussion of Figure
1 (which precedes it anyway). So, you may want to remove
this property altogether.
- [ ] @timm your right. the real thing here is that here is
statistically singicicat increase
Your definition of "difficult issue" is "Issues are more
difficult when their resolution takes more time or costs
more (e.g. needs expensive debugging tools or the skills of
expensive developers)." That's not a very precise
definition, though, because it has two different
interpretations, one in terms of time and the other in terms
of effort. The reasons why you introduce this definition
become clearer only in Section 7.2, in the Threats to
Validity, and that's too late. You should move some of the
discussion of Section 7.2 here.
- [ ] @timm roger
Why would the term "delayed issue effect" be a
generalization of the rule "requirements errors are hardest
to fix"? "Delayed" seems to be quite specific as it appears
to refer to time, while "hardest to fix" may refer to other
variables, like effort or cost.
- [ ] @timm text
In Claim 3, "very much more harder" sounds like a
little bit too much ... ("more harder"?)
- [ ]
Section 4
A very minor issue: why is Figure 2 a ... figure, instead of
a table? Same for all other tables ...
- [ ]
Section 5.1
This is not a complete sentence "All the literature
described above that reports the onset of DIE prior to
delivery."
- [X]
Section 6.3
You should rephrase your definition "a defect is any change," since a defect is what existed before the change was made and not the change itself.
- [X] @llayman We now provide a more precise definition of defect in Section 6.2.
You do not explain that "QualTest" in Fig. 8 is.
- [ ]
The definition of "time per defect - The total # of defects found in a plan item during a removal phase divided by the total time spent on that plan item in that phase." is not correct as it is, as this would basically be the number of defects per unit time, instead. The roles of time and
defects should be reversed.
- [X] We now provide a more precise definition of time measurement related to defects in Section 6.2.
Section 6.4
Besides adding a more rigorous statistical analysis, you
need improve your explanations.
Figure 9. The caption says "Distribution of defects found
and fixed by phase." Is that the distribution of defects
found in some phase and fixed in that phase? The text says
"The distribution of defects found and fixed per phase in
our data is shown in Figure 9. A high percentage of defects
(44%) were found and fixed in the early phases, i.e.,
requirements, high level design, and design reviews and
inspections." which doesn't add much. If that's the
distribution of defects introduced in a phase and fixed
immediately in that phase, it is not clear why you introduce
it, though.
- [ ]
Figure 10. The explanation of the data and histograms in the
figure is, at best, confusing. The caption says "50th and
95th percentiles of issue resolution times" and the opening
sentence of Section 6.4 says "Figure 10 shows the 50th and
95th percentile of the time spent resolving issues ..."
However, these are not "times," because you write "expressed
as ratios of resolution time in the phase where they were
first injected" a couple of paragraphs below. This seems to
be the right interpretation, but right after that you write
"The BLACK and RED bars show the increases for the 50th
(median) and 95th percentile values, respectively," which
would be a third interpretation of the data in Figure 10.
- [ ]
You should provide the value of the mean, in addition to the
50-th percentile (i.e., the median).
- [ ]
You need to at least mention (or, better, discuss) some of
your results, because they appear to be "counterintuitive,"
since they show that, for example, median resolution times
can even decrease if issues are not fixed immediately, but
in later phases. For example, looking at the Reqts section,
most percentages are way below 1, which seems to indicate
that some issues are actually much easier to fix at later
stages (maybe because more or better information about the
software system is available only later). That would be a
very interesting result by itself, especially if you can
provide some statistical support for it.
- [ ] @timm really, no statisticall difference
Section 7.1
Maybe you could analyze the projects developed in a
"traditional" way and check if there is some sort of DIE
there. For example, if you look at the projects developed
with the waterfall life cycle, maybe you will find larger
DIE than for the other life cycles, which might partially
justify the claims of previous researchers about the
existence of DIE. This would also show that newer kinds of
life cycles (like Agile ones) help solve DIE. You do provide
hints about this in Section 8 ("We also note that other
development practices have changed in ways that could
mitigate the delayed issued effect") and Section 9, but
that's clearly not enough.
- [ ] @williamNichols: can we separate out the agile and the trad?
Section 8
I'm not sure what you are really trying to discuss in this
section, because you are not discussing the results of the
paper. It sounds like you are saying that maybe it's the
newer software development approaches that made DIE
disappear?
- [ ]
- [ ] Verify that the Section #'s in our responses are accurate prior to submission.
Bill, this is really the same problem as #15.
In Figure 10 - why is the histogram for Code Review in the Code phase not 10 *'s like all the other's? Also the histograms - since they are scaled by phase - would seem to not be comparable across phases but they are visually presented in a way that they look like they are comparable. I'm not sure why you scaled them.
- The statistical analysis section (§5.5) can be enhanced by better explicating the calculations. The Scott-Knott algorithm is a particular clustering algorithm, and the reader needs some clarification how it contributes to the demonstration (this can be related to sections §6.1 and §6.3 about threats to validity).
Moreover, Fig.13 is still insufficiently clear. The authors added a lengthy legend; I tend to think this would better be placed in the text. The column on the left hand (i.e. "rank") is still unclear to me. I also wonder if the size of the clusters should not be reported, i.e. how many defects are considered when 50th percentile growth is calculated? Does the calculation hold even if the clusters are minuscule in size? This aspect doesn't seem to be mentioned in §6.
Hey Bill,
is there any issue here:
Is each such cycle its own before/planning/req/des/code/test thing? or, do folks go agile spinning between (say) req/des or (say) res/code?
please advise
t
one sentence i am writing needs some comment on the overhead of TSP. ideally, the overhead is negligible (after some initial training... how much training? how much cost for an added mentor like the distinguished Dr Nochols? is that mentor needed?)
Reviewer3:
Reviewer 1:
Despite the amount of the paper that describes PSP and TSP and lifecycles - I don't know what happens in the Before and Planning phases. I don't know what this could be that isn't what is traditionally considered the Requirements phase. Without this - I don't see how you can compare your results with other studies that would presumably consider these things as part of the requirements phase. Meanwhile, a lot of the Project Phases section seems irrelevant to understanding the data.
I don't fully understand Section 5.2.2 paragraph 4, and the relation between plan items and the time tracking logs. "One or more defects are reported against a single plan item, e.g., a review session, an inspection meeting, a test execution". Multiple defects can be mapped to a plan item? But a plan item can also be "resolving a defect" (page 11:46). Also, the time tracking logs include time to collect data, prepare a fix, and its validation (page 12:50)… But what if a plan item includes both running a test suite and resolving the identified defect? The equation presented on page 13 suggests that the "time-to-fix a defect" could be inflated by e.g. running slow test cases. I'm sure the authors have all this covered, but the text confuses me. Could it possibly be revised? Maybe another figure could clarify the situation?
Biil: any comment on how we know developers found the right root cause phase of a defect
(Actually, I don't even get this comment... how does anyone find the origin phase acurrately? boehm? anyone? why are we being pinged on this issue?)
Ignore this: support comment
Please check throughout your references, some include extended first name of authors, other only the initial.
The description of the Scott-Knott ranker is not terribly clear. You write "Scott-Knott seeks the division of of l treatments into subsets of size m, n of sizes ls,ms, ns and median values l.μ,m.μ, n.μ (respectively) in order to maximize ms/ls abs(m.μ − l.μ)^2 + ns/ls abs(n.μ − l.μ)^2" but it's not clear to me what "subsets of size m, n of sizes ls,ms, ns and median values l.μ,m.μ, n.μ (respectively)" means. Are these 5 subsets? How are they related?
adas
In Section 4.2 you say “we examined 171 software projects conducted between 2005 and 2014.”, but in the rest of paper you mentioned 2006.
In some parts I found the tone of text a little assumptive. For example “Lest the reader find this too little to justify our conclusions, we note that …”: trust the ESEC FSE readers to be able to make their own conclusions. Later on: “the phase delay myth persists. Hence, this paper. “ let readers judge whether the paper convince them or not. Please revise and lighten tones throughout.
Bill: you got any comments on this:
ignore this bit, just support text:
Figure 11 - I like this figure, especially the final distribution column! But please clarify the year… I'm sure many projects are not completed within a calendar year. Does the year show when the project started or finished (released)?
Section 6.3 paragraph 5 - "Figure 14 shows that no such effect occurs…" How? Severity is not presented in the figure. Moreover, did your stratification cover high-severity defects?
The argument referenced is from Tim -- I confess to never understanding it myself. Tim or maybe Forrest needs to rework to justify why we don't think severity distribution matters.
Weaknesses
I would have also liked to see more discussion on the evolution of SE discipline and how novel paradigms may have impacted on the truisms here challenged. The fact that the studied project adopted TSP, and this could be a reason of the results observed is quickly mentioned among the threats to validity. I think this is an important feature of this study that should be taken into account while making claims along the paper.
Also, one would assume that the TSP process was developed while
keeping the Phase Delay effect in mind. That might be one cause why
the effect is not predominantly seen in the projects studied, that
point is not mentioned in the paper anywhere. One independent metric
would have been to study some loosely developed projects and see if
the effect is seen. The authors themselves pointed out that "The
time in final testing is particularly low suggesting that few
defects survived into testing". So, no major defect might have
survived long enough in the system.
From the first comment of Reviewer #2
Fig. 11 could be complemented with the number of defects distribution.
Consider adding this, but I would say it is optional. I have ignored their other suggestions from this comment because why remove descriptive information?
@llayman : our plot is time to fix, there data is cost to fix. is this even repairable or should we dump this figure?
The characteristics of the projects are given in duration and
number of people: no indication of size.
- Finally, looking at Fig.6 and defect type distribution, wouldn't be relevant to explore any possible effect of defect type on the results in Fig. 13? As defect severity is absent, may be defect type has some influence. It could be that certain categories of defect can be corrected any time during the project, it could even be that these defects, in particular, are delayed for later (because they are easy to repair) … It is indeed a mystery why certain defects have been fixed immediately (as reported in Fig. 12, how many in total?) and others were delayed for later (as analyzed in Fig. 13, how many in each cluster and in total?).
Bill, can you perhaps offer some response to this?
Figure 12 - Found and fixed? Does this mean each defect appears twice in the figure? Could you separate them? The current figure is hard to penetrate.
The reviewers have carefully scrutinized the revised version, and all of the found the new version being globally improved with respect to the major concerns, and thus found the manuscript being worth publishing. At the same time, changes have introduced some inconsistencies and minor issues (such as a figure caption being too extensive) that the authors should fix them before submitting the final version.
Reviewer #1: Thank you for carefully revising the manuscript. All my comments have been addressed - either elaborated in the paper or explained in the rejoinder. The manuscript is mature and my remaining comments are all minor. Unfortunately, it is not easy to refer to specific comments in the rejoinder, but I do my best below.
I think the manuscript has improved considerably, especially in regard to three key aspects. First, the data and the overall development context has greatly improved. This will help future readers understand the results and allow further interpretation. Second, the paper now connects to previous research on theory building in software engineering - already in the introduction section. While no new theories are put forward in this paper, I think the discussion is important, and the pointers to future work could inspire more work. Third, I appreciate the authors' efforts to stratify the data. Although there are no new findings, I believe the discussion belongs in the paper - I missed it in the previous version.
R1.1: Note that there is no Section 6.6 in the manuscript though, make sure you didn't forget anything.
Minor comments:
R1.2Section 4: I think the paragraphing could be improved - the highly interesting point about agile is hidden. "A goal of agile methods is to reduce /---/ little empirical data exist" appears in the middle of a paragraph headed by "Shull et al. conducted a literature survey and held a series of e-workshops". Consider restructuring this section to bring forward the agile perspective.
R1.3: I don't fully understand Section 5.2.2 paragraph 4, and the relation between plan items and the time tracking logs. "One or more defects are reported against a single plan item, e.g., a review session, an inspection meeting, a test execution". Multiple defects can be mapped to a plan item? But a plan item can also be "resolving a defect" (page 11:46). Also, the time tracking logs include time to collect data, prepare a fix, and its validation (page 12:50)… But what if a plan item includes both running a test suite and resolving the identified defect? The equation presented on page 13 suggests that the "time-to-fix a defect" could be inflated by e.g. running slow test cases. I'm sure the authors have all this covered, but the text confuses me. Could it possibly be revised? Maybe another figure could clarify the situation?
Reply1.3: We apologize for the confusion. We have rewritten this paragraph to be more understandable. The key point is that we measure defects both directly and by summing effort in defect removal phases. The total removal phases produce a greater cost estimate by including overhead.
R1.4: Section 5.2.3: There is a minor discrepancy between Figure 7 and the text. In the running text you list system tests before acceptance tests, but the opposite order is depicted in the figure. Make sure the order is correct, also in Figure 13.
Reply1.4: Our bad. System comes before acceptance. Changed throughout. Good catch!
R1.5: Figure 10 - I like this figure, especially the final distribution column! But please clarify the year… I'm sure many projects are not completed within a calendar year. Does the year show when the project started or finished (released)?
Reply 1.5: Thank you for catching this ambiguity and for the positive comment on the distributions. We reported the start year and have modified the figure to clarify.
R1.6: Figure 12 - Found and fixed? Does this mean each defect appears twice in the figure? Could you separate them? The current figure is hard to penetrate.
Reply 1.6: Each defect is counted only once. The term "found and fixed" has a specific meaning, but this meaning has not come through clearly to more than one reviewer. To TSP practitioners, "found and fixed" means that the required change was "identified and implemented". We modified the text to clarify this that this operational term is a single activity.
R1.7 Figure 13 - The caption is far too long. Could parts of it be moved to a paragraph in the running text?
Reply1.7: Moved into text
R1.8: Section 6.1 - It is not obvious to me how to interpret "conformance with Benford distribution". Could you please elaborate somewhat?
Reply 1.8: A Benford test is used in forensic accounting to detect human manipulation of data entries. The distribution is not quite, but vaguely log-normal and occurs naturally in frequency distributions, such as log tables and data entries. Human alterations (guesses) cause deviations from the expected distribution. We have applied this test to log entries to estimate those recorded in real time vs those estimated or guessed. We have re-written the paragraph and included a citation to clarify the meaning.
R1.9: Section 6.1 paragraph 7 - "Fourth" appears twice.
Reply 1.9: Fixed
R1.10: Section 6.3 paragraph 5 - "Figure 14 shows that no such effect occurs…" How? Severity is not presented in the figure. Moreover, did your stratification cover high-severity defects?
Reply 1.10: You are correct. That is an incorrect inference. We have deleted that sentence.
R1.11:
Details:
Page 24:12 - "the cased study"
Page 7:24 - "Some studies that report less-than" Remove extra "that".
Page 12:41 - "quote elaborate" quite?
Page 14:30 - "the of the"
Page 14:45 - "complete submitted"
Page 15-16 - No thousands separators.
Page 16:48 - "only 25% of the teams…" Was this supposed to be a separate item in the bullet list?
Page 18:51 - "raises" -> "raised"
Reply 1.ll: Fixed. Thanks!
Reviewer #2: This revised version (R1) of the paper contains many corrections, changes, and enhancements. The authors have considered all the reviewer's comments and have provided precise responses. My own observations (i.e. Reviewer #2) have been globally satisfied, and thanks to comments from reviewers #1 and #3, the paper has been significantly enhanced.
Thank you for that comment.
Nonetheless, there are still some issues that need some clarifications.
R2.1 My main concerns are related to data and statistics sections:
Reply 2.1: We have added a defect count distribution as suggested.
R2.2- The statistical analysis section (§5.5) can be enhanced by better explicating the calculations. The Scott-Knott algorithm is a particular clustering algorithm, and the reader needs some clarification how it contributes to the demonstration.
Reply 2.2: You are quite correct- our description of Scott-Knott was opaque. We have added more text at the start of 5.5.
2.3 Moreover, Fig.13 is still insufficiently clear. The authors added a lengthy legend; I tend to think this would better be placed in the text. The column on the left hand (i.e. "rank") is still unclear to me.
Reply 2.3: . You are correct: that explanation text in the figure caption was confusing. By moving it to to section 5.6, it can be expanded into a dot list (easier to read) and extra details
can be added (for example, as you suggest, we can discuss more the meaning of the "rank" column).
2.4 - I am troubled by Fig. 12, how many defects have in total been fixed in the same phase in which they have been found? What is the proportion from the initial 47,376 defect logs that are concerned in the statistics in Fig. 13?
Reply 2.4: "Find and fix" is a single sub-task, so all defects are "found and fixed" in the same phase. This is distinct from injected and found.
- Finally, looking at Fig.6 and defect type distribution, wouldn't be relevant to explore any possible effect of defect type on the results in Fig. 13? As defect severity is absent, may be defect type has some influence. It could be that certain categories of defect can be corrected any time during the project, it could even be that these defects, in particular, are delayed for later (because they are easy to repair)/
Reply 2.5: Indeed, some activities are more likely to find certain types of defects. This was explored in a pair of TSP Symposium papers using PSP data.
D. Vallespir and W. Nichols, “An Analysis of Code Defect Injection and Removal in PSP,” in Proceedings of the TSP Symposium 2012, 2012. , and
D. Vallespir, “Analysis of Design Defect Injection and Removal in PSP,” in TSP Symposium 2011, 2011, pp. 1–28.
However, exploring this with data from so many distinct projects presents challenges and analysis beyond the scope of this paper. This is an interesting question that should be pursued in future work.
Although some types of defect are clearly more challenging than the others (syntax is often simple), the discovery phase and activity is a much larger effect. Examining this relationship is a very good idea, but we judged that to be a significant future effort to elaborate on the more basic finding.
2.6 Minor remarks:
Reply 2.6: Fixed
Reviewer #3: The authors have adequately addressed my observations. Below is a list minor comments (with one exception, all of them are typos).
R3.1: Section 5.2.2
"quote elaborate" should be "quite elaborate"
Section 5.4
"discuss the of the projects" should probably be "discuss those of the projects"
"the logical order is described section 5.2.3 is followed" should be "the logical order described in Section 5.2.3 is followed"
In Fig. 11, the "min" Year is not reported (even though the text says it's 2006).
"teams meet at least weekly" should be "teams met at least weekly" (everything else is in the past tense)
Section 5.5
"it apples some statistical hypothesis test" should be "it applies some statistical hypothesis test"
"the division of of l treatments" should be "the division of l treatments"
Reply 3.1: These items have been fixed. Thanks!
3.2 The description of the Scott-Knott ranker is not terribly clear. You write "Scott-Knott seeks the division of of l treatments into subsets of size m, n of sizes ls,ms, ns and median values l.μ,m.μ, n.μ (respectively) in order to maximize ms/ls abs(m.μ − l.μ)^2 + ns/ls abs(n.μ − l.μ)^2" but it's not clear to me what "subsets of size m, n of sizes ls,ms, ns and median values l.μ,m.μ, n.μ (respectively)" means. Are these 5 subsets? How are they related?
Reply 3.2: That text has unclear and we have fixed it. See the second dot list in Section 5.5.
3.3 In the caption of Figure 13, you write "(these values are calculated by sorting all resolution time, then reporting the middle values of that sort) The", which should be "(these values are calculated by sorting all resolution times, then reporting the middle values of that sort). The"
Section 5.6
"examples where there exists at least N >= 30 examples" should be "examples where there exist at least N >= 30 examples"
Should "stratification's" be "stratifications"?
Section 6.1
"the data are are consistent" should be "the data is consistent" (you've always used "data" as a singular noun everywhere else in the paper)
Section 6.4
"cased study" should be "case study"
"the data are" should be "the data is"
Reply 3.3: fixed
Pretty sure this is a Bill line...
Section 6.1 - It is not obvious to me how to interpret "conformance with Benford distribution". Could you please elaborate somewhat?
Section 4: I think the paragraphing could be improved - the highly interesting point about agile is hidden. "A goal of agile methods is to reduce /---/ little empirical data exist" appears in the middle of a paragraph headed by "Shull et al. conducted a literature survey and held a series of e-workshops". Consider restructuring this section to bring forward the agile perspective.
I don’t see why in Fig. 3 you report the whole list of 10 laws surveyed, while there is no discussion or introduction of the other nine laws beyond the Phase Delay one. To me either you include some more discussion of them or better you remove the list.
bill--
worried you won't like the reduced description of TSP in this draft. now, space has to be made for the extra discussion points demanded by the reviewers. but i would not be surprised if you start adding back some of the culled details
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.