notconfusing / wiki_econ_capability Goto Github PK
View Code? Open in Web Editor NEWusing economic relfections method modelling to determine wikipedia editor and article rank
using economic relfections method modelling to determine wikipedia editor and article rank
R1.9. It is not clear how the snapshots enter into the
calibration process until Fig. 6 is introduced. It would be
good to explain better the whole process on page 4.
The ground-truth comparisons aren’t exactly exogenous! They’re
based on degree and other network artifacts measured here.
ass one of these two styles http://research.microsoft.com/en-us/um/people/merrie/temp/change_log_samples.html
As you prepare the camera-ready version of the CSCW paper, can you please include the following acknowledgement:
"This research was supported in part by the National Science Foundation under award CCF-0424422 (TRUST)."
Figure 6 should show \beta^* (R1).
Related works: R1 and R3 suggest literature that should be incorporated
in the paper. As mentioned in the first major point of this meta-review,
the related work should also show that the paper is situating itself in
the CSCW discipline. Given past work, some of the claims of the paper
need to be tempered or justified.
Both R1 and R2 find some of the mathematical notation confusing. This
should be fixed for clarity's sake. R1 has also identified some typos.
R1.6. I found the notation in Eq. 1 confusing. $k_e$ should be
a scalar, but taken literally, Eq. 1 would imply that $k_e$
is simply $N_a$ times matrix $M$, causing a lot of confusion
in eq. 2 with the $\frac{1}{k_e}$ term. The fact that you
implicitly denote a generic entry $M_{ea}$ as $M$ is buried
in the text, so it doesn't help much. In general, be more
clear when you implicitly simplify your notation.
R1.7. There is a typo in the notation of page 5: $\omega_a$
should be $\overline{\omega}_e$.
R1.8. For the sake of clarity, please specify that the \max
operator of Eq. 5 is over \alpha and \beta.
The new discussion of the bi-partite network random walker model was very
clear. The method as discussed in the paper now is more aligned with the
bipartite HITS algorithm; this corresponds to the eigenvector centrality
of the two projections of the matrix (M^TM, MM^T). I'm not sure that I
understood the beta parameter before. The analogous PageRank algorithm
would be the version in which a jump from e to a would be allowed with
nonzero probability, even with M_ea = 0.
Figure 5 is interesting, and it's a nice tie-in to Figure 3. It seems
like this is partial identifiability for the parameters.
Additional subpoint : how this model would be useful from wikipedia community ? one or two paragraph?
Expand discussion of findings back to Wikipedia and its socio-technical
practices. Many of the most significant contributions come from the
optimal alpha and beta variables found. The paper only touches briefly on
what these optimal values for various Wikipedia categories mean. The
paper should expand significantly its discussion on the broader meaning
of the alpha and beta functions. For example, I enjoyed the (too brief)
discussion on the meaning of the beta values in the Sexual acts and
Military History of the US. An significantly expanded discussion of these
findings (and perhaps for other categories) is merited. There is also a
lot of speculation in the current findings--.e.g, "presumably there are a
lot of unmediated editors fighting". Any expanded discussion should be
grounded in either empirical findings or literature.
R2.3. A related point to validation: there is discussion of result on
multiple editors creating marginal dis-value. It’s not clear to me that
this result isn’t the product of the types of editors in those fields.
If they are editors who as a population edit fewer articles per person
than in other fields, then I believe this result would appear as an
artifact of the algorithm.
=> make a graph of beta as a function of ratio = articles/editor
Parameters are sometimes introduced in the paper without proper
explanation for a broad audience. For example, the semantic meaning of
alpha and beta other than the fact they control the transition
probabilities should be explained early.
really close read for spelling/grammar
R1.12. The author claim (page 6, Discussion) to have test their
model, but what they did only a calibration against a number
of ground-truth metrics. How would you validate this model?
In other words, unless the model was really bad,
I would expect anyway that an optimization process would
give you parameters that reproduce well the ground truth,
but does the model capture really quality and expertise, or
just the ground truth you gave it to approximate? The
discussion section could be a good place where to talk about
this.
I realize there is precedence for this, but the notation that M == M_ea
(i.e., M_ea indicates the matrix) and the use of M_ea to indicate the
variable for editor e and article a is unclear.
R1.11. The text on page 6 says that $\beta^$ is stable but
figure 6 shows \rho_a and \rho_e, not $\beta^$ itself.
Could it be that \beta changed much (have a look at the
contour lines in Fig. 5 for example) even the resulting
correlation with the ground truth didn't? Without any
explanation the reader is left to wonder what is really
going on. This is important, since the robustness of the
calibrated parameters make it possible to interpret them.
The manuscript describes "ground truth" of article quality and editor
expertise, but it's unclear what specific construct was used that is
independent of the editing behavior. Editor experise is measured as labor
hours, but this seems instrinsically correlated with the number of
articles edited. Article quality was measured with a PCA combining markup
complexity, headings, article length, citations, and links. The
implementation of the adapted random walker method returns correlations
in excess of .6 with other measures of article and editor quality.
There's also a question of how much gain does this walker method produce
over more naive methods that don't account for the network structure and
influence to serve as a baseline.
Per R3's comment: The paper needs to motivate better its
"state-of-the-art ground-truth evaluations" for editor expertise and
article quality. It should explain how (and whether we should expect)
article quality and expertise measures to be independent from each other.
R3 is concerned about the clarity of the paper and found him/herself "losing the forest for the trees" when trying to connect the "interpretations to bigger picture questions about what are social mechanisms for facilitating the production of high-quality user-generated content." I agree with this statement.
R2. Much of the previous work on the method of reflections is done on
country exports and the global economy, but some of the discussion on
this method and global economic analysis seems unrelated.
R2's suggestion for "a comparison of performance to degree, or a
correlation of the output of the method to degree" should be considered.
R2.2. Centrality measures and the method of reflections.
The model presented is based on the previously established method of
reflections, and uses the PageRank-like variation on that method
suggested by Caldarelli et al. (2012). This means the inferred ranking is
a bipartite PageRank variation similar to a bipartite HITS algorithm,
which has been previously analyzed. Most importantly, the rankings output
by these types of algorithms will be strongly correlated with degree
(here, number of articles edited or number of editors edited by). Since
degree is one of the major components of the validation/ground-truth
rankings, a significant amount of agreement could be accomplished this
way.
2. A comparison of performance to degree, or a correlation of the output
of the method to degree, should be shown or acknoweldged in the text.
Provide the audience with an intuitive way with examples to understand the meaning behind the alpha and beta transition probabilities. The text on pg. 4 describes what the
various ranges of alpha and beta mean, but the authors should expand on this and possibly use illustrations/figures to show how changing the values leads to different relations between article quality, # of edits, and/or expertise.
The paper should address R1's question of why binary values were used
in M.
Both R1 (point 12) and R2 (point 1) brings up a good point that alpha
and beta are optimized to rho_e and rho_a and then used to validate the
performance of the model (or at least the rate at which the algorithm
converges towards rho_e and rho_a). So in some sense, the higher
correlations are somewhat unsurprising because that is what we are trying
to maximize. The paper should explain this limitation; the authors may
also consider one of the further examinations suggested by R2 (see R2:
Major points: 1. Validation). For example, an out-of-sample test would be
helpful (and maybe doable in the time period).
R1.12. The author claim (page 6, Discussion) to have test their
model, but what they did only a calibration against a number
of ground-truth metrics. How would you validate this model?
In other words, unless the model was really bad,
I would expect anyway that an optimization process would
give you parameters that reproduce well the ground truth,
but does the model capture really quality and expertise, or
just the ground truth you gave it to approximate? The
discussion section could be a good place where to talk about
this.
R2.1 The model has some parameters (alpha, beta) which are fit using the rank
correlation metric (rho_e and rho_a). However, these parameters are also
used to validate and describe the overall performance of the model. At
the exploratory stage, this seems excusable, but merits at least one of
the following: (a) an exploration of which aspects of the ground truth
metrics are un/correlated (see [2]); (b) another type of validation
(prediction? other rankings?); (c) an out-of-sample test, using the
parameters fit to either the previous time steps or other subject areas
in the model; (d) a discussion of how the model’s ranking qualitatively
differs from the ground-truth.
Explain the research's contributions/broad impact. The paper only cites
one CSCW article. I don't have a problem with this, but I think this is
indicative that the paper needs to better situate its contribution to the
CSCW discipline. What can we do with this new way of modeling article
quality and expertise? What are its implications towards Wikipedia and/or
other similar online collaborative editing systems? How do the optimal
alpha and beta parameters and the model inform socio-technical theories
of Wikipedia practices? What would be the next steps (beyond tweaking the
algorithm)?
R1.1. Introduction: It is a bit surprising that the topic of
information cascades is introduced referencing only two
unpublished works (one of them under review). It would be
great if the authors could also include other pointers to
the literature of team organization and complexity.
R1.2. The authors claim their work to be “the first attempt to
quantify the value of collective contribution environments
from the collaboration structure alone”, but I think that,
at least in the domain of Wikipedia, the work by Luca de
Alfaro and collaborators should be credited. A lot of work
in the field of trust and reputation management tried also
to look quality of contributions, as well as some works by
Ulrik Brandes on collaboration networks.
R1.3. The related work section would use a more thorough review
of the literature. Some words should be spent on the
connection between public goods and team production for
example. In Microeconomics, for example, this is studied as
a coordination problem. There is also a large literature in
software engineering about software teams and quality.
Another work that looked at the growth of Wikipedia pages
and quality is the one by Huberman and Wilkinson.
R1.4. I would tone down the claim that measuring quality is
impossible. This is simply not true. Experts are able to
evaluate non-code artifacts, and even opinions by
non-experts can be aggregated into meaningful signals. OTOH,
it seems a good idea to talk also about unpredictability of
quality (cf. the work by Watts, Dodds and Salganick). Also,
natural language can be evaluated with a lot quantitative
metrics (some of which you actually use as ground truth
later in the paper), so please tone down also the language
there.
R1.5 The point about number and quality is well taken. To this
end, I would suggest to look at the work by Scott Page on
diversity and teams.
I’d be interested in any systematic patterns in the inferred
rankings, and how they differ from the ground-truth rankings -- the
maximum achievable rank correlations are quite low on some categories.
R3 says:
Do you know how to do this easily?
The issue of coordination raised by R3. I think this can be solved by
acknowledging other definitions and then defining clearly what the
authors meant by coordination (since that term is used throughout the
paper).
However, the paper constructs some terms like "coordination"
problematically. The argument that the paper "[reverse engineers] a
measure of coordination" (p 2) appears to conceive of coordination as a
feature to be quantified rather than an on-going process that produces
other measurable outcomes. There's also an underlying confounding of
coordination and article quality, which may not be wrong but is
inaccurate: there may be articles that are extensively coordinated
(Israel-Palestine, global warming, abortion) that may not necessarily be
high quality. Given that the article is measuring article quality
throughout and is only looking at static structures rather than changes
over time, I would recommend the authors be very cautious about
generalizing to coordination processes.
R1.10. Section “Data” reports about both the data used in the
study and on the calibration procedure itself. I would split
it in two separate sections for the sake of clarity.
The paper should address R2's concern that variability across types of
editors is a major factor in the creation of "marginal dis-value."
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.