seneubert / maven-review Goto Github PK
View Code? Open in Web Editor NEWProposal for the methodology of a 2-tiered collaborative review process of scientific analyses.
Proposal for the methodology of a 2-tiered collaborative review process of scientific analyses.
Feedback from Mike S.:
In American English "backlog" has a slightly bad connotation. It is associated with things that should have already been done but which have somehow fallen through the cracks in the table.
Consider finding another term.
Maybe: "Analysis Plan"
Get away from review concept.
More for discussions feature design etc
Feedback from Mike S.:
The analysts should be responsible for all analysis documentation. They should not be told by other people what to do and how to prioritize things. The backlog (or analysis plan) is a very good idea and should be made public. But it should be maintained by the analysts. Reviewers, including the maven can give comments on items on the backlog.
It should be rethought if and in which form the reviewers are allowed to post items onto the backlog.
Feedback from Mike S.:
The maven review will be very valuable for certain analyses. In particular the key analyses (B2mumu, various CP-violation measurements ...) will profit. Invite working groups and analysis teams to try out the process. Don't force it onto them. Chances are that by Run3 the majority will have picked it up.
The term Maven has been a bit misleading and shifted the focus away from the central idea of the proposal.
"Iterative Review" is not as colorful but it might point more to the original idea.
Any other, more creative suggestions?
Feedback by Mike S.:
Before the Maven is assigned at least an analysis plan has to be there. Need not be "day one" of the analysis.
Tier 2 review need not be done in parallel to the working group review. Ideally when they come in the analysis is in a state, through the work of the tier 1 reviewers, that tier two can proceed quickly.
Intermediate steps of the analysis should be reviewed by "external" reviewers if there is an important milestone which you don't want to change later, such as the final selection in a blind analysis, before you unblind. In such a case it makes sense to get the selection approved by external, unbiased reviewers. In BaBar there was a successful model: for the unblinding approval a member of the WG who was not involved with the analysis up to that point (i.e. NOT the maven) does the review and approval to go to unblinding. This reviewer then also becomes an additional member of the tier 2 review committee.
Take into account that there is a rule in LHCb the one should only be on one review committee at a time.
People expect "substantial" amount of work to be done before they start reviewing. They want enough material for discussion.
Collection of resources on the topic of "review". Mostly software focussed but some of the points also apply to any kind of scientific review process.
Thoughts on computing in general: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745
Feedback from Mike S.:
Maven understood as senior physicists that act as a resource to improve the physics of the analysis. People who could fill this role might be too senior to be willing to dig into the code. But they are very much appreciated to give feedback on the analysis strategy.
Therefore: Add another role, a dedicated software reviewer. This could be a young person. Additional benefit could be that by doing the code review they would learn a lot (about the software and about the physics analysis).
The collaboration reviewers are appointed by the PC after the working group approval.
In order to emphasize the change from a staged to a tiered, iterative review model we should redefine the role of the working group approval. What should be approved there is the plan how to do the analysis. This should contain a list of steps documented in the (public) analysis logbook that define what the result of the analysis will be and which steps are needed to get there. This includes a list of systematic checks and the infrastructure necessary to do them.
The working group approval also should contain any preliminary studies that show the feasibility and potential of the analysis. Ideally the team has built a simple pipeline that demonstrates all basic steps of the analysis (excluding systematic checks and with possibly simplified models, fitters, selection etc). This prototype can also be run on monte carlo alone.
After the analysis is approved by the working group, the team proceeds to implement the full analysis. The collaboration reviewers are appointed and immediately start to give feedback on the prototype, the analysis plan as laid out in the analysis log and the progress of the team as new features are added to the analysis.
what do you say?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.