seneubert/maven-review

Proposal for the methodology of a 2-tiered collaborative review process of scientific analyses.

maven-review's Issues

Suggestion: Rename backlog

Feedback from Mike S.:
In American English "backlog" has a slightly bad connotation. It is associated with things that should have already been done but which have somehow fallen through the cracks in the table.
Consider finding another term.
Maybe: "Analysis Plan"

Rename Maven to sparring partner

Get away from review concept.
More for discussions feature design etc

Suggestion: The analysis team owns and maintains the backlog

Feedback from Mike S.:
The analysts should be responsible for all analysis documentation. They should not be told by other people what to do and how to prioritize things. The backlog (or analysis plan) is a very good idea and should be made public. But it should be maintained by the analysts. Reviewers, including the maven can give comments on items on the backlog.
It should be rethought if and in which form the reviewers are allowed to post items onto the backlog.

Suggestion: Offer the maven review as an alternative process, not a replacement

Feedback from Mike S.:
The maven review will be very valuable for certain analyses. In particular the key analyses (B2mumu, various CP-violation measurements ...) will profit. Invite working groups and analysis teams to try out the process. Don't force it onto them. Chances are that by Run3 the majority will have picked it up.

Rename the Maven Review to Iterative Review

The term Maven has been a bit misleading and shifted the focus away from the central idea of the proposal.
"Iterative Review" is not as colorful but it might point more to the original idea.

Any other, more creative suggestions?

Suggestion: Bring in reviewers only when substantial work has been done

Feedback by Mike S.:
Before the Maven is assigned at least an analysis plan has to be there. Need not be "day one" of the analysis.
Tier 2 review need not be done in parallel to the working group review. Ideally when they come in the analysis is in a state, through the work of the tier 1 reviewers, that tier two can proceed quickly.
Intermediate steps of the analysis should be reviewed by "external" reviewers if there is an important milestone which you don't want to change later, such as the final selection in a blind analysis, before you unblind. In such a case it makes sense to get the selection approved by external, unbiased reviewers. In BaBar there was a successful model: for the unblinding approval a member of the WG who was not involved with the analysis up to that point (i.e. NOT the maven) does the review and approval to go to unblinding. This reviewer then also becomes an additional member of the tier 2 review committee.

Take into account that there is a rule in LHCb the one should only be on one review committee at a time.
People expect "substantial" amount of work to be done before they start reviewing. They want enough material for discussion.

(Software) review resources

Collection of resources on the topic of "review". Mostly software focussed but some of the points also apply to any kind of scientific review process.

• 11 Best Practices for Peer Code Review
• Code reviews, the lab meeting
• Have a checklist so it is clear to reviewer and reviewee what will happen and define when they are done
• "A Foolish Consistency is the Hobgoblin of Little Minds", don't get distracted by discussing "{code,writing} style", discuss what the {code,writing} does. (Use a robot to determine if it passes style or not)
• When you give people rules/guidelines they will start applying them to the letter ("No lines longer than 80characters!!!") which is useless and means they miss the gorilla. This is a hard problem to solve. If you become unsure/lack sufficient expertise on the topic you are reviewing it is easy to simply "apply the rules" instead of review the substance. The alternative would be to admit (in public) that you lack the expertise, which is extremely hard to do. It also takes a lot of practice and conscious effort not to get sucked into bible slinger mode.

Thoughts on computing in general: http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745

Suggestion: Make Maven and code reviewer two separate roles

Feedback from Mike S.:
Maven understood as senior physicists that act as a resource to improve the physics of the analysis. People who could fill this role might be too senior to be willing to dig into the code. But they are very much appreciated to give feedback on the analysis strategy.
Therefore: Add another role, a dedicated software reviewer. This could be a young person. Additional benefit could be that by doing the code review they would learn a lot (about the software and about the physics analysis).

Redefine the role of the working group approval

The collaboration reviewers are appointed by the PC after the working group approval.

In order to emphasize the change from a staged to a tiered, iterative review model we should redefine the role of the working group approval. What should be approved there is the plan how to do the analysis. This should contain a list of steps documented in the (public) analysis logbook that define what the result of the analysis will be and which steps are needed to get there. This includes a list of systematic checks and the infrastructure necessary to do them.
The working group approval also should contain any preliminary studies that show the feasibility and potential of the analysis. Ideally the team has built a simple pipeline that demonstrates all basic steps of the analysis (excluding systematic checks and with possibly simplified models, fitters, selection etc). This prototype can also be run on monte carlo alone.

After the analysis is approved by the working group, the team proceeds to implement the full analysis. The collaboration reviewers are appointed and immediately start to give feedback on the prototype, the analysis plan as laid out in the analysis log and the progress of the team as new features are added to the analysis.

what do you say?