klainfo/patterns

How to assemble 'good' papers is SE

This repository is an experiment in letting a large community write and revise the patterns they expect to see in different kinds of papers. Here we say:

• Reviewers review by quickly sniffing a paper for its "smells", before diving into the details.
• Where do these "smells" come from? Well:
  • Papers have classes
  • Classes may conform to patterns
  • Patterns contain good and bad smells (things the community generally likes or questions).

Authors should be able to say review 'my paper according to "this" pattern'. And if they don't see "their" kind of pattern, they should be free to propose modifications and/or extensions to a library of patterns.

In this models, experiences researchers (e.g. Associate editors or program board members)
review proposed changes to the patterns library, perhaps rejecting or improving some ideas before they get added to the pattern library.

Patterns provide guidance for best practices when writing:

• Motivation: a motivational statement on why doing some new work is important;
• Method: the details on this will be investigated;
• Results: the results seen when the authors apply the method.
• Future Work: statements of what should follow on from this work.

Further, some artifacts are orders of magnitude slower to generate that others:

• A good motivational statement might be written in a few hours/days;
• But a good study might need months/years to complete.

It is therefore wise (and fast) to get reviewer feedback on the initial plan, lest they are wasting time on a paper that will never get accepted (since it has the wrong artifacts).

XXX nice to have a catalog of open issues

Not all papers need details on of all these four parts. In fact, it is good practice to divide these into separate papers:

• An initial (and short) open issues paper can just be about motivation and method. Reviewers can offer improvements to make this paper conform to known patterns. By keeping these papers short, they can be kept short, thus enabling quick review times.
• A subsequent (and longer) findings paper that describes the results from the method.

Patterns-friendly venues encourage both open issues and findings papers by encouraging open issues authors faster review times if they submit the subsequent findings paper to the same venue.

Patterns and Artifacts

Patterns have a structure; i.e. a list of artifacts that make up a paper (and different kinds of papers will need different patterns). Here is a list of artifacts that might make up a paper. This list is hardly complete and many of the following are not relevant for all papers. Also, just to say the obvious, some papers will be artifacts not listed below.

1. Motivational statements or reports or challenge statements or lists of open issues that prompt an analysis;
2. Hypotheses, about expected effects in some area;
3. Checklists used to design the analysis (see also, the Checklist Manifesto (http://atulgawande.com/book/the-checklist-manifesto/);
4. Bibliographies, comprehensive, annotated, and insightful (e.g. showing the development or open areas in a field);
5. Study instruments such as surveys interview scripts, etc;
6. Statistical tests used to analyze results (along with some notes explaining why or when this test is necessary);
7. Commentary on scripts used in the analysis;
8. Examples of particularly informative visualizations (e.g. Sparklines http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001OR )
9. Baseline results against which new work can be compared;
10. Sampling procedures e.g. ``how did you choose the projects you studied?'';
11. Patterns describing best practices for performing this kind of analysis;
12. Anti-patterns describing cautionary tales of ``gotchas'' to avoid when doing this kind of work;
13. Negative results that are anti-patterns, backed up by empirical results;
14. Tutorial materials: Guides to help newcomers become proficient in the area. Some of these tutorial materials may be generated by the researcher and others may be collected from other sources.
15. New results that offer guidance on how to best handle future problems.
16. Future work: From the results, there many be speculations about open issues of future issues that might become the motivation for the next round of research.
17. The actual text of an author's papers;
18. Any data used in an analysis
    • Either raw from a project;
    • Or some derived product. Note that some data is too large to fit into the standard on-line freely available repos (e.g. Github only allows 1GB reps). For such data, we suggest using some file XXX.goto; each line of which is one url where the related data can be collected.
19. Scripts used to perform the analysis (the main analysis or the subsequent statistical tests or visualizations; e.g. the Python Sparklines generator). Scripts can also implement some of the patterns identified by the paper.
20. Executable models that can generate exemplar data; or which offer an executable form of current hypotheses;
21. Programs that realize the algorithms presented or used in the paper;
22. Delivery tools to let novices automatically rerun the analysis; e.g.
    • Config management files that can
      • build the system/ paper from raw material and/or
      • update the relevant files using some package manager
    • Virtual machines containing all the above scripts, data, etc, pre-configured such that a newcomer can automatically run the old analysis.

FAQ

leads to stupid papers?

what about paradigm breaking ideas.