researchart/patterns

How to assemble 'good' papers is SE

This repository is an experiment in letting a large community write and revise the patterns they expect to see in different kinds of papers. Here we say:

• Reviewers review by quickly sniffing a paper for its "smells", before diving into the details.
• Where do these "smells" come from? Well:
  • Papers have classes
  • Classes may conform to patterns
  • Patterns contain good and bad smells (things the community generally likes or questions).

Authors should be able to say review 'my paper according to "this" pattern'. And if they don't see "their" kind of pattern, they should be free to propose modifications and/or extensions to a library of patterns.

In this models, experiences researchers (e.g. Associate editors or program board members)
review proposed changes to the patterns library, perhaps rejecting or improving some ideas before they get added to the pattern library.

Patterns provide guidance for best practices when writing:

• Motivation: a motivational statement on why doing some new work is important;
• Method: the details on this will be investigated;
• Results: the results seen when the authors apply the method.
• Future Work: statements of what should follow on from this work.

Further, some artifacts are orders of magnitude slower to generate that others:

• A good motivational statement might be written in a few hours/days;
• But a good study might need months/years to complete.

It is therefore wise (and fast) to get reviewer feedback on the initial plan, lest they are wasting time on a paper that will never get accepted (since it has the wrong artifacts).

XXX nice to have a catalog of open issues

Not all papers need details on of all these four parts. In fact, it is good practice to divide these into separate papers:

• An initial (and short) open issues paper can just be about motivation and method. Reviewers can offer improvements to make this paper conform to known patterns. By keeping these papers short, they can be kept short, thus enabling quick review times.
• A subsequent (and longer) findings paper that describes the results from the method.

Patterns-friendly venues encourage both open issues and findings papers by encouraging open issues authors faster review times if they submit the subsequent findings paper to the same venue.

Patterns and Artifacts

Patterns have a structure; i.e. a list of artifacts that make up a paper (and different kinds of papers will need different patterns). Here is a list of artifacts that might make up a paper. This list is hardly complete and many of the following are not relevant for all papers. Also, just to say the obvious, some papers will be artifacts not listed below.

1. Motivational statements or reports or challenge statements or lists of open issues that prompt an analysis;
2. Hypotheses, about expected effects in some area;
3. Checklists used to design the analysis (see also, the Checklist Manifesto (http://atulgawande.com/book/the-checklist-manifesto/);
4. Bibliographies, comprehensive, annotated, and insightful (e.g. showing the development or open areas in a field);
5. Study instruments such as surveys interview scripts, etc;
6. Statistical tests used to analyze results (along with some notes explaining why or when this test is necessary);
7. Commentary on scripts used in the analysis;
8. Examples of particularly informative visualizations (e.g. Sparklines http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001OR )
9. Baseline results against which new work can be compared;
10. Sampling procedures e.g. ``how did you choose the projects you studied?'';
11. Patterns describing best practices for performing this kind of analysis;
12. Anti-patterns describing cautionary tales of ``gotchas'' to avoid when doing this kind of work;
13. Negative results that are anti-patterns, backed up by empirical results;
14. Tutorial materials: Guides to help newcomers become proficient in the area. Some of these tutorial materials may be generated by the researcher and others may be collected from other sources.
15. New results that offer guidance on how to best handle future problems.
16. Future work: From the results, there many be speculations about open issues of future issues that might become the motivation for the next round of research.
17. The actual text of an author's papers;
18. Any data used in an analysis
    • Either raw from a project;
    • Or some derived product. Note that some data is too large to fit into the standard on-line freely available repos (e.g. Github only allows 1GB reps). For such data, we suggest using some file XXX.goto; each line of which is one url where the related data can be collected.
19. Scripts used to perform the analysis (the main analysis or the subsequent statistical tests or visualizations; e.g. the Python Sparklines generator). Scripts can also implement some of the patterns identified by the paper.
20. Executable models that can generate exemplar data; or which offer an executable form of current hypotheses;
21. Programs that realize the algorithms presented or used in the paper;
22. Delivery tools to let novices automatically rerun the analysis; e.g.
    • Config management files that can
      • build the system/ paper from raw material and/or
      • update the relevant files using some package manager
    • Virtual machines containing all the above scripts, data, etc, pre-configured such that a newcomer can automatically run the old analysis.

FAQ

leads to stupid papers?

what about paradigm breaking ideas.

patterns's Issues

Lack of an artifact

Among the invalid criticism, the document states "Lack of an artifact should not be grounds for rejection of the scientific paper. There may be valid reasons, such as industrial non-disclosure agreements".

While I can partially agree on that, I think there should be a different standard (weight) between "industry data", "open source data", and "personal data". I do agree that in industrial applications there are NDA. Further, studies with human participants may include sensitive data.

But papers that mine data from open-source projects from Github (or similar) have no "major" reasons not to include an artifact.

I would suggest that that bullet point for criticism is revised with regard to the context (industrial or not) of the paper on which such a criticism is valid or not

registerred reports

reviewers cant be on v2 of the RR

• all authors have to sign off on author change
• standards deny reviewers

extraordinary attributes

I'm not sure about the extraordinary attributes. These seem like they're beyond the researcher's control. Is it wise to evaluate a piece of work based on factors beyond the creator's control?

what is an industrial report

John: here's "case study" from the sigsoft standard . does this match to "industrial reprot" as you understand it?

Tim: IMO they are related but different, but let me read in detail and reflect back...

1. Description
   1. Presents a detailed account of a specific instance of a phenomenon at a site. The phenomenon can be virtually 2 anything of interest (e.g. Unix, cohesion metrics, communication issues). The site can be a community, an organization, a team, a person, a process, an internet platform, etc.
   2. Features direct or indirect observation (e.g. interviews, focus groups)—see Lethbridge et al.’s (2005) taxonomy.
   3. Is not an experience report (cf. Perry et al. 2004) or a series of shallow inquiries at many different site
2. Essential
     1. explains why the case study approach is appropriate for the research question
   2, justifies the selection of the case or site that was studied
   3. describes the context of the case in rich detail o defines unit(s) of analysis
   4. presents a clear and well-argued “chain of evidence” from observations to findings
   5. clearly answers the research question(s)
3. Desirable
   1. reports the type of case study "
      1. descriptive case study describes—in vivid detail–a particular instance of a phenomenon
      2. an emancipatory case study identifies social, cultural, or political domination “that may hinder human ability”
         (Runeson and Host 2009), commensurate with a critical epistemological stance
      3. an evaluative case study evaluates a priori research questions, propositions, hypotheses or technological artifacts
      4. an explanatory case study explains how or why a phenomenon occurred, typically using a process or variance theory
      5. an exploratory case study explores a particular phenomenon to identify new questions, propositions or hypotheses
      6. an historical case study draws on archival data, for instance, software repositories
      7. a revelatory case study examines a hitherto unknown or unexplored phenomenon
   2. describes external events and other factors that may have affected the case or site
   3. explains how researchers triangulated across data sources, informants or researchers
   4. cross-checks interviewee statements (e.g. against direct observation or archival records)
   5. uses quotations to illustrate findings (note: quotations should not be the only representation of a
4. Extraordinary
   1. multiple, deep, fully-developed cases with cross-case triangulation
   2. uses multiple judges and reports inter-rater reliability (cf. Gwet & Gwet 2002)
   3. uses direct observation and clearly integrates direct observations into results
   4. created a case study protocol beforehand and makes it publicly accessible
5. Evaluation
   1. Use: credibility, multivocality, reflexivity, rigor and transferability (see Glossary).
   2. Don't crtique via: replicability, generalizability and objectivity typically do not apply. Also, don't say
      1. Does not present quantitative data; only collects a single data type. 12
      2. Sample of 1; findings not generalizable. The point of a case study is to study one thing deeply, not to generalize to a population. Case studies should lead to theoretical generalization; that is, concepts that are transferable in principle.
      3. Lack of internal validity. Internal validity only applies to explanatory case studies that seek to establish causality.
      4. Lack of reproducibility or a “replication package”; Data are not disclosed (qualitative data are often confidential).
      5. Insufficient number of length of interviews. There is no magic number; what matters is that there is enough data that the findings are credible, and the description is deep and rich.

more succinct

Remember that the longer and more verbose the standard, the fewer people will read it. We need to speak as simply and concisely as possible. Try not to explain so much. Explanations go in papers and books. Standards should be mostly bullet-points, with reference to longer discussions in papers and books.

If we're going to say that "lack of an artifact should not be grounds for rejection of the scientific paper" then we can't have "Artifacts are available" in essential criteria. In fact, if lack of artifact is not grounds for rejection, then this standard shouldn't have an "essential" section - basically everything is desirable. That's why the open science supplement doesn't have any essential criteria. That said, I don't 100% agree with making openness forever optional. For example, suppose I do a questionnaire study. There are all sorts of reasons I might not be able to release the data. But what possible justification could there be for keeping the questions secret? If a questionnaire study comes in with no list of questions or pdf printout or something I think it's totally reasonable to reject the paper. How this applies to code I'm not sure.

"All these approaches have their positive and negative aspects so authors should be free to use whatever sharing technologies they feel is appropriate." - some wordsmithing is needed here. The reviewer should be allowed to say the sharing technology isn't appropriate - they just shouldn't be unreasonably picky about it.

Pauls comments2 add footnotes

Beyond that, please make use of footnotes. (The reviewing system will eventually display footnotes as tooltips). The Specific Attributes should be as concise as possible, so please push most examples and explanations into footnotes. For example, all that stuff about README files and requirements.txt can go in footnotes. You can find an example of how to do the footnotes in the optimization standard .

ethics

The note about ethics needs to find its way into specific criteria somehow. Same goes for the note about some artifacts being difficult to run. In general, the notes would be better if they were more concise. Again, it's very important to avoid turning the standard into a position paper.

for damine

Some of these, like the Motivation one, are already covered by the general standard and can be removed. Maybe take another look at the general standard and edit for redundancy

for damien

sir

must be a gap analysis

ideally, thee is also an experiment into that gap

Discussion about common issues for artifact evaluations based on my experience from ROSE@ICSME'20

Hi Tim,

Thanks for the initiatives! I really like the artifact standards. https://github.com/researchart/patterns/blob/master/standards/artifact.md

Below, I added some common issues when I co-chaired the ROSE@ICSME'20 that I received from reviewers?

• An artifact requires many dependencies (java, windows, ...) but reviewers do not have appropriate environments to evaluate (e.g., can't run an exe file, can't run a jar file, too many library dependencies) so how can we ensure that the artifact submission is cross-platform. Should we add technical expertise (e.g., Windows or Linux, java or python or R) when bidding artifacts, in addition to the domain expertise?

• For a python package, I believe requirements.txt is recommended.

• An artifact takes forever to run so how can we ensure that an artifact can be reproducible in a manageable amount of time. Should we add a max runtime (e.g., 15 mins)? Anything more than that requires a smaller size of data?

• We need an example README template as well. I recommend (this one).

• Some highly recommended resources https://guides.lib.berkeley.edu/reproducibility-guide.

Kind regards,

Kla

pauls comments1

We need to make up our minds whether we're talking about code artifacts or all artifacts. Right now there's like a nod toward all artifacts but then the rest of the standard is clearly about code artifacts. We already have an Open Science supplement. So maybe here we should just stick to code artifacts? Alternatively, instead of "Essential" and "Desirable" we should distinguish between "all artifacts" and "code artifacts." There should be criteria for all artifacts beyond "available" and "related." What do you think?

FAIR Principles for Artifacts

I think the FAIR principles should at least be mentioned, ideally be put into context of the artifact guidelines.

FAIR has become a standard term in many disciplines and is also starting to get frequently mentioned by University guidelines and within calls by funding agencies. While FAIR is restricted to data, our artifact definition covers data and should, therefore, cover FAIR. Consequently, FAIR is also mentioned and encouraged in this years Data Showcase call of the MSR.

If you agree, I could work on a PR, but not immediately (probably not before March, later if daycares stay closed). If this should be included earlier, some else should pick this up.

improve anti patterns

For antipatterns, try to avoid writing complex instructions for reviewers. Try to be more concise and write from the creator's perspective. Instead of:

"Artifact review is not code review. Taking longer than a few hours to install and run an artifact is rarely a good use of time. Thus, artifacts that are not immediately usable should be returned for improvement."

Say something like:

"providing artifacts that are difficult or time-consuming to install and run."

?just compiles

The general quality criteria basically just says "complies with the specific attributes." If you've nothing else to say here, just take out the section. Alternatively you could list some more general ideas like "usefulness" and "re-usability".

Likewise, the example acceptable deviation doesn't add much and can be dropped.

in general, there should be wow!

weite u investing pattern

contract string ley hebelis: wont be lebelved

contract early held believes; itneresting

conformwnat we areeladt know: duklll!!!!

More kinds of papers

Reply papers

Industry

A good resource about choosing software licenses

https://choosealicense.com/