emsejournal/openscience

Empirical Software Engineering journal (EMSE) open science and reproducible research initiative

openscience's Issues

Register EMSE as accepting RR with OSF

EMSE can ask David Mellor at OSF to list it as a journal accepting RR on http://cos.io/rr

Hypothesis testing vs. generating

Should RRs focus solely on hypothesis testing and not allow hypothesis generating research? Currently, the impression is that hypothesis testing is the sole focus. This could be intentional of course :)

papers & posts related to open science, artefact evaluation and reproducibility

Prepare MSR author instructions for Phase 1

OSF has author templates: https://osf.io/pukzy/

The author guidelines should comply with the eventual policies EMSE decides based on #16

add support for preregistered studies

We can support registered report / preregistered studies. Basically, reviewers review a study plan, and if approved, regardless of whether the results are positive or negative, the study will be published. There’s a proposal to do so at CSCW: https://blog.communitydata.cc/a-proposal-to-mitigate-false-discovery-in-cscw-research/ (suggestion by Neil Ernst)

See also https://en.wikipedia.org/wiki/Registered_report

MSR - RR dates and timeline

Tentative timeline, based on other MSR tracks:

• Pre-registration plans due: February 6, 2020
• Author notification: March 2, 2020
• 1 page summary plan / camera-ready: March 16, 2020
• Reports registered with OSF registry: Mar 31, 2020

The period between initial plan and decision is only 4 weeks, which is pretty short if the registration is complex.

MSR RR - check with OSF on registration process

@neilernst will talk to OSF folks about getting EMSE/MSR approved(?) as journal following RR process, so authors can submit the IPA (in principle, stage 1 acceptance).

Consider using a rebuttal phase for phase 1 RR

Given the is a new approach, it seems certain some submissions will miss out on some of our requirements. It would be foolish to desk-reject without some chance for rebuttal.

See also #10 (comment)

tool support for reproducible science

post to this issue to share, subscribe to this issue or watch the repo (right-hand side) to receive pointers about reproducible science

Figure out hosting of RR phase 1

Phase 1 registrations need to be hosted somewhere. For example, OSF maintains a hosting infrastructure with templates. We could also consider allowing embargo periods (where the registration is held in confidence).

See https://osf.io/registries

Bayesian and frequentist statistics

Should we make a statement on frequentist vs Bayesian statistics as others have done?

https://openscience.bmj.com/pages/registered-reports-guidelines/

https://www.elsevier.com/__data/promis_misc/Registered%20Reports%20Guidelines%20JESP%202018.pdf

https://bmcmedicine.biomedcentral.com/about/registered-reports