jalalal-tamimi/many_analyses

Many Speech Analyses

Many Speech Analyses

Welcome!

This project also lives on the OSF.
General project information can be found here.

Brief summary

We recruted collaborators interested in forming data analysis teams to examine the same dataset. The data analysis teams wrote a journal-ready methods and results section and then peer-reviewed each others analyses. Next, we used meta-analytic techniques to explore the data analysis teams' findings.

Overview

Recent efforts to replicate published findings have uncovered surprisingly low success rates across disciplines. Moreover, several studies have highlighted the large degree of analytic flexibility in data analysis which can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches that derives not only from aspects of statistical modelling but also from decisions regarding the quantification of the measured behavior. In the present study, we gave the same speech production data set to 30 teams of researchers and asked them to answer the same research question. Using Bayesian meta-analytic tools, we have observed substantial variability between teams and submitted analyses, with analytic and researcher-related predictors having little or no effect on the reported effects.