Supplementary material from "Data availability, reusability and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition"
Posted on 2018-07-23 - 17:46
Access to data is a critical feature of an efficient, progressive and ultimately self-correcting scientific ecosystem. But the extent to which in-principle benefits of data sharing are realized in practice is unclear. Crucially, it is largely unknown whether published findings can be reproduced by repeating reported analyses upon shared data (‘analytic reproducibility’). To investigate this, we conducted an observational evaluation of a mandatory open data policy introduced at the journal Cognition. Interrupted time-series analyses indicated a substantial post-policy increase in data available statements (104/417, 25% pre-policy to 136/174, 78% post-policy), although not all data appeared reusable (23/104, 22% pre-policy to 85/136, 62%, post-policy). For 35 of the articles determined to have reusable data, we attempted to reproduce 1324 target values. Ultimately, 64 values could not be reproduced within a 10% margin of error. For 22 articles all target values were reproduced, but 11 of these required author assistance. For 13 articles at least one value could not be reproduced despite author assistance. Importantly, there were no clear indications that original conclusions were seriously impacted. Mandatory open data policies can increase the frequency and quality of data sharing. However, suboptimal data curation, unclear analysis specification and reporting errors can impede analytic reproducibility, undermining the utility of data sharing and the credibility of scientific findings.
CITE THIS COLLECTION
DataCite
3 Biotech
3D Printing in Medicine
3D Research
3D-Printed Materials and Systems
4OR
AAPG Bulletin
AAPS Open
AAPS PharmSciTech
Abhandlungen aus dem Mathematischen Seminar der Universität Hamburg
ABI Technik (German)
Academic Medicine
Academic Pediatrics
Academic Psychiatry
Academic Questions
Academy of Management Discoveries
Academy of Management Journal
Academy of Management Learning and Education
Academy of Management Perspectives
Academy of Management Proceedings
Academy of Management Review
Hardwicke, Tom E.; B. Mathur, Maya; MacDonald, Kyle; Nilsonne, Gustav; Banks, George C.; Kidwell, Mallory C.; et al. (2018). Supplementary material from "Data availability, reusability and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition". The Royal Society. Collection. https://doi.org/10.6084/m9.figshare.c.4175039.v1
or
Select your citation style and then place your mouse over the citation text to select it.
SHARE
Usage metrics
AUTHORS (14)
TH
Tom E. Hardwicke
MB
Maya B. Mathur
KM
Kyle MacDonald
GN
Gustav Nilsonne
GB
George C. Banks
MK
Mallory C. Kidwell
AH
Alicia Hofelich Mohr
EC
Elizabeth Clayton
EY
Erica J. Yoon
MH
Michael Henry Tessler
RL
Richie L. Lenne
SA
Sara Altman
BL
Bria Long
MC
Michael C. Frank