Do normative scientific practices and incentive structures produce a biased body of research evidence? The Reproducibility Project is the first known empirical effort to estimate the reproducibility of a sample of studies from the scientific literature. The project is a large-scale, open collaboration involving dozens of scientists from around the world.
The investigation is currently sampling from the 2008 issues of three prominent psychology journals - Journal of Personality and Social Psychology, Psychological Science, and Journal of Experimental Psychology: Learning, Memory, and Cognition. Individuals or teams of scientists follow a structured protocol for designing and conducting a close, high-powered replication of a key effect from the selected articles. We expect to learn about:
The overall rate at which peer-reviewed, published psychology studies produce replicable results
Obstacles that arise in gathering information about and replicating original study procedures
Predictors of replication success, such as the journal in which the original finding was published, the citation impact of the original report, and the number of direct or conceptual replications that have been published elsewhere
Aspects of a procedure that are or are not critical to a successful direct replication, such as the setting, specific characteristics of the sample, or details of the materials.
"If [our] discoveries are true, other people in other places should be able to rely on them, and build on them, to push further, and build a cumulative body of knowledge. These values are central to what it is, for me, to be a scientist, and the Reproducibility Project expresses those values."
"This project makes doing replications more rewarding (you get to work with excellent researchers, you learn a lot about how to do a good replication study, [...] if you don't know how to do something, there is help available). Showing this can be done, and documenting how a replication study should be done, will help other researchers in the future [...] and hopefully, give replications the place in journals they deserve."
"I hope that we will inspire researchers everywhere to think more carefully about the decisions that they make throughout the research process and to be more transparent and thoughtful about their research process [...]. I also hope that it will allow people to really appreciate the challenges and complexities of replications and the important value that replications serve for our field."
The Reproducibility Project is open to anyone who is interested in the reproducibility of psychological science or in participating in a large-scale, open science project. Contributors receive authorship on project reports (see Executive Summary for details). Ways to contribute include:
If you'd like to get involved, fill out our new contributor survey and we'll get in touch with you about opportunities that are a match for your skills and resources. We welcome contributions from students and citizen scientists as well as researchers and scientific professionals. If you have any questions, feel free to contact our volunteer coordinator at <a href="mailto:firstname.lastname@example.org">email@example.com</a>.
The Reproducibility Project is open and transparent. You can access background information about the project, materials about the study design, and information about the present progress and results. Our main policy discussions are carried out on an archived, publicly available mailing list. Some key links are below:
For a comprehensive list of Reproducibility Project resources and documentation, see our links and resources page.