Overview

Do normative scientific practices and incentive structures produce a biased body of research evidence? The Reproducibility Project is the first known empirical effort to estimate the reproducibility of a sample of studies from the scientific literature. The project is a large-scale, open collaboration involving dozens of scientists from around the world.

The investigation is currently sampling from the 2008 issues of three prominent psychology journals - Journal of Personality and Social Psychology, Psychological Science, and Journal of Experimental Psychology: Learning, Memory, and Cognition. Individuals or teams of scientists follow a structured protocol for designing and conducting a close, high-powered replication of a key effect from the selected articles. We expect to learn about:

1. The overall rate at which peer-reviewed, published psychology studies produce replicable results

2. Obstacles that arise in gathering information about and replicating original study procedures

3. Predictors of replication success, such as the journal in which the original finding was published, the citation impact of the original report, and the number of direct or conceptual replications that have been published elsewhere

4. Aspects of a procedure that are or are not critical to a successful direct replication, such as the setting, specific characteristics of the sample, or details of the materials.

What our researchers say about the Reproducibility Project

"If [our] discoveries are true, other people in other places should be able to rely on them, and build on them, to push further, and build a cumulative body of knowledge. These values are central to what it is, for me, to be a scientist, and the Reproducibility Project expresses those values."

"This project makes doing replications more rewarding (you get to work with excellent researchers, you learn a lot about how to do a good replication study, [...] if you don't know how to do something, there is help available). Showing this can be done, and documenting how a replication study should be done, will help other researchers in the future [...] and hopefully, give replications the place in journals they deserve."

"I hope that we will inspire researchers everywhere to think more carefully about the decisions that they make throughout the research process and to be more transparent and thoughtful about their research process [...]. I also hope that it will allow people to really appreciate the challenges and complexities of replications and the important value that replications serve for our field."

How to get involved

The Reproducibility Project is open to anyone who is interested in the reproducibility of psychological science or in participating in a large-scale, open science project. Contributors receive authorship on project reports (see Executive Summary for details). Ways to contribute include:

• Conducting or helping with a replication study
• Helping with statistical analyses or power / sample size calculations
• Reviewing and contributing to replication protocols or reports
• Coding published studies (extracting hypotheses, sample characteristics, and requirements for replication)
• Programming tasks such as creating stimulus presentation scripts or automated data processing

If you'd like to get involved, fill out our new contributor survey and we'll get in touch with you about opportunities that are a match for your skills and resources. We welcome contributions from students and citizen scientists as well as researchers and scientific professionals. If you have any questions, feel free to contact our volunteer coordinator at <a href="mailto:reproducibilityproject@gmail.com">reproducibilityproject@gmail.com</a>.

Project materials

The Reproducibility Project is open and transparent. You can access background information about the project, materials about the study design, and information about the present progress and results. Our main policy discussions are carried out on an archived, publicly available mailing list. Some key links are below:

• Executive Summary - An overview of the project's goals and approach
• Researcher Guide - Steps and tasks involved in carrying out a replication study
• Project Progress and Results Spreadsheet - Our central listing of available studies, replications in progress, and contributions by volunteers
• Google Discussion Group : The discussion list for the Open Science Collaboration (including the Reproducibility Project). Anyone can join the list or review the archives.
• Project Materials Dropbox: A repository of articles that are eligible for replication and other background reading materials (email <a href="mailto:reproducibilityproject@gmail.com">reproducibilityproject@gmail.com</a> for access)

For a comprehensive list of Reproducibility Project resources and documentation, see our links and resources page.