I sent to one of my colleagues a link to Craig Newmark's post asking about the lack of replication studies in economics. From the authors of a new paper on the subject:

�This study arose out of our experiences attempting to replicate published empirical research and our concerns about the way unaudited research is used in public policy formation,� said Ross McKitrick, study co-author and Fraser Institute senior fellow.

�When a piece of academic research takes on a public role, such as becoming the basis for public policy decisions, then practices that obstruct independent replication prevent the proper functioning of the scientific process and can lead to poor public decision making.�

McKitrick, economics professor at the University of Guelph, and co-author Bruce D. McCullough, professor of decision sciences at Drexel University in Philadelphia, summarize replication efforts of more than 1,000 economics articles published since the 1980s. Most authors did not release their data when asked, and of those who did, only a small number of results were reproducible. McKitrick and McCullough also examined cases from a broad range of academic disciplines, including history, forestry, environmental science, health, and finance, in which influential studies were later found to be flawed, but only after lengthy battles to get access to the underlying data and, in some cases, years after policy decisions had been made based on the flawed results.

My colleague has a simple response to why it hasn't happened:

Everyone wants replication but no journal will publish the work and no department will value the work for tenure.

I think that's right, as far as it goes. I haven't read the paper yet, but it sounds like the six examples in Prof. Newmark's list are just the ones we found. How many more?

Labels: economics