My scientist friends have always been puzzled by my responses to questions about replicating studies in Economics. It’s just not done very often. In fields like astrophysics and biology, replication is almost as important, if not more important in some cases, as the novel finding itself, but not so in Economics. I’ve seen evidence that other social sciences are similar and there was some recent debate about the replication of psychology experiments and the failure to come to the same conclusions using similar methodologies. (There were other pieces on this, but this is one that I found today). In short, journals favor novel and interesting outcomes, so obvious or unsurprising results are far less likely to be published. The publication of the novel results leads to a power imbalance (she already published this, so she’s the expert and gets the soapbox). No one wants to fund or highlight research that’s already been done. Replications that confirm are boring and replications that challenge established findings have to be 110% on everything.
It’s really hard to challenge established findings. Look at how long (three years after publication) and how many papers it took for Emily Oster to admit her paper on missing women and Hepatitis B was wrong. Regardless, she still has a job and now tenure at Chicago. Or how many papers have been written challenging Donohue and Levitt’s abortion paper and they still stand by it.
I got a bit far afield, though. Economists are not generally in favor of duplication of effort. If someone’s doing it already, unless you can do it a lot better, you shouldn’t really do it. Hence persistent ideas of comparative advantage and gains from trade.
However, the recent spate of randomized control trials, particularly in development settings, has prompted more and more debate about the validity of these experiments and appears to have resulted in at least one group that’s eager to test and replicate in order to confirm (or deny?) the validity of certain projects.
Clearly, there are limits to what can be replicated using existing data, and limited funding to collect new data using similar methods.It’s unclear to me how they will choose appropriate experiments to reproduce or test, and as much faith as economists tend to put in a sample size of one, I’d bet we won’t be too happy with a sample size of two, but I think it’s a good start. The Development Impact Blog by the World Bank will keep up with the process of replication, so worth following if you’re interested. I know I’ll be watching.
h/t @JustinWolfers
Though kind of dated now, Daniel Hamermesh’s paper on replication in economics is here.
ERIN K. FLETCHER 