New working paper: “Is Economics Research Replicable?”

Posted: October 7th, 2015 | Author: | Filed under: Data Policy, found on the net | Tags: , | 2 Comments »

400_ReplicAndrew Chang and Phillip Li, two researchers working at the Board of Governors of the Federal Reserve System and at the Office of the Comptroller of the Currency/U.S. Department of the Treasury, attempt to replicate 67 papers published in 13 well-regarded economics journals (American Economic Journal: Economic Policy, American Economic Journal: Macroeconomics, American Economic Review, American Economic Review: Papers and Proceedings (P&P), Canadian Journal of Economics, Econometrica, Economic Journal, Journal of Applied Econometrics, Journal of Political Economy, Review of Economic Dynamics, Review of Economic Studies, Review of Economics and Statistics, and Quarterly Journal of Economics), using author-provided replication files that include both data and code.

Some journals in the sample of Chang and Li require data and code replication files, and other journals do not require such files. Aside from 6 papers that use confidential data, they obtain data and code replication files for at least 29 of 35 papers (83%) that are required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that are not required to provide data and code replication files.

Chang and Li were able to successfully replicate the key qualitative result of 22 of 67 papers (33%) without contacting the authors. They found that their replication success rate is significantly higher when they attempt to replicate papers from journals that have a mandatory replication data and code submission policy.

Excluding the 6 papers that use confidential data and the 2 papers that use software the two researchers do not possess, they were able to replicate 29 of 59 papers (49%) with assistance from the authors.

Because Chang an Li were able to replicate less than half of the papers in their sample (even with help from the authors!), the two researchers assert that economics research is usually not replicable. They conclude their working paper with some interesting recommendations on improving replication of economics research, based on the outcome of their research.

The working paper is available here. Great to see that there is a growing community interested in the topic of replicable economics research!

Photo: own work. License: CC-BY-SA 3.0

2 Comments on “New working paper: “Is Economics Research Replicable?””

  1. 1 Sven said at 14:09 on October 21st, 2015:

    This working paper has also been discussed in the “Times Higher Education“: https://www.timeshighereducation.com/news/papers-in-economics-not-reproducible

  2. 2 Sven said at 14:11 on October 21st, 2015:

    This working paper is also mentioned in an ariticle published by David Matthews in the “Times Higher Education“: https://www.timeshighereducation.com/news/papers-in-economics-not-reproducible