Data Availability Policy: American Economic Review

Posted: February 9th, 2012 | Author: Sven | Filed under: Data Policy, EDaWaX | Tags: Data Policies, Data Sharing, Replication | 1 Comment »

As announced in my previous blogpost, I ‘m starting the presentation of some data availability policies and replication policies with the American Economic Review (AER). The AER is a flagship of the economic profession and one of the top ranked journals in this scientific discipline.
The AER was published in 1911 for the first time. Only 7 – 10 percent of the submissions are accepted and later on published.

The AER adopted a so called replication policy in 1986 – despite the fact that studies (for example by Dewald, Thursby and Anderson) already claimed, that a replication policy is not enough to promote replicable results.
In their policy, the Review pledged authors to provide datasets and code for processing the data to other scientists that are interested in replicating the results on request.

Replication policies have often failed, even if the corresponding author is willing to support other researchers…and I imagine that this szenario is not very common …After publishing an article, authors mostly don’t have incentives to prepare the data and code for other researchers. It costs time and the rewards the scientific system pays for sharing data often are marginal.

In addition many researchers to not want to share their data with competitors, especially when the data might be worth generating more than one publication. And of course some scientists fear that others might find failures within their computations – another disincentive for sharing data and code…But incentives are another story. I promise to come back to you with a well-founded analyse on incentives for data-sharing when our partners from the International Max Planck Research School for Competition and Innovation (IMPRS-CI) have concluded their report for the EDaWaX workpackage 1.

But back to the AER: The results weren’t really surprising when McCullough and Vinod tried to replicate all empirical articles of a single issue of the AER  in 2003: half of the authors failed to honour the AER replication policy.
As a reaction, the Review implemented a mandatory data and code policy in 2005 and some of the top journals followed.

In summer 2008, the Review conducted a project to evaluate the quality of the data and code stored in its online data archive. The study, written by Philip Glandon, is also available in the Download-section of this blog. He found out, that 79% of the 39 investigated articles believed replicable without contacting the authors. This is a very good result, compared to all other studies that had tried to replicate the results of research published in economic scholarly journals.

So the question is: Why is the data availability policy of the AER such a strong policy?

In my opinion, there are a few reasons for it…

1. The AER is a top journal. Every author might feel very happy when a submission is accepted and he or she will do a lot to comply with the policies of this journal. This is a big difference to many others where authors might ponder the efforts for preparing the data on the one hand and their personal outcome on the other hand.
   If a similar ranked journal offers the possibility of publishing the paper without complying to a data policy, a researcher might chose the second option.
2. The Review’s data policy is mandatory.
3. The Review has a  very detailed data availability policy – for simulations and econometric papers as well as for experimental papers. The author knows what kind of data and materials he has to submit prior to publication of his article.
4. The quality of the policy. Not only datasets are required for making available to everybody, but also the code, programs and a detailed readme-file are mandatory parts of the submission.
5. Last but not least, the polices defines a procedure what authors have to do, when their paper relies completely or in parts on proprietary or confidential data. In Economics, where proprietary data is not seldom, a defined procedure even for these cases is reasonable.

So far for today – a long post for introducing the first policy…others will follow!

 

Graphics:
– photo:Viktor-Mildenberger_pixelio.de
– Table I: Philip Glandon: Report on the American Economic Review Data Availability – Compliance Project, Vanderbilt University, November 2010

 

 

 

---