Replications in the social sciences: New study confirms ongoing challenges

Much has been said on the importance of replications. Recently, nature has published another comment that deals with this question. Paul Gertler, Sebastian Galiani and Mauricio Romero (GGR) conducted a survey, in which they focussed on the fields of economics, political science, sociology and psychology. They conclude that ‘the current system makes original authors and replicators antagonists.’

They found, that in the top-tier economics journals only few articles are replications – and all of those refute the original results. That said, GGR also asked 35 editors and co-editors of these economics journals about their perceptions towards publishing replications. While all editors who responded would publish a study that refutes the original findings, only a fourth can image to publish a study which confirms the original results. Read the rest of this entry »

New working paper: “Is Economics Research Replicable?”

Andrew Chang and Phillip Li, two researchers working at the Board of Governors of the Federal Reserve System and at the Office of the Comptroller of the Currency/U.S. Department of the Treasury, attempt to replicate 67 papers published in 13 well-regarded economics journals (American Economic Journal: Economic Policy, American Economic Journal: Macroeconomics, American Economic Review, American Economic Review: Papers and Proceedings (P&P), Canadian Journal of Economics, Econometrica, Economic Journal, Journal of Applied Econometrics, Journal of Political Economy, Review of Economic Dynamics, Review of Economic Studies, Review of Economics and Statistics, and Quarterly Journal of Economics), using author-provided replication files that include both data and code.

Some journals in the sample of Chang and Li require data and code replication files, and other journals do not require such files. Aside from 6 papers that use confidential data, they obtain data and code replication files for at least 29 of 35 papers (83%) that are required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that are not required to provide data and code replication files. Read the rest of this entry »

Open Access to Research Data: Strategic Delay and the Ambiguous Welfare Effects of Mandatory Data Disclosure

Patrick Andreoli-Versbach and Frank Mueller-Langer (two economists from the Max Planck Institute for Innovation and Competition in Munich) have published a paper on strategic delay and the ambiguous welfare effects of mandatory data disclosure (2014).

By setting up a model describing the incentives of a researcher to share self-generated data with the research community in the context of a published article, they found that these incentives can be distorted by a policy they call “First Paper Policy” (requires authors to share their data immediately after the first publication): Read the rest of this entry »

EDaWaX-Study on Data Policies of Journals in Economics and Business Studies (Part I)

One work package (WP2) of EDaWaX’s second funding phase deals with a broader analysis and comparison of journals’ data policies in economics and business studies. In the project’s first funding phase we already have conducted a similar survey, but it primarily focused on journals in economics.

Because research data and methodology in business studies are not necessarily identical to those employed in economics, we found it to be important to compare journals’ data policies of both branches of economic research. Read the rest of this entry »

Introducing the OJS-Dataverse Integration Project

Today I want to introduce the PKP- Open Journal System / Dataverse Integration Project to our readers, an approach that is in some parts quite similar to our own approach in the field of economics.

Funded by a $1 million Alfred P. Sloan Foundation grant, the OJS-DVN project intends to develop a plugin for journals that are using the Open Journal System (OJS), a journal management and publishing system that has been developed by the Public Knowledge Project (PKP), to expand and improve access to research. Read the rest of this entry »

JoRD Project presents Results

The JoRD-project (Journal Research Data Policies ) recently has published some final results. As mentioned in one of my previous blog posts, the JoRD-project -which was funded by JISC as part of its Digital Infrastructure Programme- aims to conduct a feasibility study into the scope and shape of a sustainable service that will collate and summarise journal data policies. The purpose of the service would be to provide a ready reference source of easily accessible, standardised, accurate and clear guidance and information, on the journal policy landscape relating to research data. Read the rest of this entry »

Research Data Management in Economic Journals (Part I)

This article is cross-posted from the blog of the Open Economics Working Group

Background

In Economics, as in many other research disciplines, there is a continuous increase in the number of papers where authors have collected their own research data or used external datasets. However, so far there have been few effective means of replicating the results of economic research within the framework of the corresponding article, of verifying them and making them available for repurposing or using in the support of the scholarly debate.

In the light of these findings B.D. McCullough pointed out: “Results published in economic journals are accepted at face value and rarely subjected to the independent verification that is the cornerstone of the scientific method. Most results published in economics journals cannot be subjected to verification, even in principle, because authors typically are not required to make their data and code available for verification.” (McCullough/McGeary/Harrison: “Lessons from the JMCB Archive”, 2006)

Harvard Professor Gary King also asked: “[I]f the empirical basis for an article or book cannot be reproduced, of what use to the discipline are its conclusions? What purpose does an article like this serve?” (King: “Replication, Replication” 1995). Therefore, the management of research data should be considered an important aspect of the economic profession.

Read the rest of this entry »

Requirements for Data Availability Policies to enable Replications

In our analyses for work package 2 we collected some criteria to evaluate the quality of the data policies we found in our sample.

It was important to identify some core requirements that aim to ensure the replicability of economic research.  This was not an easy task, because we had to find some criteria that are suitable for many fields of research in economics.

Therefore we consulted several research papers and used the recommendations we found in the papers as a basis for analysing and assessing the suitability of data availability policies of economics journals in our study.

We’d like to discuss these criteria with our readers. Feel free to submit comments or send me an e-mail.

Read the rest of this entry »

Second Phase of our Survey just started

The EDaWaX Project today has send a reminder to 350 editors of economic journals to fill out the online questionnaire dealing with data availability policies in scholarly journals. One of the questionnaire was designed for journals without data policies; the second questionnaire was created for journals that already own a data policy.

Read the rest of this entry »

EDaWaX-Project launches Online-Surveys for Editors of Economic Scholarly Journals

The EDaWaX Project today has launched two online questionnaire for editors of scholarly journals. One of the questionnaire was designed for journals without data policies, whereas the second questionnaire was created for journals that already own a data policy.

Read the rest of this entry »

Data Availability Policy: American Economic Review

As announced in my previous blogpost, I ‘m starting the presentation of some data availability policies and replication policies with the American Economic Review (AER). The AER is a flagship of the economic profession and one of the top ranked journals in this scientific discipline.
The AER was published in 1911 for the first time. Only 7 – 10 percent of the submissions are accepted and later on published.

The AER adopted a so called replication policy in 1986 – despite the fact that studies (for example by Dewald, Thursby and Anderson) already claimed, that a replication policy is not enough to promote replicable results.
In their policy, the Review pledged authors to provide datasets and code for processing the data to other scientists that are interested in replicating the results on request.

Replication policies have often failed, even if the corresponding author is willing to support other researchers…and I imagine that this szenario is not very common …After publishing an article, authors mostly don’t have incentives to prepare the data and code for other researchers. It costs time and the rewards the scientific system pays for sharing data often are marginal.

Read the rest of this entry »