The ReplicationWiki – an important resource for economists

The ReplicationWiki gives information about empirical studies in the social sciences. It offers a database containing empirical research and shows where to find program codes and data, which methods were used and which software. One can search for keywords, authors, databases or journals. In particular it lists so far 515 replications, 20 corrections, and 10 retractions; including information about the results of replications. 3,388 studies are already covered. The pages have been accessed more than 5.9 million times.

It helps researchers see which results have already been revisited. For instructors it helps to identify practical examples for which data and code are available in a software format that is accessible to their students. In our experience it helps to motivate students to study quantitative methodology and to approach published results with healthy scientific scepticism.

The ReplicationWiki also offers information on literature on the topic of replication, on journal policies on data availability, on data and software repositories and on projects that employ replication in the different social sciences.

Originally the Wiki was founded for economics. By now a number of studies from other social sciences were included, in particular political sciences and sociology. It has been cited as a project to follow also from fields like empirical law and empirical archaeology. Contributors are welcome! As the project is set up as a wiki website, any researcher can participate. After registration with one’s real name and an institutional email address one can for example add replications or studies that should be replicated, announce relevant events, discuss suggestions how to improve the project, set up one’s own user page and vote on which studies should be replicated. Registrations just to signal support for the project are also welcome. The ReplicationWiki is meant for an international audience and is therefore set up in English. Adding studies in other languages to its database is however welcome, and users can write in their own language on their user page to help others find them.

A detailed description how to add further studies can be found here: for replications and replicated studies and for studies that have not yet been replicated.

Further reading: ReplicationWiki – Improving Transparency in the Social Sciences, Jan H. Höffler, D-Lib Magazine, March/April 2017, Volume 23, Number 3/4, doi: 10.1045/march2017-hoeffler.

Picture: Image by Jan Vašek from Pixabay . License: click

Replications in the social sciences: New study confirms ongoing challenges

Much has been said on the importance of replications. Recently, nature has published another comment that deals with this question. Paul Gertler, Sebastian Galiani and Mauricio Romero (GGR) conducted a survey, in which they focussed on the fields of economics, political science, sociology and psychology. They conclude that ‘the current system makes original authors and replicators antagonists.’

They found, that in the top-tier economics journals only few articles are replications – and all of those refute the original results. That said, GGR also asked 35 editors and co-editors of these economics journals about their perceptions towards publishing replications. While all editors who responded would publish a study that refutes the original findings, only a fourth can image to publish a study which confirms the original results. Read the rest of this entry »

KNAW recommends making replication studies a normal and essential part of science

Over the past few years several systematic series of replication studies have been unable to reproduce many important scientific findings in a whole slew of disciplines. This has led to a debate within the scientific community about the way science is currently being conducted and the role of replication studies.  Our blog also published many posts which deal with this question.

Now, KNAW (the Royal Netherlands Academy of Arts and Sciences) has published a report, which analyses the causes of non-reproducibility, assesses the desirability of replication studies and also offers recommendations for improving reproducibility and for conducting replication studies. Read the rest of this entry »

Against the replication crisis: New international journal encourages replication studies

Read the rest of this entry »

American Economic Review publishes AEA’s Session Papers on Replication

Two weeks ago, the American Economic Review published the ‘Papers and Proceedings‘ of the 129th annual meeting of the American Economic Association (AEA) held in January, 2017.

At this year’s meeting, one session was dedicated to the topic of ‘Replication in Microeconomics‘ while another focussed on ‘Replication and Ethics in Economics: Thirty Years after Dewald, Thursby, and Anderson“.

In both sessions, very interesting and excellent papers were presented.

Below, I list all presentations of these sessions and the corresponding links to the papers (if available): Read the rest of this entry »

German Research Foundation (DFG) publishes Statement on Replicability

The German Research Foundation (DFG) has currently released a statement on the replicability of research results.

Interestingly (at least for me), the five-pager first starts with a broader definition of what replicable research is NOT.

Of course, replication is a very important method for testing empirical knowledge claims based on experimental and quantitative research in medicine, the natural, life, engineering, social and behavioural sciences, as well as the humanities.

But, according to DFG, there are limitations:

• Replicability is not a universal criterion for scientific knowledge.
• Ascertaining the replicability or non-replicability of a scientific result is itself a scientific result. As such, it is not final but subject to methodological scepticism and further investigation.
• Non-replicability is not a universal proof by falsification.
• Non-replicability is not a universal indicator of poor science.

‘Well, an unorthodox starting point for a paper on reproducible research‘ – so, at least, were my thoughts when I read the first page of the statement. Wouldn’t it be more common to first depict the important aspects of reproducible research and to suggest measures to support it, instead of rowing back at the beginning of such a statement? Read the rest of this entry »

New Working Paper: “Perceptions and Practices of Replication by Social and Behavioral Scientists”

One of our project partners has just released a publication that deals with the replication crises in economics and the social sciences.

In the abstract the three autors state:

We live in a time of increasing publication rates and specialization of scientific disciplines. More and more, the research community is facing the challenge of assuring the quality of research and maintaining trust in the scientific enterprise. Replication studies are necessary to detect erroneous research. Thus, the replicability of research is considered a hallmark of good scientific practice and it has lately become a key concern for research communities and science policy makers alike.

In their discussion paper Fecher, Fräßdorf and Wagner analyze perceptions and practices regarding replication studies in the social and behavioral sciences. Their analyses are based on a survey of almost 300 researchers.

Read the rest of this entry »

Contemporary, useful and subject-based: The replication network

Today I would like to introduce the replication network (TRN) to our readers, a network whose purpose is “to encourage economists and their journals to publish replications.” This is all along in line with the purpose of our own project.

The website of the replication network serves as a channel of communication to both update scholars about the state of replications in economics and to establish a network for the sharing  of information and ideas among economists. It offers important information on the possibility to publish replication studies in economics journals and provides lists of publications dealing with the topic of replications in economic research. Also a list of published replication studies is available. Read the rest of this entry »

New working paper: “Is Economics Research Replicable?”

Andrew Chang and Phillip Li, two researchers working at the Board of Governors of the Federal Reserve System and at the Office of the Comptroller of the Currency/U.S. Department of the Treasury, attempt to replicate 67 papers published in 13 well-regarded economics journals (American Economic Journal: Economic Policy, American Economic Journal: Macroeconomics, American Economic Review, American Economic Review: Papers and Proceedings (P&P), Canadian Journal of Economics, Econometrica, Economic Journal, Journal of Applied Econometrics, Journal of Political Economy, Review of Economic Dynamics, Review of Economic Studies, Review of Economics and Statistics, and Quarterly Journal of Economics), using author-provided replication files that include both data and code.

Some journals in the sample of Chang and Li require data and code replication files, and other journals do not require such files. Aside from 6 papers that use confidential data, they obtain data and code replication files for at least 29 of 35 papers (83%) that are required to provide such files as a condition of publication, compared to 11 of 26 papers (42%) that are not required to provide data and code replication files. Read the rest of this entry »

Biomedical Sciences: Journals unite to forster reproducible research

When it comes to the topic of replication, it always is a good idea to consult the webpages of the nature journal. Yesterday, for instance, the journal reported that a group of editors representing more than 30 major journals, representatives from funding agencies as well as scientific leaders discussed principles and guidelines for preclinical biomedical research in June 2014.

The gathering was convened by the US National Institutes of Health, Nature and Science.

The attendees agreed on a common set of principles and guidelines in reporting preclinical research that list proposed journal policies and author reporting requirements in order to promote transparency and reproducibility. Read the rest of this entry »

EDaWaX: First funding period terminates with evaluation workshop

A week ago our project held its final evaluation workshop. We presented the main results of some of our work packages and also introduced a beta version of our pilot application for the management of publication-related research data in journals.

In preparation of the workshop we invited more than 30 editors of scholarly journals and almost a dozen scientists from 15 journals accepted our invitation. Read the rest of this entry »

New Nature Special: Challenges in irreproducible Research

nature has published a new special issue on challenges in irreproducible research. The journal addresses the challenges and barriers of reproducibel research:

No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process. In studying complex entities, especially animals and human beings, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of further studies. Nature has published a series of articles about the worrying extent to which research results have been found wanting in this respect. The editors of Nature and the Nature life sciences research journals have also taken substantive steps to put our own houses in order, in improving the transparency and robustness of what we publish. Journals, research laboratories and institutions and funders all have an interest in tackling issues of irreproducibility. We hope that the articles contained in this collection will help.

All articles within this issue are available free of charge. The table of contents is available here.

Graphic: pasukaru76, www.flickr.com

RunMyCode.org – Make research easier to use and replicate

Last week Patrick, one of our project partners, made me aware of a very interesting website and service for researchers that is called runmycode.org. The concept of RunMyCode can be viewed as a novel attempt to provide  an executable paper solution.

Therefore I am very happy that Prof. Pérignon, one of the co-founders, has written a short introduction for our blog. If you would like to get more information about RunMyCode just visit the website or contact the team. Read the rest of this entry »

Requirements for Data Availability Policies to enable Replications

In our analyses for work package 2 we collected some criteria to evaluate the quality of the data policies we found in our sample.

It was important to identify some core requirements that aim to ensure the replicability of economic research.  This was not an easy task, because we had to find some criteria that are suitable for many fields of research in economics.

Therefore we consulted several research papers and used the recommendations we found in the papers as a basis for analysing and assessing the suitability of data availability policies of economics journals in our study.

We’d like to discuss these criteria with our readers. Feel free to submit comments or send me an e-mail.

Read the rest of this entry »

OSF- Reproducibility Project tries to replicate the results published in three psychological journals

“If you’re a psychologist, the news has to make you a little nervous…”. With this statement Tom Bartlett introduced his article  “Is Psychology About to Come Undone?” in the Chronicle of Higher Education.

The source of his fears is the Reproducibility Project  – a group of researchers that aim to replicate every study within the three journals Psychological Science, the Journal of Personality and Social Psychology and the Journal of Experimental Psychology: Learning, Memory, and Cognition published in the year 2008.

The project is part of Open Science Framework (OSF), a group that is interested in increasing the alignment between scientific values and scientific practices. Despite developing some tools and infrastructure projects its stated mission is to “estimate the reproducibility of published psychological science.”

Read the rest of this entry »