Article: The Economics of Reproducibility in Preclinical Research
Posted: June 15th, 2015 | Author: Sven | Filed under: found on the net, Research Data | Tags: articles, costs, journals, reproducibility | Comments Off on Article: The Economics of Reproducibility in Preclinical Research
In 2012, an estimated 114.8 billion $ in the US were spent on life sciences research. Roughly half of it is spent on preclinical research, with government sources providing the majority of funding – approximatly 38 billion US$.
Now, three researchers calculate the costs of irreproducible research in preclinical research near 28 billion $ – only for the United States alone. That is the conclusion of a study published in PLoS Biology a few days ago.
In the opinion of the study’s authors, the giant amount of 28 billion $ accumulates, because low reproducibility rates within life science research undermine cumulative knowledge production and contribute to both delays and costs of therapeutic drug development.
From time to time, discussions on reproducible research spark in almost every scientific field. Last year for instance, the nature magazine has published a special issue on reproducible research (“Challenges in irreproducible research“) and a growing number of research organisations, funders and also political bodies started to frequently adress the topic of reproducibel research.
Now this touchy issue is brought up in a paper in PLoS Biology (“The economics of reproducibility in preclinical research“) – estimating the costs of irreproducible research at 28 billion $ for the US.
In the foreword of their article the three researchers stated their motivation to publish their estimations about the costs of irreproducible research:
Flawed preclinical studies create false hope for patients waiting for lifesaving cures; moreover, they point to systemic and costly inefficiencies in the way preclinical studies are designed, conducted, and reported. Because replication and cumulative knowledge production are cornerstones of the scientific process, these widespread accounts are scientifically troubling.
Indeed, there are many different perspectives on the size of this problem. And Leonard P. Freedman, Iain M. Cockburn and Timothy S. Simcoe (the three authors of the paper) are not the only ones in the field dealing with the size of the market for irreproducible research. But – and that’s bad news- other estimations of irreproducibility range from 51% to 89% – a disastrous magnitude.
Replication is the cornerstone of the scientific method. Replicable research is subject to the scientific principle of verification; non-replicable research cannot be verified. To cite Prof. Gary King (IQSS – Harvard University)- “[…] if the empirical basis for an article or book cannot be reproduced, of what use to the discipline are the conclusions?”
Freedman at al. do not want to pinpoint the exact irreproducibility rate, but to estimate the direct costs of irreproducible research. Based on examples from within life sciences, application of economic theory, and reviewing lessons learned from other industries, the three authors conclude that
“community-developed best practices and standards must play a central role in improving reproducibility going forward.”
For their paper, the researchers adopt an broad definition of irreproducibility that encompasses the existence and propagation of one or more errors, flaws, inadequacies, or omissions that prevent replication of results. The team did not examine whether the results of any individual paper were reproducible. Instead, the researchers surveyed existing analyses of factors that contribute to irreproducibility, using these factors to estimate an overall reproducibility rate.
Overall, the team found that poor materials made the largest contribution to reproducibility problems (36%), followed by study design (28%) and data analysis (26%). The team estimates the overall rate of irreproducibility at 53%, but the researchers also states that the true rate could be anywhere between 18% and 89%. Based on these estimations the potential economic cost of irreproducibility ranges between $10 billion and $50 billion per year.
Several institutions are taking steps to address the problem. In November, for example, the US National Institutes of Health urged journals to adopt guidelines aimed at boosting reproducibility. This surely is a promising way to foster reproducible research – and also one of the goals of our own project.
Compared to economic research, there are comparatively numberous journals in life sciences which have a data policy – but also in economics, journals are increasingly implementing such policies as we found out in one of our work packages.
Picture: “Indian Lab Researchers Questing for an AIDS Vaccine” by AIDSVaccine on flickr.com. License: CC-BY-NC-SA 2.0
