Reimagining the Path to Reproducibility for Cancer Research

Irreproducibility in medical research can cause undue harm to human subjects and also lead to wasteful financial and opportunity costs. For example, on the basis of a nonrandomized, nonblinded preclinical trial in a mouse model of amyotrophic lateral sclerosis (ALS; ref. 1), the National Institutes for Neurological Diseases and Stroke (NINDS) sponsored a minocycline clinical trial in 412 patients with ALS from 2003 to 2007. This trial failed, prompting the NINDS and the broad scientific community to reexamine the standards for preclinical research (2).

It is particularly notable that the number of retracted scientific publications increased by 10-fold from 1975 to 2012 (3), with retractions between 1992–2012 estimated to carry a cost of approximately $58 million (4). Several commentaries from industry in 2011 and 2012 also raised the specter of irreproducibility in preclinical cancer research and prompted the research community to take action. This led to the Reproducibility Project: Cancer Biology (https://www.cos.io/rpcb) and provoked the ire of the 114th U.S. Congress (Sec 116 of S.3084 – American Innovation and Competitive Act) that led to a recent National Academies report on reproducibility and replicability in science (https://www.nap.edu/catalog/25303/reproducibility-and-replicability-in-science).

Sources of irreproducibility include flawed study design, nontransparent reporting of the methods used, incorrect choice of statistical methods, variability in research materials including mistaken cell line identities, and scientific misconduct. Although the Reproducibility Project sponsored by the Center for Open Science aims to replicate and highlight flaws of published studies, the sources of irreproducibility still need to be codified and made actionable. In this regard, in the aftermath of the failed ALS minocycline trial launched on the basis of a flawed preclinical study, Landis and colleagues (2) called for transparent reporting of experimental methods for preclinical studies. They proposed standards that included measures for randomization, blinding, sample-size estimation, and data handling.

The quality and identity of research reagents can contribute significantly to reproducibility. Chemical impurities in research reagents or approved drugs can have measurable biological impact. Significant concerns arose when MacLeod and colleagues (5) reported a survey of cell line cross contamination in 252 human tumor cell line cultures and found 18% of cell lines were contaminated predominantly with Hela cells. This led to the establishment of the International Cell Line Authentication Committee (https://iclac.org) registry of contaminated sources in 2012 and the Cellosaurus database (https://web.expasy.org/cellosaurus/) in 2018 covering over 10 × 10⁴ cell lines, and the use of Research Resource Identifiers (https://www.rrids.org) that include 1.5 million registered antibodies.

Responsible Conduct of Research (RCR) is a cornerstone for advances in our scientific knowledge (National Academies, https://www.nap.edu/catalog/21896/fostering-integrity-in-research). The lack of research integrity can profoundly affect reproducibility. Potential factors affecting RCR were studied by comparing retracted and corrected articles with control articles matched by journal and issue published in 2010–2011 (6). The studied factors included institutional or national RCR policies, culture, peer mutual criticism, pressure to publish, early career ethos, and gender. The authors reported that misconduct was more likely in countries that lacked research integrity policies or provided cash for individual publication performance. Cultures that hampered mutual criticism were also associated with higher cases of research fraud.

RCR can be enhanced by promoting research integrity policies, improving mentoring and training, and encouraging transparent communication among researchers. The major reporting key elements required to improve transparency and reproducibility in experimental biology were codified by the MDAR (materials, design, analysis, reporting) Working Group that provided a checklist for authors (https://osf.io/bj3mu/). A study of articles published in two journals in 2013 (prechecklist) compared with those in 2015 (after checklist) revealed improved transparency of preclinical animal studies (7).

It is important to note that implementation of checklists can be burdensome and distracting for authors. However, due to technological advances, software services are now available that can analyze scientific manuscripts for sources of irreproducibility. Specifically, the availability of a software that can analyze the “Materials and Methods” section for compliance can reduce some of the burdens authors may face when implementing checklists during the submission of manuscripts. To this effect, the AACR Journals have initiated a pilot with the software service SciScore (https://www.sciscore.com) that analyzes the “Materials and Methods” section of the submitted article for common checklist items. This change will hopefully foster an ethos of research integrity and create a means to catalyze transparent disclosure of research agents and methods.

Common checklist items (such as cell lines, reagents, animals, human subjects, study, and laboratory protocols) relevant to cancer research only address a portion of the transparency and reproducibility challenges facing the community. Increasingly, published research relies on intensive and complex data analysis mediated by custom algorithms implemented in computer code. This can present difficulties for both the peer-review process and interpretation and replication of the published results. To help address this challenge, the AACR Journals have integrated their submission system with Code Ocean (https://codeocean.cm), a research collaborative platform that allows users to develop, share, and publish code alongside the data it was used to analyze. It provides direct cloud computing access of deposited code and data via digital object identifiers for peer-review and catalyzes ease of “reuse” and reproducibility of software and algorithms.

Reproducibility is paramount for optimized, efficient use of research resources, public trust, and advancements that ultimately help patients and relieve the burden of cancer. With an ethos of research integrity and the means to catalyze transparent disclosure of research agents and methods through implementation of SciScore and Code Ocean, we can together reimagine a path to discern cancer research truths that change lives and diminish the costly impact of irreproducibility.

No potential conflicts of interest were disclosed.