Pitfalls in Cancer Biomarker Discovery and Validation with Emphasis on Circulating Tumor DNA

A longstanding dream of clinical biochemists and medical oncologists is to use simple blood-based tests to diagnose cancer early. Such a development would allow for more effective, targeted, and less toxic treatments and would constitute a major win in the war against cancer (1). On the basis of this premise, the U.S. NCI (Bethesda, MD) formed the Early Detection Research Network (EDRN), an organization with a mandate to discover and validate cancer biomarkers (2). EDRN's long-term goal is to bring biomarkers to the clinic, so that they can contribute to better patient care through disease prevention and personalized management.

In recent years, technological advances such as genomic, proteomic, and other omic profiling of tumor tissues and fluids have helped to elucidate multifaceted genetic drivers in cancer, revealed cancer heterogeneity, and provided clues for cancer evolution. These advances have highlighted the need for a new generation of biomarkers that capture the complicated and ever evolving proteogenomic landscape of tumors. Although many new technologies have been employed in this effort (3, 4), this research area has been met with limited success. We have not yet been able to bridge the age-old gap between initial biomarker discovery and subsequent clinical translation (5–8). The reality is that the current handful of clinically useful circulating cancer biomarkers were discovered 40 to 60 years ago (9).

The task of discovering new and improved noninvasive biomarkers for cancer, under the general umbrella of “liquid biopsy” (i.e., analysis performed in a tube of blood) has likely been underestimated. In this review, we will attempt to explain some reasons for these apparent failures, despite an abundance of funding, the use of revolutionary multi-omics technologies, the availability of biobanks, and the utilization of innovative bioinformatics solutions. We will focus on circulating cancer biomarkers. The field of molecular biomarkers, which are usually analyzed in tumor tissue for prognosis, selection of therapy, and predicting therapeutic response, is beyond the scope of this review. Because many difficulties in biomarker discovery and validation have already been discussed by our group and others (10, 11), this review will place special emphasis on circulating tumor DNA (ctDNA), the most promising new cancer biomarker (12–14). To drive positive changes in cancer biomarker research, we will also delve into the efforts of our group and others to promote scientific debates, with the idea that some current obstacles could be overcome with lessons learned from the past.

A literature search on cancer biomarkers will return thousands of hits. Why do many biomarkers that display promising in vitro results not make it to the clinic? There are two main reasons (15). One is that the performance of the majority of newly discovered biomarkers, as exemplified by their sensitivity and specificity, is often statistically significant (P < 0.05) when comparing study groups with benign lesions of similar tissue type versus malignant, cancerous tumors (15). Nevertheless, the new biomarkers are often clinically unhelpful. In other words, if the information provided by the biomarkers is either weak or not essential for clinical decision-making (nonactionable), clinicians will not use them. Second, even biomarkers that attempt to address a clear clinical unmet need, and show reasonable sensitivity and specificity, may not be adopted clinically if their positive predictive value (PPV) and negative predictive value (NPV) are below a clinically defined threshold. It is not immediately apparent to the nonspecialist, but the PPV of a test (the chances for having a disease if the test is positive) is dependent not only on the sensitivity and specificity of the test but also on the disease prevalence among the tested population. For relatively rare cancers (such as pancreatic and ovarian cancer, for which their prevalence in the tested population is <0.05%), a good screening test must be characterized by very high sensitivity (e.g., >90%) and even higher specificity (>99%) to yield a PPV of >10%. Furthermore, proving the clinical value of a new test requires detailed validation with large studies, which are expensive and time consuming (9). Consequently, diagnostic companies will be reluctant to license, validate, manufacture, and sell a test that has questionable clinical indications. In short, the qualification bar for tests aiming to migrate from discovery to clinical use is very high. This is especially true for tests designed for cancer screening programs. For more detailed descriptions on the characteristics of diagnostic tests and the requirements for their clinical use, please see ref. 15.

Beyond the complex logistics of introducing a new biomarker into clinical practice, there are often pitfalls in the initial discovery phase that threaten the reliability and utility of what could first appear to be an auspicious new marker. Upon closer examination, most candidate biomarkers identified from preliminary studies fail validation in subsequent, more rigorous studies. False discovery (defined as a discovery that was made by an error) occurs when an initial finding related to a new biomarker cannot be reproduced in independent sets of samples or by other laboratories. The main reasons for false discovery are study design and execution deficiencies/errors, including preanalytical, analytical, postanalytical, and bioinformatic shortcomings, as we discussed previously (5, 15). For example, poor-quality clinical samples can contribute to false discovery, even when using a meticulous study design (16). Bias can often play a significant role in false biomarker discovery (17). Bias can be defined as a systematic error due to an unfair sampling (i.e., site to site variations and confounding variables such as age that were not taken into account in study design and data analysis), or poor execution of an experiment (17).

False discovery is closely related to the contemporary and highly discussed problem of scientific irreproducibility (18–21). Some limitations that lead to false discovery and irreproducibility are very subtle and can only be identified years after publication of seemingly revolutionary data. Examples of this sort have been described by us elsewhere (5, 15). Raising awareness on false discovery and scientific irreproducibility are critical first steps in reducing some dangers and enhancing the quality in biomarker research. We recently proposed some ideas to reduce scientific irreproducibility and false discovery (18, 20, 21). In Table 1, we summarize some general suggestions which could minimize false discovery in biomarker studies.

The intricate nature of tumor biology is also to blame for the frequent failures in cancer biomarker research. In the past, cancer was considered to be a single disease. With the surge in various genomic methods such as whole-exome and whole-genome sequencing, and even more recently single-cell sequencing, we have learned tremendously about the vast landscape of tumor heterogeneity in recent years. The diversity of genetic mutations is markedly palpable between tumors of the same organ, within the primary tumor, and between the primary and metastatic tumors (22). Innovations in imaging and computer-vision technologies have allowed for the spatial mapping of cancer cells in histologic samples to visualize heterogeneity between and within patients (23). Aside from intertumor and intratumor variability, tumors evolve over time, resulting in temporal heterogeneity that often accounts for chemoresistance, metastasis, and recurrence (24, 25). More recently, numerous studies have demonstrated the presence of cancer stem cells that drive mutations and create vastly different genetic intertumor and intratumor mutational landscapes over time (26). Mounting experimental data has espoused the coexistence of genetically distinct cell clones in tumors, which drive clonal evolution giving rise to the heterogeneity (27). Revelations in tumor heterogeneity can be harnessed to our advantage for translational utility. Studies have begun to focus on combining molecular subtyping, genomic profiling, and clinical data with traditional biomarkers to predict clinical outcomes in different cancers (28–30). New studies have also examined the association of the extent of heterogeneity and molecular subtypes in tumors with drug resistance, recurrence, and metastasis (30). Along these lines, molecular characterization of tumors has also been exploited for customizing therapy selection and patient management (31–34). However, this new knowledge also suggests that it is highly unlikely we will find circulating cancer biomarkers that can be equally informative in all, or most, patients. More likely, tumor heterogeneity gives rise to biomarkers that are related to the mutational landscapes of each patient (see also below).

With increasing evidence on the high variability in tumor composition among patients with the same cancer type, there is a new sense of urgency to challenge the status quo of biomarker research. We propose to slowly shift from the orthodox discovery of biomarkers that are common for a cancer type to screening of serum samples of each patient, identifying the most informative set of markers for their unique tumor. We recently suggested this concept and coined the putative markers as “personalized” or “rare” tumor markers (35). We hypothesized, based on the status of the biomarker field over the past decades, that it is unlikely to unveil new biomarkers that perform with high specificity and sensitivity for all patients, and surpass the performance of already established and widely used biomarkers (such as PSA for prostate cancer or CA125 for ovarian cancer). There are hundreds to thousands of biomarkers in the literature that show high specificity but low sensitivity and these have not been further studied as a result (35). Nevertheless, compiling a library of such “rare” tumor markers, which are informative in only 5%–15% of patients, may offer a robust tool to screen patients to identify a custom set of markers that are most informative for their unique tumor (35). This approach could capture the heterogeneity of each tumor. In the future, personalized biomarkers, tailored to each patient, could find applicability in selecting targeted therapies, determining the extent of therapeutic response and monitoring for signs of recurrence.

The idea of finding noninvasive, circulating biomarkers that provide comprehensive information on the genetic makeup of each tumor is gaining traction. In the last few years, the emergence of genome sequencing, bolstered by the analysis of cell-free or circulating-free DNA (cfDNA) in the circulation, created considerable excitement. The potential is considerable for noninvasive cfDNA testing as one of the most promising new cancer biomarkers, particularly to pave the way toward personalized biomarker tests (36–42). cfDNA in the circulation can be derived from both normal and tumor cells, with the latter fraction known as ctDNA. By isolating and sequencing cfDNA using various genomic techniques, the small fraction of ctDNA can be detected and distinguished from cfDNA originating from normal cells based on genetic alterations, epigenetic modifications, or fragment length (Table 2). This distinction forms the basis for noninvasive pan-cancer detection tests, where blood-based samples can be screened for ctDNA in individual patients to assess tumor burden, as a form of liquid biopsy.

Numerous academic and commercial laboratories have focused on ctDNA testing and have developed sophisticated technologies for detecting the common genetic abnormalities associated with ctDNA, with impressive preliminary clinical results. The sensitivity of cancer detection, depending on cancer type, ranges from 50% to 70% at 90%–98% specificity. Recent revelations from CancerSEEK reported by Cohen and colleagues showed that combining information of genetic changes found in ctDNA with information from traditional circulating cancer biomarkers increases specificity to >99% and sensitivity up to 98% for some cancers (40). The advent of ctDNA holds potential for various important applications, including early diagnosis, prognosis, monitoring therapy success, and estimating tumor volume (43). Although these applications are now being slowly implemented in patient care, early detection of cancer using ctDNA markers has not yet reached the clinic. This underwhelming clinical application of ctDNA testing can be attributed to the general challenges mentioned above for circulating protein biomarkers. However, in this study, we wish to draw attention to some key issues that are particular to ctDNA as a cancer biomarker, with emphasis on early detection.

So far, all published investigations regarding cancer detection using ctDNA have reported sensitivity that was calculated from retrospectively collected samples with a mixture of patients with early- and late-stage cancers (40–42). These cancers were invariably diagnosed by clinical symptoms and imaging/biopsy. Thus, to better understand the ability of ctDNA tests to detect cancer preclinically, at an early stage, we previously used published empirical data to correlate tumor size with the expected amount of ctDNA in a 10-mL blood draw, mutant allele fraction (MAF; the percentage of ctDNA in comparison with the total cfDNA), and the likelihood of tumor detection using ctDNA genomic analysis (Table 3; ref. 13). We concluded that in general, these technologies are unlikely to detect tumors that are smaller than 10 mm in diameter; the test will likely suffer a high rate of false negatives in early-stage cancer (13). The major reason behind the false negative results associated with ctDNA testing is sampling error, where the retrieval of sufficient ctDNA from the circulation to allow for robust data analysis and interpretation is a limiting factor.

Similar to other cancer biomarkers, ctDNA testing may inevitably suffer from relatively low specificity in some cases. While investigators usually report sensitivities at 90%, 95%, or 98% specificity, these seemingly high thresholds may still not be enough when screening for less common cancers such as ovarian and pancreatic cancer. For example, if the cancer prevalence in the screened population is 1:400, a test with 100% sensitivity and 95% specificity will yield a PPV of 5%, meaning there is a 5% chance of a patient truly having cancer if the test is positive. This example highlights the well-known fact that screening tests must have extremely high specificity to ensure high PPV and avoid false positives. As we grow the genomic database of the healthy and cancer population, questions have been raised regarding just how cancer-specific are the genetic mutations found in ctDNA? Despite the expectation that cfDNA retrieved from normal individuals should not contain any genetic alterations, a plethora of new data shows otherwise (44). Mutations identified in normal tissues can be age related, may have functional significance in some tissues (such as the brain), and may be associated with precancerous lesions of unknown clinical significance (44). In addition to a questionable specificity to cancer, ctDNA analysis also usually cannot identify the affected organ/tissue and therefore cannot be used for diagnosis of specific cancer types (41). Cristiano and colleagues (41) reported a 61% chance of correct prediction of tumor localization, which is not far from tossing a coin. Therefore, a positive result by ctDNA analysis must be followed with a series of additional tests to corroborate the findings and identify the tumor origin. This could be a serious issue, in addition to the cost of individualized genomic analysis and downstream testing, because the uncertainty of cancer specificity and tumor localization may induce patient anxiety and other side effects.

Recently, it has been brought to attention the impact of clonal hematopoiesis, a precancerous condition during which hematopoietic cells acquire mutations, has on complicating ctDNA analysis and compromising specificity. Ptashkin and colleagues found that 5% of all mutations found in tumors are due to clonal hematopoiesis and they drew attention that some patients may receive inappropriate therapies due to such errors (45). On the other hand, Razavi and colleagues found that more than 50% of identified mutations in ctDNA are due to clonal hematopoiesis, thus necessitating sequencing of white blood cell DNA to verify the original mutations (46, 47).

New work has shown that cfDNA fragmentation patterns can be used to diagnose cancer (14, 41). In this newest twist, it is hypothesized that the fragment lengths of cfDNA could be exploited for noninvasive genomic analysis and early detection of cancer (14, 41). Fragment length and position within the genome are both informative, as assessed by low-depth genomic sequencing of multiple 5 Mb regions, dispersed within the whole genome (for mechanisms of DNA fragmentation, please see refs. 14 and 41). The important observation is that ctDNA is, on average, slightly shorter, by about 3–6 bases, and cfDNA fragments derived from tumors are more variable in length compared with fragments in normal controls (14, 41).This revelation created a paradigm shift, from investigating one or a few genetic changes found only in ctDNA, to examining the whole spectrum of cfDNA fragment length variation and position to infer the presence of cancer (14, 41). By employing cfDNA fragment analysis alone, the reported sensitivities in 236 patients with breast, colorectal, lung, ovarian, pancreatic, gastric, or bile duct cancer ranged from 57% to 99% at 98% specificity (41). The sensitivity could be raised to 91% by combining this method with mutation-based ctDNA analysis (14, 41).

Virtually every paper using cfDNA analysis for early cancer detection utilized clinically identified cancer cases and normal controls to calculate sensitivity and specificity. These approaches almost certainly overestimate the sensitivity of the test for early detection. Recent experimental data from the company GRAIL that wishes to develop a ctDNA test for cancer, can attest to these predictions and demonstrate the magnitude of the overestimation (48, 49). The GRAIL method was recently validated with clinically detected breast cancers and asymptomatic breast cancers detected by mammographic screening (49). These data have shown that the sensitivity drops by 4- to 5-fold, from 50% in clinically detected cancers to 10% in asymptomatic, mammographically-detected cancers (49).

GRAIL recently presented their latest validation data with an independent patient set at the inaugural American Society of Clinical Oncology (ASCO) Breakthrough conference in the fall of 2019 (50). It is important to mention that the patients in this set have also already been diagnosed clinically, so the experiment does not closely mimic a screening scenario which usually includes asymptomatic individuals. In this new validation set, the diagnostic specificity was raised to 99% (from 95%), a significant achievement. The overall sensitivity for multiple cancer sites was 55%, but for early-stage tumors (stage I) that are least likely to be detected clinically or through screening, the average detection rate (sensitivity) was only 18%.

Another way to validate the sensitivity of new technologies, such as cfDNA, for early cancer detection would be to analyze samples collected longitudinally, as part of prospective randomized trials such as the Prostate, Lung, Colorectal, and Ovarian (PLCO) Cancer Screening Trial (51). This large trial sponsored by the NCI is an example of an ideal sample set for determining the effects of screening on cancer-related mortality (primary end point) and other secondary endpoints (i.e., progression-free survival, etc.), in men and women ages 55 to 74 (51).

Under the auspices of EDRN, we used samples from the PLCO (180 ovarian cancer cases and 660 controls) to evaluate and rank 49 ovarian cancer biomarkers for early, preclinical ovarian cancer diagnosis (52). We found that none of these biomarkers were effective, alone or in combinations, in detecting asymptomatic disease (52). The conventional ovarian cancer biomarker CA125, which has about 80%–90% sensitivity in detecting clinical disease, exhibited less than 50% sensitivity in asymptomatic patients (52). This finding underlines the challenges of using biomarkers to detect cancer at an early stage, where individuals are usually asymptomatic. The lead time from asymptomatic to clinical diagnosis must also be long enough (e.g., more than 6 months), if earlier therapeutic interventions are expected to have some positive effect in overall and disease-free survival. Shorter lead times for certain cancers would render early detection useless, and even high-performing biomarkers for these cancers may be of low clinical value if there are no significant benefits in patient survival associated with early detection.

We previously correlated MAF (the abundance of mutant alleles associated with ctDNA in comparison with normal alleles found in the total cfDNA in a plasma sample) with other tumor characteristics such as size and weight (Table 3; ref. 13). We concluded that when there is less than one copy of ctDNA intermixed with 10,000 copies of normal cfDNA (MAF of 0.01%), tumor diagnosis is unlikely due to sampling error; in such case, there will be no retrieval of even a single copy of ctDNA in a 10-mL blood draw (13). At a MAF of 0.01%, we calculated that the tumor will be about 12 mm in diameter (13). In most published studies, the MAF is 0.1% or higher, which yields favorable assay sensitivity (39). However, the tumor diameter will likely be more than 27 mm with a MAF of 0.1%, and the tumor will be easily detectable by imaging. In the latest report by Cristiano and colleagues, the MAF of the cancers studied is not mentioned (41). Some studies stratify patients according to stage, but stage and tumor size do not always closely correlate, because some relatively large tumors remain localized and could be classified as early stage. We strongly recommend that in future studies, the MAF of all utilized samples is mentioned to provide an idea of tumor size, to assess the utility of the biomarker compared with imaging tests.

As research makes strong inroads into cancer detection, prognosis, and monitoring with the new field of ctDNA testing, the amount of sequencing data we accumulate grows exponentially, which allows for sophisticated analysis using artificial intelligence (AI) and machine learning. Studies have begun to use AI algorithms to combine not only information on MAF, but also cfDNA fragment length and position, to provide a holistic prediction of the presence of cancer (14, 39). In theory, cfDNA-based tests could work even if there is less than one representative complete genome in a 10-mL blood draw because the information is derived from a fraction of genomic DNA regions originating from normal and cancer cells (41). In general, this method covers approximately 2.6 GB of DNA (504 windows of 5 Mb each; approximately 20,000 sequencing reads per window; ref. 41). In total, the sequencing generates about 10,000,000 reads at a depth of about 1×. When the mutant allele fraction is 1:10,000 (0.01%), approximately 10,000,000 reads will represent normal DNA and only 1,000 reads will represent cancer-derived DNA. Although we do not know the details of the algorithm, one would expect that as the MAF decreases, the representation of the ctDNA will also decrease, along with the strength of the AI prediction algorithm. Consequently, it may be necessary to demonstrate the strength of the prediction algorithm at various MAF levels to avoid overestimation of ctDNA sensitivity and to understand the variability in the data. Although experimental data is lacking, we predict that at a MAF of 0.001%, when tumors are expected to be approximately 6 mm in diameter, the mutant DNA reads will be too few for AI algorithms to make an accurate prediction. All in all, coupled with sampling error, as we mentioned above, where ctDNA may not be retrieved at sufficient amounts in a 10-mL blood draw, the current uncertainty for AI programs to perform effectively at extremely low MAF (<0.01%) constitutes another roadblock of ctDNA testing for early cancer detection.

Is it a good idea to challenge questionable data? Science progresses faster if published results are debated in the appropriate fora. In most cases, differences in opinions originate from honest oversights that could produce false discoveries (data fabrication is a rare phenomenon). However, highlighting mistakes and flaws in published literature, especially reports on new and exciting biomarkers that seem to hold promise for revolutionizing cancer care, takes courage from the challengers, who put their reputation on the line. In short, there is no room for mistakes when criticizing other people's mistakes. For this reason, challenges of published data are not very common. On the other hand, those who published problematic data are very reluctant to accept their mistakes and retract their papers, due to the stigma and possible career consequences. Naturally, a large number of false reports still resides in the scientific literature.

In biomarker research, false discovery can be concealed by various explanations such as different assays, different algorithms, different samples, etc. At the end of the day, the ultimate judge is time. False discoveries never reach the clinic or can lead to start-ups that eventually fail, sadly wasting funds and energy. We provided many examples of presumed false discoveries in previous papers (Table 4; refs. 5, 15). Even though many of these “discoveries” have been disproved, the original papers have not been retracted and some are still highly cited.

One of our own proposals to reduce false discovery is the 5- or 10-year reflection, which mandates that authors of significant papers in biomarkers and other fields write a 5- to 10-year perspective on their initial discoveries (20, 21). While authors are very keen to speak to the media at the time of publication, they are often unwilling to examine the fate of their discoveries 5–10 years later, for obvious reasons. There is room for meta-analyses which can track the fate of seemingly revolutionary discoveries (including biomarkers) to identify the reasons for their ultimate failure and the lessons learned.

The road to new cancer biomarker discovery and validation is plagued with barriers that are hard to cross. Regardless of the newest and most advanced technology that is employed, judging the true value of a new biomarker returns to the familiar factors of clinical importance (unmet needs), sensitivity, specificity, PPV, and NPV for the cancer type at hand. The extensive interrogation of the cancer genome and tumor heterogeneity has opened a whole new realm for cancer biomarker research, where personalized biomarkers, in the form of a repository of protein or gene-based markers that can be used to screen each patient, may be the future.

Among these possibilities lies the new concept of ctDNA testing, where its potential applications as a groundbreaking liquid biopsy assay spans from cancer detection, assessing therapeutic response, to monitoring tumor growth and relapse. Equipped with a sharpened understanding of the utility and drawbacks of cfDNA as a new cancer biomarker, it is undoubtable that this field holds capacity for entering routine clinical care in the future if the current limitations are addressed appropriately with careful study design, larger datasets, and continuous technological progression in sequencing and bioinformatics methods. Regarding ctDNA, we suggest that the sensitivity should be determined using samples with a known MAF, with a focus on early-stage cancers, if possible, to examine its feasibility for early detection. In general, the specificity of new biomarkers should be assessed by including age- and sex-matched patients with benign and inflammatory diseases, and sample size calculations should take into account the prevalence of the cancer type.

Additional steps for validating new biomarkers would be to test prediagnostic samples available from prospective clinical trials like the PLCO, to calculate both sensitivity and lead time before clinical diagnosis. Furthermore, the lead time achieved with early detection will tell us whether patient outcomes can be influenced through earlier therapies. Although these stepwise approaches are intuitive to biomarker discovery, screening experiences, such as with ovarian cancer, have not supported this expectation (53). Another important issue is to make sure that screening, particularly in low prevalence cancers, avoids overdiagnosis and overtreatment, as outlined by us and others elsewhere (54–57). The best way to consolidate the success rate of new biomarker discoveries is to encourage respectful scientific debates surrounding new breakthroughs. The impediments encountered in biomarker development are less likely to hamper new discoveries if they are taken into account and implemented in the study design along each step of the pipeline. In the end, the recipe for fulfilling the mandate of the EDRN is an intricate balance of challenging common oversights and voicing concerns over flaws in research, which will lead to decreased incidences of false discovery as well as better usage of funding and time, by focusing resources on high-performance biomarkers that will eventually transform routine cancer patient care.

No potential conflicts of interest were disclosed.