Bridging the Gap: Moving Predictive and Prognostic Assays from Research to Clinical Use Free

Despite enthusiasm on the part of patients, providers, and clinicians regarding the use of predictive and prognostic biomarkers, to date few assays have met the standards needed to convince the clinical community that they can be used to guide treatment decisions that will improve outcomes (1–5). In 2008, the President's Office of Science and Technology Policy (6) recommended that the National Institutes of Health “develop a funding program for academic/industry collaborative projects addressing biomarker standardization, statistical methods, and other aspects of study design necessary for validating the clinical utility of molecular diagnostics based on genomic correlations with disease characteristics,” and that the U.S. Food and Drug Administration (FDA) develop “standards for study design and product performance with regard to regulatory review of new diagnostic products” as well as a “regulatory approach to codevelopment of diagnostics and therapeutics.” These statements were strongly supported by a combined effort of the FDA, American Association for Cancer Research, and National Cancer Institute (NCI), known as the Cancer Biomarkers Collaborative [CBC (7)]. The CBC supported >27 recommendations covering biospecimens, reference standards, analytical performance, standardization and harmonization, bioinformatics, collaboration and data sharing, regulatory issues, policy, and education. But what are the major issues that prevent wider use of molecular diagnostics in cancer patients? Recent reviews have defined several issues related to the complexity and heterogeneity of cancer, lack of appropriate tumor specimens, bias inherent in the assay platform or analysis, study design issues, analysis and interpretation of results, lack of appropriate controls or standards applicable to complex assays, and technical validation of assays (8). Elsewhere in this Focus issue, Poste and colleagues (9) and others (10, 11) describe aspects of this problem by defining the different types of markers and their uses in clinical trials, as well as differences between the identification of markers during discovery and clinical assay development. In addition, Schilsky and colleagues (10) and others (11, 12) describe the use of integral markers that are essential for the performance of a trial, such as identification of patients as candidates for a therapeutic, risk stratification, modification of the dose of a therapeutic, or some other aspect of medical decision-making. The use of such integral markers requires validation of the assay's analytical performance so that the assay can be performed in a laboratory accredited under the Clinical Laboratory Improvement Amendments (CLIA) of 1988. In this article we review the background of analytical validation of a clinical assay, discuss why a complete validation of most molecular diagnostic assays is currently not possible, and describe a new NCI pilot program that is designed to help predictive and prognostic assays move from research to clinical readiness.

The development of a prognostic or predictive assay that can guide treatment for cancer patients is usually an iterative process that begins with the discovery of a molecular characteristic or signature and preliminary correlation with a clinical outcome. Subsequent steps involve optimization of the assay (including technical or analytical validation of the assay), clinical validation (clinical qualification) or evidence that the result of the analytically validated assay correlates with the clinical outcome of interest, and assessment of clinical utility for the intended use [i.e., does use of the assay result in a better outcome than standard methods or treatment? (Fig. 1)].

The pharmaceutical industry has incorporated markers into its workflow for drug development (11, 12). However, at a workshop in 2003 (13), members of the American Association of Pharmaceutical Scientists, the Ligand Binding Assay Bioanalytical Focus Group, and the U.S. Clinical Ligand Society suggested that analytical validation of assays for markers should not follow the same process employed for bioanalytic assays commonly used by industry. Instead, the first step in assay development should be to define the intended clinical use for the result of the assay. For predictive and prognostic assays, most clinicians consider the intended use to reflect how the marker will assist medical decision-making in a specific clinical situation. The workshop participants suggested that, for the clinical laboratory scientist, a “useful starting point in determining the direction of a biomarker assay validation is to consider the assay to be used and the data type that the assay will generate” (13). Their report defined several different types of assays depending on the type of data obtained. Definitive quantitative assays are those in which the result is a continuous number expressed using a definitive, approved, or certified reference standard. An example is human insulin. Relative quantitative assays are those in which the output is a continuous number but the reference standard is not well characterized or fully representative of the endogenous biomarker. This category constitutes the great majority of clinical biomarkers for which the reference standard is not approved or certified by a regulatory body but is added to the matrix of the sample in which the clinical assay is to be performed. This “spike in” can be affected by matrix components that interfere with detection of the analyte, and hence lead to only relative accuracy. Quasi-quantitative assays are defined as those that also have an uncertified reference standard but are expressed in relation to a baseline characteristic of the sample. A fourth type of assay, which is very common in clinical practice, is the qualitative assay, in which the results are presented in categorical terms–either as numbers (ordinal) or in non-numeric (nominal) form. For example, in the immunohistochemistry (IHC) assay, results are usually expressed as low, medium, or high, or as 1+ to 3+. These assays generally lack calibrators but may have standards for the different categorical values that are usually not certified by a regulatory body. This report (13) formed the foundation upon which the development of fit-for-purpose markers was initially based.

Subsequently, Lee and colleagues (14) expanded this report to describe in greater detail what is meant by “fit-for-purpose,” which generally means ensuring that the assay performance meets reasonable expectations for its intended use. Lee and colleagues (14) outlined the steps necessary for an iterative approach for assay improvement that focuses on the intended use of the assay, and method development and validation intended to meet that purpose. They focused on assays for biomarkers as a component of drug development, as opposed to the development of a clinical diagnostic. As a result, the intended use becomes critical and requires one to focus on developing an assay that is validated within the matrix expected for the intended use, as well as considerations of analyte stability rather than the stability of the reagents used in the assay. Another major difference is that a diagnostic performed for medical decision-making must be performed in a CLIA-certified laboratory, following standards provided by the Clinical Laboratory Standards Institute. Lee and colleagues (14) and others (15–19) also set out guidelines for the number of replicates needed to validate the performance of molecular diagnostic assays, as well as such considerations as the linearity of assay response, dynamic range, limits of detection, analyte stability within the intended matrix, and intra- and interlaboratory coefficient of variability (CV; Box 1). Although several of these authors prescribed how many replicates should be performed and the limits of allowable variability, these recommendations may need to be altered according to the type of assay and marker to be developed. In a previous Focus section, Chau and colleagues (16) recommended that validation of quantitative assays should include “at least five different concentrations of VS [validation samples] analyzed in duplicate on at least six different runs during the prestudy validation because quantitative biomarker assays often exhibit nonlinear calibration curves.” The reproducibility of an assay is generally measured by the % CV, which is defined as the standard deviation divided by the mean of the assay result expressed as a percent. Chau and colleagues (16) suggested that the % CV should be <25%; however, many clinical laboratory scientists will only accept much lower % CVs. The reproducibility of qualitative assays such as IHC is a distinct problem in that it is difficult to measure variation. For IHC assays, reproducibility is generally measured in terms of the κ statistic (22) and percent agreement (23) among different observers. The ideal level of agreement or concordance in such qualitative assays is unclear, although a level of agreement of ≥85% is considered to be acceptable.

An example illustrates this problem. The Eastern Cooperative Oncology Group (ECOG) led a randomized phase III stage II colon carcinoma adjuvant trial in which the 18q loss of heterozygosity (LOH) assay was an integral marker used to assign adjuvant therapy. The assay was based on the original description of 18q LOH by Jen and colleagues (24), who suggested that loss of 18q/DCC is associated with a poor prognosis (25) that might benefit from adjuvant therapy. Because the assay was not validated before initiation of the trial, the Cancer Diagnosis Program performed an interlaboratory validation study (26) that involved the reference laboratories for ECOG, the North Central Cooperative Treatment Group, and the Cancer and Leukemia Group B, all of which participated in this and other related phase III trials in colon carcinoma. All 3 laboratories used a single standard operating procedure, common Taqman primers and probes for the PCR assays, and similar ABI Prism equipment for this assay, which was performed in formalin-fixed paraffin-embedded (FFPE) diagnostic samples. All of these elements are considered to ensure concordance among observers. LOH is reported as a categorical variable, and interobserver analysis was performed with DNA extracts from 50 pairs of tumor (colon carcinomas) and normal tissue, tested by the 3 laboratories. The interobserver agreement was only 73% among the 3 laboratories, which is surprisingly low given the common DNA extracts as a starting point. Of interest, a second phase of concordance testing was performed when one of the collaborators, Dr. Thibodeau, suggested that the concordance could be improved by the individual reference pathologists using their own macrodissection techniques to prepare the DNA extracts. When this was done in a second set of >100 samples, the interlaboratory agreement improved to 92% (26). This experience suggests that even when almost all of the potential variables are standardized, there may still be variables that can affect the results, e.g., individual laboratory optimization of tumor content to increase enrichment of tumor DNA to improve detection of LOH.

Analytical validation of an assay involves assessments of its accuracy, precision, specificity (ability to detect the analyte in a complex matrix), and sensitivity. It is often difficult to validate novel biomarker assays due to the lack of a gold standard assay, and there may not be any approved reference standard that can be used to assist validation of the assay method. In addition, Chau and colleagues (16) remarked that only the definitive quantitative assay can be truly accurate because only with this type of method can the exact amount of the endogenous marker be measured, because there is a gold standard, certified quality control that is suited for the matrix of the clinical assay. All other categories of assays only estimate the accuracy of a marker with standards provided by the assay developer, but not at a standard that is approved by either the U.S. Pharmacopeia (USP) or the FDA (27). A reference material is defined by the National Institute of Standards and Technology [NIST (28)] as a material that is homogeneous and stable enough to be fit for use in a measurement process. A certified reference material is a type of “reference material characterized by a metrologically valid procedure for one or more specified properties, accompanied by a certificate that provides the value of the specified property, its associated uncertainty, and a statement of metrological traceability”. A NIST standard reference material is a “certified reference material issued by NIST that also meets additional NIST-specific certification criteria and is issued with a certificate or certificate of analysis that reports the results of its characterizations and provides information regarding the appropriate use(s) of the material.” The USP and the National Formulary list certified reference materials that meet the standards of the International Organization for Standardization (ISO) for reference materials in their ISO 17025 (29) and ISO Guide 34 (30) for reference material producers. Standard or certified reference materials are critical for interlaboratory harmonization efforts because they can serve as a common reference standard. Clearly, complex endogenous genes, proteins, and lipids can undergo many alterations during malignant transformation, and present a far greater challenge to the development of certified reference materials than the development of certified reference materials for analytes such as sodium, other electrolytes, and even enzymes for which clinical chemistry tests are routine and accepted. However, until this challenge is met, it will be difficult for both the clinical laboratory and the clinician to accept the accuracy of any laboratory-developed test that is not approved or cleared by the FDA or accepted by major healthcare insurance payers.

The Centers for Disease Control and Prevention (CDC), through its initiation and support of the GeT-RM project (31), has made reference materials publicly available through the Coriell Repository to support genetic tests. In addition to a number of reference materials for cystic fibrosis, Huntington disease, fragile X syndrome, and several genetic conditions with relatively high prevalence in the Ashkenazi Jewish population (32–35), it has also provided information for several pharmacogenetic markers, including members of the CYP450 gene family, VKORC1, and UGT1A1 (36). The GeT-RM is now providing reference materials for both genomic and somatic mutations that occur in hematologic and solid malignancies (37, 38).

An example illustrates the importance of certified reference materials. HER-2 testing in breast and other cancers can identify patients who might benefit from treatment that targets the gene (39). HER-2 expression is identified by IHC for protein, or FISH for gene amplification. However, studies have shown considerable discordance in assay results even between CLIA-certified laboratories testing the same samples (40, 41). The College of American Pathology/American Society of Clinical Oncology issued guidelines in 2007 (42) that cover all aspects of pre- and postanalytic procedures for the qualitative IHC and quasi-quantitative FISH assays, and recommended proficiency testing twice a year with standard controls contained in each assay run. Because the NIST does not provide a standard reference material for HER-2 or Her-2 in its catalog (43), and the USP does not have a HER-2 or Her-2 analyte in its compendium (44), there is no certified traceable standard for the IHC and FISH assays. As a result, the accuracy of the tests can only be estimated, although the precision of the assay can be measured. Similar situations exist with other FDA-approved or -cleared or commonly accepted diagnostics, such as ALK translocation. This will become an increasing problem as clinical trials successfully demonstrate the clinical utility of an integral marker, because demands for analyzing the marker in community hospitals in the same manner as it was tested in the trial will also increase. Unfortunately, it is very difficult to ensure that clinical laboratories will be able to provide the assay as it was performed in a clinical trial without standard or certified reference materials that trace back to specimens in the trial. The uncertainty raised in payers, clinicians, and patients makes this an important issue. In the future, with appropriate support from government agencies and the pharmaceutical industry, it may be possible to ensure that samples from pivotal trials are preserved to develop standard or certified reference materials for subsequent assay development.

Another major problem for scientists in academia or small business is the lack of funding to develop and validate clinical predictive and prognostic tests. The funding that supports the development of new clinical and biological discoveries vastly exceeds that which supports clinical assay development and validation. In fiscal year 2011, 0.8% of the total NCI budget was spent on development of clinically useful assays, as compared with 5.3% for identification of biomarkers (45). The actual difference between discovery and clinical assay development support may be considerably greater because it is difficult to determine how much of the molecular diagnostics portfolio is designated for development of assays within a CLIA-certified laboratory. The other major source of support for development and validation of clinical assays is diagnostics companies. These companies support the development of diagnostics for use in clinical cancer care, and/or collaborate with the pharmaceutical industry to develop companion diagnostics. There has been a recent increase in this activity (46–49), but this source of support is generally not open to academic investigators or even many small biotechnology firms.

Several larger academic cancer centers (e.g., the MD Anderson Cancer Center, Memorial Sloan Kettering Cancer Center, Dana-Farber Cancer Institute with the Massachusetts General Hospital and the Brigham and Women's Hospital, Vanderbilt-Ingram Cancer Center, Moffitt Cancer Center, and the Washington University Genomic Center) recently began to genotype patients who come to their institutions as a means of identifying patients who may be candidates for trials involving targeted therapy (50). These centers have resources to bring promising assays from the research laboratory to the CLIA-certified laboratory. The costs for these efforts are shared by the institution and the sponsors of the drugs that are under development. However, investigators at other institutions or small companies do not have access to these resources and may find it a difficult and bewildering process to convert their biologically important discovery into a clinically useful diagnostic. These investigators may have obtained biologically important findings, but may not be able to develop them into clinically useful tools unless they can turn to a clinical assay developer.

In response to the need for technically validated assays to be incorporated into NCI-supported clinical trials as integral assays, the NCI developed the Clinical Assay Development Program (CADP) and the Molecular Characterization Clinical Assay Development Laboratory (MC-CADL), The CADP is designed to overcome some of the hurdles faced by researchers during the analytic and clinical validation of a biomarker assay (see Table 1 for relevant definitions of terms). Integral assays are those that must be performed on every patient entered in the trial before the trial can proceed (10). These assays are used for patient selection (eligibility), stratification, and treatment assignment. It is especially important to perform analytical and clinical validation studies before using an assay in a clinical trial (6–10). Whether an Investigational Device Exemption (IDE) from the FDA is needed depends on the risk the assay poses to the patient as a result of its planned clinical use [see Meshinchi and colleagues (51) in this Focus issue]. Preparation for use of the assay in a CLIA-certified laboratory as a laboratory-developed test or for submission for an IDE will require careful assessments of accuracy, precision, specificity, and sensitivity. The goals of the CADP are to (i) provide the needed services to evaluate, optimize, and validate the analytical performance and clinical validity of promising assays that will be used in a clinical trial to demonstrate clinical utility; (ii) educate and engage researchers and others in a process of assay validation that leads to robust, clinically useful assays to guide cancer treatment; and (iii) develop and evaluate standard operating procedures, controls, and calibrators for clinical molecular characterization of malignancies. CADP is not a grants program; rather, it is a resource for investigators who want assistance in converting their research assay into an assay that is performed in a CLIA-certified laboratory. Investigators apply to the program through an electronic submission process described on the CADP website (52). Investigators with markers that have a defined clinical use and biologic rationale, and a research assay that functions in human tissues but whose performance needs to be improved and validated for use in a clinical laboratory are excellent candidates for the CADP. Assays that are further along in development but may need to be transferred to a CLIA environment, or may require additional refinement or a platform transfer, are also suitable for CADP. If approved, investigators will be provided access to the NCI's full suite of assay development and validation resources. Appropriate attention to the validation of these assays will increase the probability for success of the clinical trials in which they are employed, resulting in improved patient management.

The Clinical Assay Development Network (CADN) comprises 5 academic and 3 commercial (or reference) CLIA laboratories that are designed to optimize and analytically validate complex assays. These laboratories were selected competitively for their proven ability to convert research assays to clinical assays. Once an assay is selected for development by the CADP, each CADN laboratory bids according to its ability to develop that assay. After the choice of laboratory (or laboratories) is made, the investigator/assay submitter works with the CADP management team and members of the CADN laboratory to develop the assay, which is then returned to the assay submitter. The CADN is intended to be very interactive with the original investigator and may continue to collaborate with the investigator once the assay has been developed and validated. This may be very beneficial to the investigator who wants to use the assay in clinical trials but lacks the resources to do so directly in his or her institution. Available assay technologies in the CADN include IHC, FISH, chromogenic in situ hybridization, reverse transcriptase PCR, genomic sequencing, and LC/MS. Each assay to be developed will be assigned a project management team, which includes the assay submitter. The project management team is responsible for devising a timeline, milestones, and go/no-go decision points for the project.

The Specimen Retrieval System (SRS) is designed to provide needed specimens for analytic and clinical validation studies depending on the assay to be supported. It is anticipated that most tissue-based assays will use FFPE tissues, because they are the predominant form of diagnostic material currently available in U.S. hospitals (53, 54). The CADP has contracted with the Health Maintenance Organization, Kaiser Permanente Northwest, to provide such samples. This organization was selected because it participates in the Cancer Research Network of the Health Maintenance Organization Research Network and can provide FFPE tissues from pathology archives that use standard handling and storage procedures. Tissues are obtained from patients who have undergone surgery for a malignant disease, after appropriate Institutional Review Board approval has been obtained. In typical surgeries, more FFPE blocks are created than are needed by institutions for diagnosis. The excess blocks are deidentified and provided to the CADP, and the providing institutions retain the diagnostic blocks.

The specimens are accompanied by clinical data that is routinely abstracted by Tumor Registries [part of the quality management system of the Commission on Cancer (55)]. These clinical data are derived from pathology reports, clinic visits, oncology visits, and pharmacy records. These records are deidentified by scrubber software derived from an earlier NCI-supported program for the retrieval of FFPE specimens from diagnostic archives for research purposes (56). This software has been modified by investigators from Harvard working with clinicians at Kaiser Permanente Northwest (57) and is now >99% effective in removing all personal health identifiers and information. Records are still manually checked so that when specimens are forwarded to the CADP, the specimen and the patient's clinical history are deidentified. This enables the CADP to access specimens that fit the clinical context for assay development, as well as the clinical outcome information that is necessary to begin the process of qualifying assay results by correlating assay results with clinical outcome. The CADP may also contract with other sites as necessary for similar deidentified tissues (e.g., frozen tissue, blood, and plasma) if they are required for a particular assay.

The CADP provides advice on assay development, including potential development strategies, and focuses on an appropriate and feasible intended use for the assay, through the involvement of the personnel in the Cancer Diagnosis Program. Statistical advice is also available through the NCI's Biomedical Research Branch. In addition, the potential applicant may be referred to other potential collaborators (e.g., other academic investigators, Cooperative Groups, and other tissue resources) with which they can negotiate their own collaboration.

Academic researchers, commercial entities, and government researchers may apply electronically to the CADP for needed validation services (50). Applicants are asked to describe the potential clinical relevance of the proposed assay; must stipulate a single, well-defined intended clinical use; and must have a prototype assay that has been tested in relevant tumor tissue, as well as an estimate of the prevalence of the abnormality the assay will detect. Detailed data on additional previous work (if available) and relevant literature may be submitted as appendices. Assays that have progressed further toward validation are also encouraged. Applications are reviewed by a Special Emphasis Panel that is constituted under the Federal Advisory Committee Act and is composed of experts external to the NCI. Members of this committee include academic experts in disease, molecular biology, and diagnostics; members of industry with similar expertise; and patient advocates. Applications are evaluated on the basis of scientific rationale, feasibility, potential clinical impact, and plans for assessment of clinical utility (see Table 2).

Applications are then evaluated internally to ensure that the necessary resources can indeed be obtained by the NCI for the proposed assay or test. Unsuccessful applicants may resubmit the same project once and/or receive additional input regarding their efforts from CADP staff. Applications are accepted 3 times a year (see Fig. 2).

The MC-CADL is operated by SAIC-Frederick under contract to the NCI. Established to provide a means of translating genome discoveries into clinical applications, the MC-CADL will focus on newer genomic technologies such as whole transcriptome gene expression and next-generation sequencing. Of importance, the MC-CADL will apply these technologies using well-defined, robust protocols, and appropriate standards and controls to ensure reproducible results. The goal is to discern the molecular properties of a cancer that would explain a response (or lack of response) to the investigational targeted therapy. It will assist with early phases of assay development and transition to clinical laboratory readiness and support NCI-sponsored clinical trials within and outside the institute by developing standards and calibrators that can be used to compare assays among different laboratories or across platforms and can be shared with others. In this capacity, the MC-CADL will also work with the FDA, NIST, and other regulatory bodies that may develop standard and certified reference materials.

The MC-CADL will use FFPE specimens because they are the dominant form of tissues available throughout the country, and will facilitate adoption of developed assays. It will collaborate with the NCI's Office of Biorepositories and Biospecimen Research [OBBR (58)] and the Cancer Genome Atlas to study characteristics of specimens that affect the molecular characterization of malignancies, particularly in massively parallel sequencing platforms. As part of the CADP, successful efforts by the MC-CADL can be transferred into the CADN and/or the MC-CADL can serve as a reference laboratory for the CADN and the investigator.

Many challenges remain in the effort to incorporate molecular diagnostics into routine clinical practice. In addition to the performance of the assay, the association of the result with a clinical endpoint, and the demonstration that such an association will be clinically useful, there are the other concerns highlighted by the Office of Science and Technology Policy (6) and the CBC (7) that involve lack of appropriate biospecimens, standardization and harmonization, bioinformatics, collaboration and data sharing, regulatory issues, policy, and education. It is difficult to address all of these issues at once. However, the CADP addresses several of them because it enables independent academic and small-company investigators to develop clinically rigorous assays that then can enter clinical trials for clinical qualification and ultimately assessment of clinical utility. In addition to providing assistance with assay validation, members of the CADN may be able to assist investigators with supporting clinical assays in clinical trials. A major unmet need recognized by the Office of Science and Technology Policy (6) and the CBC (7) is the need for reference standards as a means to assess the accuracy of many molecular diagnostic assays. The MC-CADL is also beginning to address the reference issue through the creation of resources that may be able to be certified and distributed as standard or certified reference materials. It is hoped that as we move forward in the next few years, certified reference materials will become available so that investigators can actually measure the accuracy of tests, and correlation of results across different laboratories will be facilitated.

No potential conflicts of interest were disclosed.