Lessons Learned from the Investigational Device Exemption Review of Children's Oncology Group Trial AAML1031

On December 14, 2010, Dr. Elizabeth Mansfield from the Office of In Vitro Diagnostic Device Evaluation and Safety (OIVD), Center for Devices and Radiologic Health (CDRH), U.S. Food and Drug Administration (FDA), announced at a workshop introducing the Clinical Assay Development Program of the National Cancer Institute [NCI (1)] that the FDA would now perform pre–Investigational Device Exemption (IDE) and IDE reviews for all integral markers in all NCI-supported clinical trials. An integral marker is a marker or diagnostic that is essential for the performance of a trial and is used to assign treatment, stratify risk, or determine the dose of a therapeutic agent, as further described by Schilsky and colleagues (2) in this Focus section. Concerns about the reliability and reproducibility of genetic markers that are used to guide medical decision-making have been publicly expressed (3). Thus, the FDA is concerned that patients may be harmed by novel diagnostics that are rapidly entering clinical practice and used for medical decision-making independently of corroborating clinical or pathologic evidence. Genetic tests in which an assay identifies a specific mutation in a gene for which there is a possible targeted therapy are a special concern because the test is the sole arbiter of whether the patient may be a candidate for the therapy, and there may be no ancillary clinical information that can assist the physician in making medical decisions.

The Children's Oncology Group (COG) randomized phase III trial in acute myelogenous leukemia (AML), AAML1031, was affected by this announcement because the trial had been approved by the NCI Cancer Therapy Evaluation Program (CTEP) and was in the last stages of review by the Pediatric Central Institutional Review Board, but now needed to undergo a pre-IDE review by the FDA before it could be activated for patient enrollment. The design of AAML1031 includes 3 genetic markers (described below) that are used, in part, to stratify patients into different risk groups to modulate treatment intensity and identify patients for potential hematopoietic stem cell transplantation (HSCT). Patients are randomized to different induction therapies, some of which involve investigational agents provided by the NCI under an Investigational New Drug (IND) application, based on these risk categories. In addition, after an initial cycle of induction therapy, patients are restaged to high- or low-risk cohorts based on the presence of minimal residual disease (MRD) as determined by multidimensional flow cytometry at a 0.1% blast threshold. Those with MRD would receive more intensive chemotherapy and HSCT from the most suitable donor, and those without MRD would continue with standard chemotherapy. The identification of these 3 genetic markers and the MRD assay as markers that are essential or integral for the conduct of the trial prompted the COG, NCI, and FDA to work together to expedite a pre-IDE review. The purpose of this article is to characterize the lessons learned during this first FDA pre-IDE review so that clinicians, investigational scientists, and clinical laboratory scientists can better understand the new processes and procedures related to the use of integral markers in clinical trials.

The FDA has a website (http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/default.htm) that is dedicated to defining the process and regulations surrounding approval and marketing of devices, with all of the information needed to go through a pre-IDE review and then a formal IDE application if a device poses a significant risk to patients. An in vitro diagnostic or “device” is defined by the FDA (4) as “an instrument, apparatus, implement, machine, … in vitro reagent, or other similar or related article … which is (1) recognized in the official National Formulary, or the United States Pharmacopeia, or any supplement to them; (2) intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease, in man or other animals; or (3) … does not achieve its primary intended purposes through chemical action within or on the body of man or other animals and which is not dependent upon being metabolized for the achievement of its primary intended purposes.” Thus, the purpose of the pre-IDE and IDE reviews is to ensure that assays for use in medical decision-making are reliable and reproducible, and that the risk-benefit of the use of the diagnostic is sufficiently safe to benefit the patient within its clinical context of use. The 3 genetic assays and 1 MRD assay in COG AAML1031 are in vitro diagnostics that, although they are not listed in the National Formulary or U.S. Pharmacopeia, are intended for use in diagnosis of disease and do not act by chemical or biological means within the body.

The pre-IDE review identifies the concerns the FDA may have for a formal IDE application. The pre-IDE review is an informal, nonbinding review that may take 60 days, but may require shorter or longer time. In contrast, an IDE application has a firm 30-day time limit (Fig. 1). The pre-IDE review has 2 components: an assessment of the analytical performance of the assay for the diagnostic, and an evaluation of the risk posed to the patient by use of the diagnostic. The assessment of the risk posed by the diagnostic has to be considered within the context of its use and whether the assay and diagnostic are “fit for purpose” (5, 6). Often the risk of false-negative or -positive assay results may seem small if a patient has an advanced cancer with only a few months to live. However, that risk becomes much greater if the patient is being treated with curative intent or has a long expected survival, as may occur in an adjuvant setting. No assay is absolutely free of errors, so the risk of false-positive or -negative results must be presented in the protocol and during the informed-consent process. The analytical performance of the assay determines the probability of a false-positive or -negative result and its risk.

After determining that one or more devices are present in a trial, the FDA will evaluate whether the devices pose a significant risk to the patient. A significant risk device is defined by the FDA [in 21 CFR 812.3(m)(3)] as one that has “a use of substantial importance in diagnosing, curing, mitigating, or treating disease, or otherwise preventing impairment of human health and presents a potential for serious risk to the health, safety, or welfare of a subject.” In addition, the FDA recently indicated (7) that “If [a diagnostic device is] used to make critical treatment decisions, such as patient selection, treatment assignment, or treatment arm, a diagnostic device is likely to be a significant risk device under 21 CFR 812.3(m)(3)…. In such cases, FDA will expect the sponsor to conduct the trial under full IDE regulations.” Thus, because integral markers will be used for medical decision-making for individual patients, they generally will be significant risk devices and require an IDE for performance of the assay in a clinical trial. In addition, the assay for such an integral marker will also need to be performed in a Clinical Laboratory Improvement Amendments (CLIA)–certified laboratory (2).

If a molecular diagnostic is an integral marker, the investigator and the clinical laboratory scientist need to ensure that the molecular diagnostic performs adequately in samples similar to those for its intended clinical use. Often a retrospective analysis is performed with samples that were collected as part of a previous clinical trial, and the assay's analytical performance is validated, including establishment of any cut-points. Critical considerations in this process include the reproducibility of the test, and, if assays are to be performed in more than one laboratory, the interlaboratory concordance between test results. This process of analytical validation of an assay is also required for any laboratory-developed test performed in a CLIA-accredited facility so that the performance characteristics of the laboratory-developed test will be well known to the clinical laboratory scientist who develops and performs the assay. The FDA's Device Advice website (http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/default.htm) provides investigators and clinical laboratory scientists with contact information so that they can contact the appropriate FDA staff to begin discussions about the pre-IDE review. Diagnostics whose assays are integrated (i.e., performed in all subjects in a trial or a statistically defined subset, but not used for decision-making) or for research (also not for decision-making in individual patients) are not subject to the FDA's pre-IDE review.

When data are sent to the FDA for the pre-IDE review, the information should be in the format that is also used for the IDE application (Table 1, Fig. 1). The FDA reviews the letter of intent (LOI), concept, or protocol to ensure that:

• the description in the protocol and the assay performance data from a CLIA-certified laboratory are sufficient to allow reviewers to identify the risk and benefits of the diagnostic

• documentation that patients were informed of the risks of the diagnostic during the informed-consent process is provided

• evidence that the analytical performance of the assay is robust and reliable

Because the FDA assesses the analytical performance of the assay for the diagnostic, information on reliability and reproducibility must be provided. The Cancer Diagnosis Program (CDP), CTEP, the Investigational Drug Steering Committee, and national experts have created templates for immunohistochemistry and FISH assays (with a somatic mutation template under development) that are posted on the CDP website (http://cdp.cancer.gov) to document an assay's analytical performance. During its review, the FDA also evaluates whether the diagnostic depends on the “art of medicine.” This term refers in part to whether a qualified and certified diagnostician will analyze the test. Thus, tests such as immunohistochemistry and FISH, which are assessed and interpreted by pathologists and cytogeneticists, are part of the art of medicine. The other aspect of the art of medicine is whether other clinical or pathologic information supports the diagnostic's use in medical decision-making. Finally, the assay may generate risk if it is measured without an expert's interpretation of the test (e.g., it scored or analyzed solely by a machine, such as a sequencer) and is used for medical decision-making without any supporting clinicopathologic data. Such a test may involve a somatic mutation that can be used to classify a disease or assign patients to a specific therapy without any supporting clinical or pathologic information. At the end of the pre-IDE review, the investigators will receive a report indicating any concerns of the FDA and whether an IDE is necessary. At this point, the investigators may consult with the FDA to get suggestions on how best to proceed. Although the pre-IDE review is nonbinding, the FDA will expect investigators to respond to any concerns when they submit their IDE.

The FDA may decide that a formal IDE application is not necessary for a particular assay. However, if the pre-IDE review indicates that a formal IDE application is required, the formal application must be submitted before the trial can begin, and the FDA review must be completed within 30 days (Fig. 1). At the end of the formal review, an IDE is either granted (allowing the trial to proceed after other reviews and approvals have been obtained) or there is further interaction with the FDA. The trial may also proceed if no comments are received within 30 days of receipt by the FDA. An alternative path is described below that permits the IDE application to be included within an IND.

Once clinicians and laboratorians understand the nuances of the IDE review, it is important for them to anticipate fitting that review into the workflow of creating the protocol. A major problem for clinicians and scientists involved with NCI-sponsored clinical trials is the need for faster protocol activation after approval of an LOI for early-phase trials, or concepts for later-phase trials. The NCI established the Operational Efficiency Working Group (OEWG) to “identify the institutional barriers that prolong the time from concept approval to accrual of the first patient, and develop solutions for overcoming these barriers.” As a result, CTEP has set in place a number of timelines mandating that protocols need to be developed and activated within 210 days for phase I, I/II, and II LOIs; 240 days for phase II concepts; and 300 days for phase III concepts after initial approval (Fig. 2A). These processes take at least 90 days for pre-IDE and IDE reviews, and clearly must go forward in parallel to stay within these timelines (Fig. 2B). If the FDA needs to review a diagnostic, ideally the pre-IDE review should begin when the LOI or concept is approved, and this requires that the assay for the diagnostic be analytically validated and locked down so that the analytical validity can be reported to the FDA. In addition, the NCI may facilitate an interaction between the FDA and the clinical investigators and laboratorians if the protocol chair and clinical laboratory scientists want that help. It is important to remember that the pre-IDE and IDE reviews are totally independent of the reviews of the LOI, concept, or protocol, but must be concluded before the trial may start. As a result, it is important to start the FDA pre-IDE review as early as possible in the protocol development process, and to be sure that an integral assay has already been developed and its performance has been validated in a CLIA-certified laboratory.

The first trial to undergo pre-IDE and IDE reviews was the phase III trial in AML conducted by the COG. In the COG AAML1031 trial, 3 genetic markers are used to assign therapy. One is FLT3/ITD, a mutation in the FLT3 receptor gene that is caused by duplication of a fragment of the juxtamembrane domain coding sequence of FLT3. The FLT3/ITD assay used in COG AAML1031 is a semiquantitative variation of the standard assay, in which, in addition to detecting the presence of FLT3/ITD, the ratio of the amount of mutant to wild-type product is determined [FLT3/ITD allelic ratio (ITD-AR)]. Results from three separate cohorts of children with AML have shown that in addition to the presence of FLT3/ITD, ITD-AR affects outcome, because the cohorts with ITD-AR ≥ 0.4 had a significantly worse survival than those with lower ITD-AR or those with wild-type FLT3 (8–13). The other 2 genetic markers used to stratify therapy in COG AAML1031 are the presence of mutations in Nucleophosmin 1 (NPM1) and CEPBα genes, 2 genes associated with a better outcome and measured by qualitative assays in which detection of any amount of somatic mutation qualifies as a positive assay (13). In the trial, the presence of FLT3-ITD with high ITD-AR leads to more intensive therapy treatment, whereas the presence of mutations in NPM1/CEPBα (although it is not entirely mutually exclusive) suggests that patients should receive less-intense therapy. The COG investigators first performed these 2 assays as individual PCR-based assays, but then suggested that they might bundle the 2 qualitative assays into a duplex PCR assay with minimal analytical validation. The fourth integral marker is an MRD assay, the results of which can cause a patient to be assigned to HSCT therapy.

The first important issue for the FDA was to determine whether data had been obtained to support the use of these diagnostics. As the Background and Rationale in the protocol carefully noted, all markers had data to support their use in the form of multiple publications (8–13) as well as use in prior phase III trials. The MRD assay had been in use for over a decade and its cutoff had been established in prior trials (14–16). Although this assay is also used to determine whether patients may be candidates for HSCT, a procedure that carries a risk of death and serious side effects (17–22), the assay is supported by other tests, and results are determined by the personal evaluation of a skilled, certified pathologist rather than solely measured by a machine. Thus, this assay falls within the category of the art of medicine and was not considered further by the FDA.

After an interchange between the investigators and staff of the OIVD, the investigators were provided with a contact in the FDA, who then sent the COG investigators a list of information to be supplied for the pre-IDE review (Table 1). At this point, the FDA advised the investigators to note that the risk of the assays in a trial should be mentioned explicitly in the protocol and the informed consent in order to satisfy the requirements in section 3 of the IDE application form (Table 1).

The FDA evaluated the analytical performance of each genetic assay using assay reliability and reproducibility characteristics provided by the COG from its CLIA-accredited laboratories. The FDA wants trials to use assays and cut-points that are locked down and based on retrospective and/or prospective experience. As an example, the COG indicated in its cover letter that it wanted to evaluate whether a lower percentage of blasts in the bone marrow might be used in the MRD assay to guide decisions regarding transplantation or removal from the study, in contrast to the cutoff used in previous trials, but did not yet have data in hand to justify the new cut-point. The FDA indicated that the previous cut-point should be still be used for this study, although data could be collected that might inform a more stringent cut-point in future trials. In addition, the FDA indicated that it was concerned that converting the NPM1 and CEBPα mutation assays into a duplex PCR assay when the only data that had been provided and analyzed to date were obtained by 2 separate independent assays was also problematic. The COG decided to follow the recommendations regarding the MRD cut-point and use the separate assays to determine NPM1 and CEBPα mutations for decision-making in this trial (Table 2).

Once the pre-IDE review was completed and the FDA made its recommendations, the COG decided to use an alternative pathway for the IDE application. The IDE application can be submitted directly to the CDRH or it can be bundled with an IND application that is evaluated by either the Center for Drug Evaluation and Research (CDER) or the Center for Biologics Evaluation and Research (CBER) of the FDA. The COG bundled the IDE application with an IND held by CTEP and provided a cover letter that requested the CDER to consult with the OIVD of the CDRH to evaluate their responses to the pre-IDE review. A benefit of this approach is that CTEP has a precedent that enables trials registered under INDs held by CTEP to begin accrual if all institutional review board (IRB) and other approvals have been obtained. Bundling an IDE with an IND that is not held by CTEP to either CDER or CBER is appropriate, but it would be wise to also send information to the OIVD directly and to wait the 30 days after receipt by the FDA (when the FDA has to notify sponsors if it has concerns about the IDE or IND). If no comments are received by the investigators within 30 days of receipt by the FDA, the integral marker may be used in the trial.

Emerging technologies for molecular genotyping in malignant diseases have enabled researchers to identify large numbers of disease-associated mutations in recent years. Further, whole-genome sequencing has provided a tool for unbiased evaluations of cancer genomes and identification of molecular alterations in previously unsuspected genes, increasing the number of genes implicated in cancer. In some cases, function-altering mutations have been shown to be associated with disease outcome or to be potential targets for directed therapies. As a result of such associations, commercial interest in such biomarkers has increased. In the last decade there has been an upsurge in the number of patents for cancer genes, and the patent holder can limit or completely block molecular evaluation of the genes in question. Patent holders may require laboratories to purchase a sublicense prior to testing for the gene in question, or can completely prevent individual laboratories from performing the assay, by mandating that all assays be performed in the company's central clinical laboratories. It is essential for investigators and their technology-transfer offices to determine whether a potential patent holder for a molecular diagnostic has been granted licenses that might block the use of that diagnostic in clinical trials. Investigators can now make such a determination through the use of Gene Cards (http://www.genecards.org) and GeneIP (http://www.xennexinc.com:8080/GeneIP/actions/Index.action), which identifies intellectual property related to specific genes that may be the basis for a molecular diagnostic.

The FDA recently became directly involved in regulating the use of integral diagnostic assays in clinical trials, prompted by concerns that certain assays (particularly newer assays to detect specific mutations or molecular signatures) may not be fully validated for diagnostic use, or the variability and reliability of specific assays may not be fully defined. Although few would disagree that clinical investigators should have the highest degree of confidence in the results of any diagnostic or investigational assay, it is not yet clear how the FDA's increased involvement in protocol development will improve the use of integral markers in clinical research. This new oversight raises concerns regarding both the monetary costs associated with it and the costs in terms of the time spent in meeting the requirements, which could potentially delay the onset of a clinical trial. As currently implemented, the process may take at least 3 months to complete, which is a particularly concerning delay in light of the historically long timelines of protocol development that resulted in an overhaul of protocol development processes for NCI-CTEP–supported cancer clinical trials. Potential benefits of this process include the early provision of data regarding the analytical performance of assays for molecular diagnostics, the increased attention to development within the investigational and clinical laboratory science communities, and the promotion of high-quality assays that can support clinically useful diagnostics.

One key question that emerged from this experience is whether it is more efficient to undergo a pre-IDE review or to proceed directly to an IDE application. Although the pre-IDE review may take >60 days, the dialogue between investigators and FDA staff may facilitate a timely and successful IDE application. However, there are no mandatory timelines for completion of a pre-IDE review, and thus determining the most efficient route forward is not a straightforward process. FDA guidance in this area is needed, and it might be useful for the agency to consider the metrics described by Poste and colleagues (23) in this Focus section. Finally, on the basis of our experience, we provide a list of recommendations that investigators who propose to use integral markers in their clinical trials ought to consider as early as possible during the development of their trial (Table 3).

No potential conflicts of interest were disclosed.