Phase III Clinical Trial Development: A Process of Chutes and Ladders

Commentary on Cheng et al., p. 5557

The Institute of Medicine (IOM) report entitled “A National Cancer Clinical Trials System for the 21st Century” addresses four overarching goals (1). Two of those goals directly relate to clinical trial operational issues: improving the speed and efficiency of the design, activation, opening, and conduct of clinical trials (goal I) and improving the means of prioritization, selection, support, and completion of cancer clinical trials (goal III) Additionally, the report accepted the recommendations of the National Cancer Institute (NCI) Operational Efficiency Working Group (OEWG) report, which includes specific timelines and milestones for performance in the opening of oncology clinical trials. This article presents a comprehensive review of the evidence demonstrating the need for change in the existing NCI clinical trials system.

The focus of this research is primarily on phase III clinical trials. It can be argued that phase III clinical trials have the highest likelihood of being practice-changing as they can be used as the basis for the implementation of new treatment programs (2). Because of their complex nature and accrual requirements, almost all non–industry-sponsored phase III clinical trials in the United States are designed and activated through the NCI cooperative group mechanism before they are opened at a cancer center. For clarity, we use the term “activate” to refer to the release of an approved protocol by a cooperative group to the oncology community and “open” to signify when the protocol has received local site Institutional Review Board (IRB) approval and has completed all other tasks required for the trial to be made available for patient accrual at that site.

The design and activation of phase III clinical trials requires an extensive development process, with multiple levels of internal oversight within a cooperative group and external oversight by organizations such as the NCI Cancer Therapy Evaluation Program (CTEP), the Food and Drug Administration, and the pharmaceutical company that has provided the investigational agent for the study. After activation by the cooperative group, the trial must then be evaluated locally at a site, adding additional layers of oversight and review. Whereas the NCI cooperative group program has made major advances in the treatment of cancer, this program has been heavily scrutinized almost since its inception in 1955, with the first concern over its effectiveness voiced in 1959 (3).

While such scrutiny is not novel, what is generally missing from such discussions is a detailed, evidence-based evaluation of the steps required for an idea to transit through the system and of the operational or “invisible” barriers faced by all parties in the design of phase III clinical trials (4). Pieces of this puzzle have been presented separately (4–7), but this article presents the first complete picture of the steps and time required to develop and open a phase III clinical trial that examines all participants in the process. It also highlights the barriers pervasive across the entire system.

Additionally, a nonclinical cancer researcher might hypothesize that trials that have received such extensive review by multiple parties should have a high oncology research community acceptance and, therefore, have a high likelihood of achieving their accrual goals. We present data showing that the opposite is the case: At individual cancer centers, there is a higher likelihood of zero accrual to a cooperative group trial than to non-cooperative group trials. Further, we present data showing that for NCI-CTEP–approved trials closed in an 8-year period (2000–2007), about 6% of all resulted in zero accruals. Finally, we offer a discussion of the implications of this long development process as it relates to accrual performance.

To understand the oncology clinical trial development process fully, research was conducted at a total of eight institutions involved in the development or implementation of oncology clinical trials in the United States. The sites investigated were two NCI-supported cooperative groups [Cancer and Leukemia Group-B (CALGB) and Eastern Cooperative Oncology Group (ECOG)], four NCI-designated comprehensive cancer centers (Fox Chase Cancer Center, The Ohio State University Comprehensive Cancer Center, University of North Carolina-Lineberger Comprehensive Cancer Center, and Vanderbilt-Ingram Cancer Center), the NCI CTEP, and the Central Institutional Review Board (CIRB) supported by the NCI. It is important to note that the cancer centers were not randomly selected; they were chosen because of their excellent scores on the clinical research management portion of their respective Cancer Center Support Grant applications.

For each organization, the Dilts and Sandler (4) case study and process map method was followed. An interdisciplinary team of experts from schools of medicine, engineering, and management collected data from multiple sources through (a) extensive staff interviews, (b) analysis of existing process documentation and records, (c) archival analysis of clinical trial initiation data, and (d) electronic mail and database records. Personal interviews involved questions targeting the research objectives of identifying processes and barriers to the opening of clinical trials. Archival data collection from written and electronic sources was chosen because (a) such documentation was broad and permitted multiple time frames, settings, and events, and (b) archival records are more precise and have less reporting and selectivity biases than an individual's recollection of steps. This process allows for triangulation among the collection of descriptive data (what was said was being done as uncovered in the interviews), normative data (what the policies and procedures documentation indicated should be done), and archival data (what the clinical trial record review showed was actually done). This three-part method resulted in capturing a complete understanding of the development process structure. The end result of this effort was validated process map and timing data. The method was replicated at multiple institutions to ensure data consistency across sites. Additional, independent data were collected from the CTEP for closed studies during the 8-year period from 2000 to 2007. This study was approved by the Vanderbilt University IRB (IRB #060602) and supported by the NCI and Vanderbilt University.

The research has three primary outcome metrics: (a) process mapping, (b) timing analysis, and (c) accrual performance.

This metric identified and mapped existing process steps required to activate or open an oncology clinical trial. The total development time is defined as beginning with the first formal submission of a concept or a letter of intent (LOI) to a cooperative group and ending with the opening of the study for patient accrual at a cancer center. These data were collected by means of more than 30 onsite sessions across more than 100 personnel interviews, additional follow-up e-mail correspondence, and a series of at least two clarification teleconferences each with members of the cooperative groups, CTEP, and the comprehensive cancer centers. Informal clarification interview sessions were conducted at each site to ensure that the data and identified processes were accurate prior to finalization of results.

Steps and timing were cross-validated multiple times at different levels of institutional hierarchy. The interviews were conducted in both individual and group settings, with the input remaining anonymous. The interview process included both open- and closed-ended questions. On completion, the interviewees were asked to clarify specific acronyms, decision point names, position titles, and responsibilities. Objective data stored in databases or e-mails were cross-checked whenever possible with other independent records. Final validation of the process was conducted at each site with all involved participants to ensure that the process was captured correctly and that the interfaces among the participants were accurate.

Once the process map was complete and verified for accuracy, the calendar days needed for each of the major steps in the process map were collected. For example, a timing analysis of a single trial at one site required the review of archival data compiled from more than 400 historical e-mail correspondences, 15 file reviews, and validation of timing data through queries to the clinical research management database. Evaluating both the process and the timing of each process allows for an understanding of the maximum yield, or output, that the process can achieve. Identifying this and the identified process variation allows the selection of more effective strategies for process and productivity improvements (8, 9).

A retrospective analysis of therapeutic trials completely closed to enrollment was conducted at each of the institutions. Data were acquired from the various databases that were implemented at each of the institutions. Random clinical trials within the cohort were selected and checked for consistency against physical records.

Results are divided into four segments: (a) number of steps, loops, and groups involved in activating and opening a phase III NCI-approved therapeutic clinical trial; (b) time to activate and open such studies; (c) accruals to all phases of cancer center and CTEP-approved therapeutic studies; and (d) the relationship between time to activate or open and likelihood of eventual accrual success.

Using a representative cooperative group and comprehensive cancer center, activation of a phase III cooperative group trial requires a minimum of 458 processing steps in a cooperative group and 216 at the CTEP/CIRB, with an additional 95 needed at a comprehensive cancer center to open the trial after activation, for a total of at least 769 processing steps (Table 1). Processing steps have two types: work steps (i.e., those that require activities to be performed) and decision points (i.e., steps that require different routing of the study). There are two important caveats for these data. First, there is some “double-counting” of steps; for example, when one organization's process step requires that a proposed trial be sent to a different organization, the receipt of this trial will be counted as a processing step in the receiving organization. This results in a small degree of overstatement of steps.

Second, it is possible in some parts of the process to bypass or repeat steps. Using the children's game “chutes and ladders” as a comparison, such a bypass represents a ladder. For example, if a site accepted the results of a CIRB, this would represent a ladder where the local IRB steps could be skipped. Another example would be a study that had expedited review, thus bypassing the full-board review steps. However, it is much more likely that the 769 steps identified are an accurate estimate of the minimum number of steps because virtually all trials are returned for reprocessing sometime during their design. There are a total of 49 points, termed “loops” or chutes, where a trial may be returned to an earlier part of the process flow. For example, if an IRB review requires a change in the science of a study, the proposed protocol must be returned to repeat some of the steps of scientific review.

Neither chutes nor ladders are inherently good or bad. From a positive perspective, a chute that requires a reevaluation of a protocol because of a major scientific or safety flaw is a good thing. However, if this rework is repeated multiple times, it is a symptom of a deeper underlying flaw with either the protocol or the protocol design system itself. The OEWG report determined that for cooperative group phase III trials activated between 2006 and 2008, nearly all (97%) required two or more revision loops and 34% required four or more revisions (p. 19).

Figure 1 shows the back-and-forth nature of the process for one representative phase III trial. This trial was selected because it had the most complete set of timing data. The horizontal rows on the figure are swim lanes and capture those steps undertaken by the study chair, cooperative group, CTEP, and CIRB, respectively. Beginning on the top left, it is straightforward to follow the steps of the design of the concept and protocol as it flows through the activities of the various participants.

A listing of the various participants is shown in Table 2. A total of 36 separate groups or individuals may be required to approve a study before it is open to patient accrual. More importantly, there is significant overlap across the scope of reviews for scientific, data management, safety/ethics, regulatory, contracts/grants, and study start-up—each of which has the potential to experience a chute to a former step to obtain complete consensus across all reviewing entities.

The time from submission of initial concept/LOI to study activation and opening is difficult to measure because although the opening date of the study is easily known, capturing the exact time when a concept is formalized can be problematic. The earliest formal measurable date was selected, usually the initial concept/LOI proposal sent by a principal investigator to a cooperative group study committee. A total of 41 trials were evaluated (13 for CALGB and 28 for ECOG). Two comprehensive cancer centers supplied complete historic data with respect to time (n = 58 and n = 178) and two supplied small samples of studies (n = 3 and n = 4; Table 3). The median time for CTEP and cooperative group processing is approximately 800 calendar days (CALGB, 784 days; ECOG, 808 days), with ranges of 537 to 1,130 and 435 to 1,604 days, respectively. For cancer centers, the median calendar time is approximately 120 days, with a range of 21 to 836 days, with cancer center B being considered an outlier because of its small sample size. These dates are cumulative as cancer centers cannot begin processing a phase III trial until it is activated by a cooperative group. The median calendar time from initial formal concept/LOI to opening for the first patient is therefore approximately 920 days, or nearly 2.5 years. However, the range is 456 to 2,440 calendar days, or 1.25 to 6.7 years. To put these dates in perspective, this median time is longer than it took to design, conduct, and publish the first CALGB trial (10).

One major issue related to the development of clinical trials is that they require the coordination and cooperation of multiple participants. For example, designing and activating a phase III cooperative group therapeutic nonpediatric trial requires evaluation and coordination between a cooperative group and the CTEP. Multiple reviews by both parties can create a natural tension where one organization is waiting for the other organization to respond to a change. In its worst form, such tension can lead to blaming most of the start-up delay on the other organization's inefficiency. To investigate this issue, 28 studies activated between 2000 and 2005 were evaluated in depth with respect to time (Table 4). Interestingly, as can be seen from the last column in the table, no specific review or party consistently required longer time. Thus, the issue is not how to repair a single element of the system; rather, it is how to reengineer the entire process to reduce rework while simultaneously increasing the safety, quality, and performance of the trial.

Completing the arduous development process does not ensure accrual success. Data from the four comprehensive cancer centers highlight the poor accrual performance of cooperative group trials at academic medical centers (Table 5; ref. 11). Accruals at comprehensive cancer centers occurred over a multiyear period, but there was no complete overlap in the time periods evaluated. Overall, 898 trials were in the sample: 394 cooperative group trials and 504 non-cooperative group trials. Of particular interest are those trials that were opened and closed to accrual with zero enrollments. Overall, for cooperative group trials opened at cancer centers, 38.8% of trials resulted in zero accruals, whereas 20.6% of non-cooperative group trials had zero accruals; thus, a cooperative group trial was nearly twice as likely to have zero accrual at a cancer center than a non-cooperative group study (P ≤ 0.001). Expanding the range, 77.4% of cooperative group trials opened at these cancer centers had fewer than 5 patients accrued.

One argument that could be made to justify these results is that, because cooperative group trials are opened at multiple sites, it is reasonable to have low accruals at any individual site because of the high number of sites contributing to the single accrual goal. Because it is difficult to understand why any cancer center would open such a high number of adult cancer trials if they expect zero accruals, we decided to investigate this question from a more global perspective, that is, looking at the aggregate, national accruals to such trials. Investigating accruals for all CTEP-approved clinical trials completely closed to accrual from 2000 to 2007 showed that of the 911 trials of all phases in the sample period, 6.4% resulted in zero accruals (Table 6). With respect to phase III trials, 39.1% of the trials closed to accrual with 20 or fewer enrollments.

The relationship between accruals and development time was also investigated for phase III cooperative group trials closed to patient enrollment at one cooperative group between January 2000 and July 2006 (n = 15). Figure 2 highlights the relationship between development time and accrual. Two of the trials were stopped because of scientific/safety reasons. Of the trials that required less than the median development time (n = 7; shown in black), five achieved more than their expected minimum accrual by study closure, and one achieved 94% of its goal. On the other hand, of those trials with development time greater than the median (n = 6), only one achieved greater than 90% of the expected accrual by study closure. The remaining trials closed with less than 15% of expected accrual. This relationship is not limited to cooperative group trials—additional analysis on a more complete data set from the CTEP, the results of which are published in this issue of CCR, (ref. 12; and see article by Cheng et al., page 5557) shows a strong negative statistical relationship between achievement of accrual goals and development time.

The most recent call for reinvigorating the cooperative group system was issued in April 2010 with the publication of the IOM report “A National Cancer Clinical Trials System for the 21st Century” (1). Although the results and the recommendations from this report may be controversial, the problems it outlines as well as its recommendations are not unique. Thirteen years ago, in 1997, an NCI report concluded, “The clinical trials methodologies used by the 11 cooperative groups and 51 cancer centers have created a system described as a ‘Tower of Babel’ (1). The problem persists today as evidenced by the summary results presented here and in recent comments alluding to the process of developing clinical trials as the “Ordinary Miracle of Cancer Clinical Trials” (13). Additional evidence of the problem can be seen in three additional studies. Go and colleagues (14), investigating a total of 495 ECOG phase II and III trials from 1977 to 2006, found that 27% of trials failed to complete accrual. Schroen and colleagues (15), in a study of 248 phase III trials open from 1993 to 2002 from five cooperative groups, discovered that 35% were closed because of inadequate accrual. Finally, Wang-Gillam and colleagues (16), in a cross-country study, found that the median time from submittal to opening for a phase II industry-sponsored thoracic oncology clinical trial was 239.5 days at a U.S. institution and 112.5 days at an Italian institution.

Improvements to both the quality of the processes and to the time required to develop clinical trials should not be done in isolation. There is a need to rekindle the collaborative spirit of the cooperative group program. As is well stated in the IOM report, such collaboration would include uniformity of back-office, support functions while simultaneously not impairing a group's ability to be creative with respect to developing clinical trials around interesting and important scientific questions. Additionally, a general philosophy of facilitation rather than primarily oversight should be fostered throughout the NCI, cooperative groups, and cancer centers.

Clearly, it is time to go beyond mere recommendations and begin the actual execution of changes in the existing system. Some changes are beginning with efforts such as the timelines developed and implemented by the NCI OEWG (17). However, this represents only one of many possible steps—and involves only some of the many participants that need to change—if the system of clinical trial development as a whole is to be reinvigorated.

We would be remiss if we did not address the other elephant in the room: financial resources. Financial incentives, or what is better characterized as removal of financial disincentives, for investigators and centers are a critical aspect of improving the cooperative group program. In 2004, the estimated median per-patient site cost for a phase III cooperative group trial was $5,000 (18) to $6,000 (19), well above the per-patient reimbursement rate of $2,000, which has remained constant since 1999 (20). In 2008, the NCI increased the rate of reimbursement for a small minority of complex trials by an additional $1,000 (21). However, even this increased rate is not adequate to completely cover all labor costs, per-subject enrollment costs, and additional research-related paperwork and reporting requirements (22). The response by the cooperative group member sites has been dramatic, with 42% planning or considering limiting accruals to cooperative group trials (23) because of per-case reimbursement or staffing issues.

However, this does not imply that the call for increasing the cooperative group budgets will in itself successfully transform the existing system of cancer research. Recall that the recommendations from the 1997 report occurred while funding was increasing for cooperative groups. Simply adding additional resources to a system in need of repair does not ensure better performance; rather, it is time to revamp the entire system with a focus on effectiveness, efficiency, and financial stability.

The findings presented in this review demonstrate the “chutes and ladders” of the existing system: Some aspects of a concept or protocol are able to bypass some process steps (a ladder), but as shown in the OEWG report, virtually all studies are reprocessed and revised (a chute), sometimes up to six times before proceeding. Although many of these chutes and ladders can be within the same organization, most are the result of interactions between organizations; thus, it is the system that needs to change. It is critical that more ladders be built and more chutes be eliminated in the entire system to achieve the overarching goal of discovering better ways to prevent, control, and cure cancer.

No potential conflicts of interest were disclosed.