Predicting Accrual Achievement: Monitoring Accrual Milestones of NCI-CTEP–Sponsored Clinical Trials Free

The recent report from Institute of Medicine (IOM) on the state of the Clinical Trials Cooperative Group Program highlighted the importance of reducing the number of low enrolling clinical trials while improving the number of high priority trials that successfully achieve the intended accrual goals (1). With approximately 3% to 5% of the adult cancer patients in the United States participating in clinical trials, individual clinical trials struggle to achieve the necessary accruals (2). Unfortunately, approximately 38% of CTEP (Cancer Therapy Evaluation Program)-supported oncology trials fail to attain the originally specified minimum accrual goal, with phase III trials having a substantially greater number of trials that fall short of achieving the accrual goal (3). These clinical trials not only are unable to achieve the necessary patient enrollment necessary to evaluate the proposed scientific hypotheses but also cause trials to remain open longer than planned, resulting in unanticipated costs from additional administrative and clinical resources (4–7).

The ability to utilize predictors of accrual performance to a trial may allow for better utilization of resources and increase the likelihood that subjects enrolled to trials will contribute to the advancement of medical knowledge. Therefore, we pose the following question: Are there early clinical trial predictors during the enrollment period that may be used to identify and assess the likelihood of a trial achieving its accrual goals?

Studies that are sponsored by National Cancer Institute (NCI) that involve collaborative efforts between cancer centers, cooperative groups, consortiums, and industry sponsors must be evaluated through the CTEP. NCI-CTEP reviews and activates approximately 500 new clinical protocols annually and is the largest supporter of phase III clinical trials sponsored by NCI (8, 9). In an effort to understand the accrual patterns of oncology clinical trials, accrual patters were evaluated in a retrospective study of CTEP-sponsored therapeutic trials between 2000 and 2007. Specifically, we assess the likelihood that a trial will attain the accrual goal at 2 accrual milestones: (i) time to enroll the first patient and (ii) time at the expected enrollment time period (i.e., the length of time that a trial should have remained open, given the projected accrual rate).

All therapeutic, nonpediatric phase I, I/II, II, and III oncology trials requiring NCI-CTEP evaluation that were activated and subsequently closed to accruals between May 1, 2000, and December 31, 2007, with complete accrual monitoring data (n = 764) were eligible for this study (Fig. 1).

The CTEP Protocol and Information Office (PIO) provided clinical trial characteristics and accrual data via the Clinical Data Update System and the Clinical Trials Monitoring Service, which monitors on a monthly basis all patient registrations to publicly sponsored cancer clinical trials. Projected accrual rates, projected minimum accrual goal, and activation dates of each trial are defined within the trial protocol and collected in the PIO database.

Studies that did not have information related to the projected accrual rates were excluded (n = 24). If minimum projected accrual goals were not available, the maximum projected accrual goal was used (n = 2). Studies that closed with zero accruals at the time of study closure were not included in the analysis, as they had no accrual rate (n = 42). To investigate whether incidence or mortality rates of the disease impact the accrual performance, and to account for factors of rarity of the disease, data on the median age–adjusted cancer incidence and mortality rate were collected from the Surveillance Epidemiology and End Results (SEER) cancer registry (10).

The accrual milestones of time to first patient enrollment and expected enrollment time period were used to evaluate the accrual performance for this study. Figure 2 provides a description of the accrual milestones during the clinical trial enrollment period and a sample phase III trial. Calculation of accrual performance at these milestones was conducted on the entire sample of 764 clinical trials and analyzed collectively.

The date of trial activation is the date that NCI-CTEP receives the notification that the study is ready to begin accruing patients. The date of activation is recorded in months for the purpose of calculations in this research and is concatenated to the beginning of the month.

The minimum projected accrual of a study is defined within the protocol and is typically calculated from a combination of investigator consensus and statistical power requirements. The minimum projected accrual goal for each trial is defined within the study design of each trial and is highly dependent upon the phase. Specifically, phase I minimum projected accrual goals assume that the dose limiting toxicity (DLT) is observed at the first dose levels; phase I/II trials establish minimum projected accruals on the basis of the phase I accrual and update the minimum accrual when the trial transitions to a phase II trial; and minimum accrual goals for phase II and III trials are based upon the number of accruals required to complete the first stage of the study design.

Final accrual performance is dichotomous, with those trials achieving 100% or more of minimum projected accrual enrollment at the time of complete study closure being defined as attaining the accrual goal and those trials not reaching this threshold as not attaining the accrual goal (We acknowledge that studies can close due to a host of reasons, including adverse events, regulatory requirements, achievement of an early stopping rule, or other unforeseen situations. The specific reason for study closure was not available.) The accrual goal percentage was calculated by dividing the final accrual by the projected minimum accrual. Final accrual of a study is the number of accruals on a study at the time the study was completely closed to accrual.

The accrual milestones are captured for each study on the basis of the dates of patient enrollments recorded to the nearest month. It is assumed that the rate of accrual is linear. Two different accrual milestones were utilized: (i) time to first patient enrollment and (ii) accrual performance at expected enrollment time period.

First, time to first patient enrollment was recorded on the basis of the number of months required from the month of study activation to the month of first enrollment. This point was evaluated in 4 groups depending upon the number of months to enroll the first patient (1–2 months, 2–6 months, 6–12 months, and >12 months). It was of interest to discover whether “fast” enrolling trials (i.e., those within 1 or 2 months) could be used as an indicator of eventual achievement of the accrual goal. The other periods were selected on the basis of often-utilized 6- and 12-month review cycles that institutions use to evaluate the trial accrual performance. Analysis at the time to first patient enrollment is conducted against the eventual attainment of the accrual goal at study closure.

Second, the accrual milestone was observed at the expected enrollment time period. The expected enrollment time period is calculated by dividing the minimum projected accrual by the expected rate of accrual. Both rates of accrual and minimum projected accrual are specified within the protocol and extracted from the study design. Studies were analyzed across 6 different groups depending upon the actual accrual performance, as a percentage of the expected performance, at the expected enrollment period (1%–19%, 20%–39%, 40%–59%, 60%–79%, 80%–99%, and ≥100%). The accrual performance at the expected time to achieve the accrual goal is compared with the eventual attainment of the accrual goal.

Descriptive statistics to summarize the accrual characteristics were conducted for the continuous variables of the minimum projected accrual and expected periods to achieve the minimum projected accrual by calculating medians and interquartile ranges (IQR). A maximum, 2-tailed α of 0.05 was maintained to determine the statistical significance. Comparison among the trial types (i.e., phases I, I/II, II, and III) were conducted using the Kruskal–Wallis test, with a post hoc comparison of statistically significant overall tests by using Mann–Whitney tests with a Bonferroni-adjusted α level of 0.008.

Categorical and ordinal groups were summarized using univariate and cross-tabulated frequency distributions. Unadjusted and adjusted ORs, along with their respective 95% CIs, were obtained using multivariate logistic regression analysis. Unadjusted ORs were calculated with the addition of adjusting for both phase of the study and the size of the study measured by the minimum projected accrual to compensate for any interactive effects. Statistical analyses were done in SPSS (version 16.0; descriptive and logistic regression).

A total of 764 oncology trials were identified as CTEP-evaluated, therapeutic, nonpediatric, phase I, I/II, II, III trials opened, and completely closed to accrual between May 1, 2000, and December 30, 2007 (Table 1). Clinical trials were focused across 20 different disease sites, and the cohort of trials consisted of 62,447 participants.

Overall, 62.8% (n = 480) of trials achieved at least 100% of the minimum projected accrual goals by closure. The number of the phase III trials that achieved the accrual goals by study closure was statistically significantly lower than the other trials by phase (38.6%, n = 22; P < 0.001). No statistically significant differences were observed between the trials excluded (n = 24) and the study sample with regard to final accrual performance (Spearman's correlation: P = 0.389).

Only 18.5% (n = 141) of the trials achieving the minimum project accrual goals met this goal within the projected period of time. Phase III studies had the high proportion of studies that met the minimum goal within the expected period with 28.1% (n = 16), followed by phase II studies with 21.6% (n = 109), phase I/II trials with 7.9% (n = 5), and phase I studies with 7.9% (n = 11).

However, on average, trials achieving the minimum projected accruals (n = 480) were 163.3% slower than planned to achieve this goal. Interestingly, phase III studies that achieved the minimum projected accruals by study closure (38.6%, n = 22) met the accrual goal quickly, that is, within 73.9% of the projected period of time. This is significantly faster (P < 0.001) than the other trial phases—the enrollment period for phase I, I/II, and II trials necessary to achieve the minimum projected accrual was 241.7%, 216.7%, and 142.9% of the projected period, respectively.

The trials that did not achieve the minimum projected accruals were open 127.9% longer than the expected period to achieve the minimum projected accrual. Phase III studies that did not achieve the minimum projected accrual on median closed prior to the expected period (47.2%). Phase I, I/II, and II trials remained open beyond the expected period by 213.3%, 150.0%, and 130.8%, respectively.

When comparing trials that achieved the accrual goal at closure with those that did not, the trials that closed without achieving the minimum projected accrual had a larger accrual requirement for phase I (P = 0.009) and phase II studies (P > 0.001). Furthermore, studies that attained the accrual goal had a significantly shorter projected enrollment period than studies that did not attain the accrual goal (phase I, P > 0.001; phase I/II, P = 0.017; phase II, P > 0.001; and phase II, P = 0.003).

To investigate the relationship between time to first patient enrollment with the accrual performance at closure of a study, multivariate logistic regression analysis was conducted to calculate the likelihood of attaining the anticipated accrual goal. Clinical trials were stratified by the number of months required to accrue the first patient from study activation. The likelihood of achieving the accrual goal is highest for those studies that accrued the first patient within the first 2 months of enrollment (Table 2). All subsequent groups had a statistically significant decreasing likelihood of achieving their goals compared with this referent group. Relative to trials that accrued the first participant within the first 2 months, trials taking between 2 and 6 months were statistically significant and less likely to achieve or enroll for the minimum projected accrual (OR: ≤0.637; 95% CI: 0.464–0.875; P = 0.005). Studies with the first accrual between 6 and 12 months and studies that had the first accrual beyond 12 months had a decreased and statistically significant likelihood of obtaining the minimum projected accrual at the time of study closure compared with the referent (OR: ≤0.208, 95% CI: 0.056–0.459; P = 0.001). The relationship between month to first patient enrollment and achieving the accrual goal was consistent when adjusting for the minimum projected accrual of the trial, phase of the study, and cancer incidences by disease.

The impact of disease type as classified by the cancer incidences collected from the SEER cancer registry was analyzed with respect to the time to first patient enrollment (Table 3). No statistical difference between the cancer incidences or mortality and the time to enroll the first patient was observed (P = 0.749 and P = 0.152, respectively).

Of the studies that are open beyond the expected period to achieve the minimum projected accrual (n = 603), the analysis of the accrual milestone at the time of expected enrollment time period resulted in the observation that studies that have not achieved at least 60% of their minimum projected accrual goal within the stated projected period of accrual result in a statistically significant decrease in likelihood of achieving the desired accrual by study closure (Table 4). Relative to studies that have achieved at least 80% of the minimum projected accrual within the projected period, trials with less than 60% of the minimum projected accrual have a statistically significant less likelihood of achieving the minimum accrual goals (OR: 40%–60% of the minimum projected accrual: 0.190, 95% CI: 0.055–0.652, P = 0.008, OR 20%–40% of minimum projected accrual: 0.121, 95% CI: 0.036–0.409, P = 0.002; OR: 0%–20% of the minimum projected accrual: 0.065, 95% CI: 0.019–0.227, P > 0.001). A total of 391 trials (64.8%) fall within the category of studies with more than 60% of the minimum projected accrual with a decreased likelihood of achieving the minimum accrual goal at study closure.

Given the previously established differences among phase, minimum projected accruals, and time to first patient enrollment on a study, the likelihood values were adjusted for these 3 variables are also summarized. No differences in the relationship between the percentage of accrual achieved at the accrual milestones were observed when adjusting for the additional factors.

This study investigates the accrual performance of oncology clinical trials and highlights opportunities to utilize early enrollment indicators as a predictor of eventual attainment of accrual goals. The analysis of NCI-CTEP oncology trials reveals that a small number of trials (11.1%, n = 85) can achieve the minimum projected accrual within their planned enrollment period. Almost 2 of 5 trials in the sample did not achieve the minimum projected accrual by study closure. For phase I, I/II, and II studies did that achieve the minimum projected accrual goal, the project accrual time period is often underestimated when compared with the actual time required; these trials are open to patient accrual 213.8%, 150%, and 130.8% longer than the expected period to achieve the minimum projected accruals, respectively. Phase III studies are unique because a higher percentage (61.4%, n = 35) of trials closed without achieving the minimum projected accruals, although phase III trials may have a greater proportion of trials that reach a scientific endpoint without achieving the originally intended accrual goal (11). It is important to again note that our endpoint, achievement of minimum projected accrual, may inflate the negative connotation with respect to trial “success,” as trials may close for a variety of justifiable causes. Although such data were not available for this research, we encourage others to investigate this issue.

We provide multiple accrual milestones of a clinical trial that can be utilized to access the likelihood of a trial achieving the minimum projected accrual. The findings show that the accrual performance of a clinical trial can be predicted as early as the time to first patient enrollment on a trial. Almost half of the studies (n = 379, 49.6%) enroll the first patient outside the first 2 months of the study enrollment, which translates into those studies having a statistically significantly lower odds of achieving the minimum projected accruals at study closure (OR: ≤0.637) than the referent. Furthermore, trial accrual performance can also be predicted at the expected period to achieve the minimum projected accrual goal. Even with the use of a more liberal definition for projected period to achieve the minimum projected accrual, a large percentage of studies (64.8%, n = 391) fall into the category of not achieving at least 60% of the minimum projected accruals by the projected period and thus have a decreased likelihood of achieving the minimum projected accrual by study closure. Evidence from this research supports CTEP early-stopping guidelines enacted after April 2004 to address the issue of slow accruing trials, yet slow accruing studies continue to persist in the portfolio (12).

Adequate accrual to clinical trials is the most fundamental and easily quantifiable measure of performance for a clinical trial (7). The ability to monitor clinical trial accrual performance allows for greater support for earlier decisions to be made about the management of clinical trials. Identifying studies with a decreased likelihood of achieving the minimum projected accrual may lead to trial decisions. Decisions can be made to add the additional resources and/or funding to implement actions that may improve accruals, such as opening a study to multiple institutions or closing studies early to release resources to support other trials with a greater likelihood of achieving their accrual goals.

We do not advocate making decisions solely on accrual performance during these 2 accrual milestones; rather, we advise utilizing accrual-monitoring metrics to complement the scientific judgment of competing accruals to each individual clinical trial when making decisions about the management of trials. Specifically, we advocate applying the findings in this research into a stage–gate model through the enrollment period by utilizing these milestones as critical evaluation points (13). This is a commonly used process control technique used in the development of products, processes, and systems whereby continuous decision making is integrated to optimize the utilization of resources throughout the lifecycle. Enrollment periods would be stratified on the basis of predetermined milestones that would define each stage. The gates would be set at each milestone where decision making about the continuation of the trial and/or allocation of resources to the trial could be made. These decisions should be made relative to the overall portfolio of trials and coupled with the dimensions of scientific relevancy, operational feasibility, and strategy importance to better optimize the number and types of trials that achieve the intended objectives.

This study utilizes accrual estimations set forth by the investigators and the study team to extract the accrual milestones for evaluation. Observations from this research find that both accrual rates and accrual requirements are being underestimated. There is a need to find the methods to reasonably estimate the accrual performance and operational feasibility of clinical trials early on in the design and development of the trial. Achieving accrual goals is the fundamental requirement that must be satisfied–without the necessary enrollments, the originally intended scientific question cannot be answered. Investigators should be held accountable to the accrual estimations in order to proactively manage the trials throughout the enrollment period. Ultimately, the barriers to clinical trial productivity including the achievement of adequate accruals require a cooperative responsibility of the research community collectively (14).

Closing studies due to poor accruals is not ideal in any circumstances. Large amounts of time and effort are consumed on the development of a clinical trial with poor accrual and, ultimately, do not allow the originally intended scientific endeavor to come to fruition (15–17). Patients may be volunteering to participate on a study enrollment in studies that do not help advance the state of medicine. Sunk cost bias ingrained from the efforts committed toward the development of the clinical trial can often jeopardize current resources to be allocated to poor accruing trials even beyond the likelihood of attaining the accrual requirements.

The results presented in this article are limited by the fact that findings are applicable for only NCI-CTEP studies–specific accrual milestones may be unique, dependent on various institutional and study characteristics. Furthermore, there are numerous reasons why studies have low accrual or why studies close to accrual before achieving the minimum projected accrual (11, 18). We provide an initial foray into a systematic analysis of accrual performance by using standard metrics set a priori to study implementation. Continued research, particularly prospective analysis, should be conducted to identify the characteristics that are attributed to studies with low accrual in order to reduce the occurrence of studies being closed without any sufficient accrual needed to gain the intended scientific objective.

The authors indicate no potential conflicts of interest.

This study was supported by grant no. 3U10 CA 21115-32 from the NCI to R.L. Comis and by subcontract to D.M. Dilts and A.B. Sandler.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.