Raising the Bar for Antineoplastic Agents: How to Choose Threshold Values for Superiority Trials in Advanced Solid Tumors

This article is featured in Highlights of This Issue, p. 937

Two general problems are increasingly recognized and debated with regard to the improvements in cancer treatment: their small incremental size and the high price of the new drugs. The two problems are connected because modern clinical trials in advanced solid tumors have become larger and larger (with associated increasing costs), resulting in statistically significant findings being achieved with smaller and smaller observed treatment effects. This drug-development strategy has been based on a rationale, risk-minimizing philosophy by the pharmaceutical industry; however, we suggest that we are approaching (or have already passed) an inflection point where continued pursuit of this strategy is not optimally productive.

Modest benefits could be considered worthwhile if associated with moderate costs and toxicity, whereas a new drug with a very high cost and/or substantial toxicity is worthwhile only if it produces sizeable clinical benefits. Hence, the relevance of statistical significance has increasingly been challenged when the treatment effect is small (1, 2). The simple solution would be to raise the bar of efficacy for approving antineoplastic agents (3, 4). As supporters of this concept (3), here we try to expand it by describing a structured approach to this problem in advanced solid tumors. We also apply our derived model to a large sample of the most important pivotal trials on biologic agents published during the last 15 years, to evaluate which of them would meet the criteria for success according to various threshold values of several efficacy parameters.

If a new treatment is to be introduced into clinical practice in the setting of “superiority” to an existing treatment, it is not sufficient to demonstrate that it is “better” than standard therapy. It should be necessary to demonstrate that its benefits outweigh its adverse effects and costs. If the benefits are expressed in terms of overall survival (OS), the increase in survival that balances the harms/costs of the treatment represents a threshold, the minimum clinically meaningful outcome (mCMO). This value should be considered as a cutoff between what is and what is not clinically meaningful. Until the treatment effect is shown with statistical rigor to be larger than this threshold, the treatment should still be considered experimental.

The extent of the benefit identifying mCMO may be a function of three factors: the prognosis of patients with that condition, the toxicity/inconveniences of the treatment, and its cost. In addition, the extent of benefit may have different relevance according to the endpoint used (OS, PFS, or others) and the way it is expressed (HR, absolute median gains, etc.). In general, depending on the prognosis and the setting of the disease, there is agreement that OS or PFS are the endpoints to be used in advanced solid cancers, with the first one being preferred whenever practical. Here, we chose OS. Extending the approach to alternative endpoints, such as PFS, is feasible, but involves a substantial increase in the number and complexity of the issues to be addressed. For instance, to allow comparisons between the effects of different drugs and to provide estimates of benefit readily understandable by patients and health administrators, it should be possible to translate effects on PFS into effects on OS, which is far from simple. For this reason, we included in our analysis only those trials where OS mature data were reported.

We first focused on the identification of the best candidate parameters for the definition of the mCMO. Once those parameters were identified, we defined the threshold levels for each prognostic condition. We then arbitrarily set the bar of the mCMO at three different levels of required benefit (high, medium, and low), assuming that it may be simple to adapt these thresholds to different toxicity and cost categories (low, medium, and high). For example, if a medium level of benefit is required, the threshold values could be maintained for a medium level of toxicity and cost, but these values could be raised to high if the toxicity or cost is high or lowered to low if toxicity or cost is low. In this way, this model could grossly implement all three major determinants of benefit.

Although HR is the parameter used in the design and interpretation of clinical trials, other summary statistical indices are also important. The median gain in OS is a straightforward figure that has the advantage of being easily understandable by patients and other stakeholders. The late effects of treatment have the same advantage, no matter whether expressed in terms of absolute increase in the survival rate at 2, 3, or 5 years (usually not so impressive, typically in the range of 5%–15%) or in terms of proportional increase in OS rates at 2, 3, or 5 years (usually more impressive figures, typically in the range of 20%–50%). Therefore four OS-related parameters may contribute to define the mCMO (Fig. 1): HR, gains in median OS, absolute, and proportional gains in the long-term survival rates, representing two conceptually different types of treatment effects: a small benefit for many (SMALL) and a large benefit for few (LARGE). SMALL may be measured both in terms of HR and gains in median OS, whereas LARGE may be more appropriately measured by proportional and or absolute increase in 2, 3, or 5-year OS. Three out of four parameters (median gain, absolute, or proportional gain in OS rates) are in general easily understandable by patients. This would provide extra value by using a language that is appropriate for the patient-doctor relationship.

We have considered two determinants for each of these four OS-related parameters: the extent of the required benefit and the prognosis of the condition under study.

• Extent of the required benefit: We have proposed three potential levels of the benefit representing the mCMO when a low, medium, or high required benefit is the desired target (Table 1). The “high” level might be used when the toxicity and/or the cost of the new drug are high, the “low” level may apply to drugs that are both nontoxic/safe and inexpensive, whereas the “medium” level could be used in intermediate cases. The threshold values proposed for the “low” level are higher than many ‘target’ values used in current ongoing phase III trials, in keeping with the general concept of “raising the bar” in oncology.

• Prognosis of the condition under study: Because the mCMO also depends on the prognosis of the condition under study, we have identified four prognostic categories, based on the median OS observed with standard therapies: <9 months, 9 to 12, 12 to 18, and > 18 months. These prognostic cutoff values were selected after a series of attempts to have a sizable and rather homogeneous number of trials in each prognostic category, but still reflecting the sense of poor, medium, and good prognosis.

The threshold values for the four key parameters were arbitrarily selected using the following criteria:

• HR for OS: The maximum variation within each level of required benefit was set at 0.1. The HR variation within the same prognostic category was 0.05.

• Gains in Median OS: The minimum and maximum survival benefit was set at 2 and 6 months, respectively, as a function of the prognostic category, to reflect the perception that no benefit of less than 2 months and any benefit more than 6 months are in general considered clinically worthwhile, regardless of the underlying prognosis, toxicity, and cost of the treatment.

• Absolute gains in late OS: The 5-year OS values would have fully satisfied the concept of LARGE, but almost no trials report such late survival results in the advanced setting. Therefore, we considered 2 or 3 years as late OS time points whenever the prognosis is below or above 12 months, respectively. Ranges between 2.5% and 20% were considered, and the range between 5% and 15% was chosen because little difference was observed between 2.5% and 5% and the selection afforded by 15% was already very high.

• Proportional increase in 2 or 3 year OS: After testing various ranges, values between 25% and 100% were chosen because these values reflects concepts of benefit that may be easily conveyed to the patients and to nonmedical stakeholders.

The postulated values defining mCMO for SMALL and LARGE were considered relative to the actual results of pivotal trials on new agents. These trials were selected according to the following criteria:

• Randomized phase III

• Advanced solid tumors (palliative intent of treatment)

• Published in the last 15 years

• Reporting OS, whether as primary or secondary endpoint.

For example, the trial on afatinib was not selected because OS data are not yet mature and only PFS data are reported in the published registration trial (5). Forty-three pivotal trials on 35,419 patients in 12 cancer types on 23 new drugs (6–49) were considered and the consequences of raising the bar of the mCMO on the outcome of the 43 trials are presented below.

Because it would be desirable to meet both criteria defining SMALL (HR for OS and gains in median OS), we have reported the number of trials that would meet both conditions (column “both” in Table 2). Only two trials reached the postulated “high” threshold values (Table 2). The number of “positive trials” increased to 8 and 15 when the bar was lowered to the “medium” and “low” levels, respectively (Table 2).

Table 2 also reports those trials meeting either criteria (column “either” in Table 2). The corresponding figures for the three levels of required benefit were 8, 18, and 23 positive trials out of 43. In addition, the table also reports which specific parameter each trial met (columns HR and median gain in OS). In general, meeting the postulated median gains was more common than achieving the HR thresholds.

Table 3 reports which drug would pass the thresholds for the different required benefit levels. It may be noted that trastuzumab, clearly a paradigm-changing agent for breast cancer, exceeds only the ‘medium’ and ‘low’ required benefit level, based upon the pivotal registration trial data (12). This underlines how high the proposed “high” threshold levels are.

The same analysis as above was performed for LARGE, as measured by the absolute OS gains and/or proportional OS increase at 2 or 3 years (Tables 4 and 5). The analysis of these parameters was done at 2 years for conditions with poorer prognosis (<12 months median survival with standard treatment) and at 3 years for conditions with better prognosis (>12 months) on trials reporting more than 20 patients at risk at these late times. Most trials lack this information: the number of qualifying trials was 18: 11 for the <12 months groups, and seven for the better prognosis groups. The table reveals that there are no trials satisfying both the absolute and proportional increase for the “high” required benefit levels (Table 4, columns “both”). Setting the bar at the “medium and low” threshold level (Table 4, sum of the 2 columns “both”) allows two and nine trials, respectively, to reach the postulated levels. If we consider the instances where trials satisfied the threshold of either endpoint (sum of the two columns “either” in Table 4), these numbers increase to 2, 8, and 11 for the high, medium, and low levels, respectively.

The current trend of conducting ever larger trials looking for smaller and smaller benefits that are statistically significant, but clinically marginal, has been strongly criticized (2–4, 50). We hope that the concepts proposed here set the stage for a new, more informed starting point for debate on these complex and controversial issues in three ways.

• By specifically defining the new concept of the mCMO, the proposal goes beyond the vague concepts of “clinically worthwhile,” “clinically relevant,” “large deltas,” etc.

• By clarifying that both LARGE and SMALL are valuable concepts.

• By providing a tentative set of thresholds for the four OS-related parameters for ranking the clinical value of currently available drugs.

It should be clear that the approach we are suggesting does not represent an attempt to model the effect of treatments on the prognosis of patients with cancer. In general, the treatment effects observed in a randomized clinical trial are compatible with a variety of statistical models, including those that assume a substantial heterogeneity in treatment effects across patients. For instance, a moderate increase in median OS may conceal a large effect in a small group of patients, with no effect in the remaining patients. What we propose here is simply an operational model that can prove useful to establish criteria for defining the clinical relevance of a new treatment.

It is quite obvious that patients and doctors would prefer pursuing LARGE rather than SMALL. However, experience in clinical oncology suggests two key lessons in this regard: (i) because cancer treatment advances are “incremental,” their cumulative effects are missed if SMALL is rejected a priory and (ii), in some instances, the detection of LARGE benefits in molecularly defined subgroups of patients did not derive directly from preclinical or early clinical studies, but this recognition was initiated by the retrospective identification of these molecular parameters within studies demonstrating SMALL (see the entire anti-EGFR story in advanced colorectal cancer). Hence, both types of the mCMO are relevant and should be pursued with the hope that LARGE will be pursued with increased frequency in molecularly defined populations.

First and most important, there is no question that the future of oncology is to navigate toward a molecular classification of cancer. This will imply recognizing driving mutations that will hopefully be targetable by new drugs. Pursuing SMALL in these conditions is clearly inappropriate. However, the SMALL philosophy, and consequently, conducting large trials on unselected patients populations, is still more prudent whenever no clear indications about the molecular determinants of treatment effect are available. On the other hand, it has been suggested that different trial strategies, such as smaller sample sizes, but relaxed alpha values for trials in molecularly defined subgroups, may provide larger benefits for patients in the long run (51). Simply “raising the bar” and aiming at larger treatment effect in smaller studies may lead to the discarding of treatments of worthwhile, though not outstanding, efficacy.

Second, our model considers OS-related parameters only. This choice derives from the need for simplicity. However, a major challenge in this regard is the relevance of crossover in studies in which PFS is the primary endpoint. The simple solution could be to adapt the required threshold level to lessen the requirements for improvement based on a certain percentage of cross-over. However, this would add an additional level of complexity because the extent of crossover is difficult to control. For example, crizotinib, a practice-changing biologic for non–small cell lung cancer (21), does not appear in the table as a success because, despite a huge PFS gain over chemotherapy alone, OS outcome has not been affected so far due to extensive preplanned cross-over. In other trials, cross-over is more limited, preserving the impact of the new agent on OS (12).

Third, prognosis is a continuum. Lumping conditions with 4 to 5 months median OS (late lines in advanced colorectal cancer) with others in the 8 to 9 months median OS range (advanced gastric cancer), imply per se 100% difference. The same can be said for conditions of much longer MST such as advanced renal cell carcinomas that now have a median OS of longer than 30 months and second-line breast cancer (much shorter median OS). With such heterogeneities, additional prognostic categories could be recommendable, again at the cost of increasing the already high complexity of the model.

Fourth, too few trials report the late effect of treatment on OS; and too often, the patients at risk at the predefined time of analysis of 2 and 3 years are too few (we accepted the very low number of 20 to have a sizable sample of trials). This limits substantially the power of this model to ascertain the parameters of LARGE. The present system of trial design, conduct, analysis, and reporting is not clearly focused on looking for LARGE. The reasons for this are understandable (time and size of trials issues). However, either prolonging the follow-up time of the trials to allow data maturation or dedicating a second publication to long-term results is to be encouraged.

We believe that the basic principles (mCMO, SMALL, and LARGE) as well as the structured approach presented here provide a critical refinement to the “raising the bar philosophy” recently emphasized. We also recognize that additional fine tuning or more substantial adjustments to these concepts or threshold values by all stakeholders will improve the model further.

A.F. Sobrero reports receiving speakers bureau honoraria from and is a consultant/advisory board member for Amgen, Bayer, Merck, Roche, and Sanofi. P. Bruzzi reports receiving speakers bureau honoraria from Bristol-Myers Squibb, Novartis, and Roche. No potential conflicts of interest were disclosed by the other authors.

Conception and design: A.F. Sobrero, P. Bruzzi

Development of methodology: A.F. Sobrero, A. Pastorino, P. Bruzzi

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): A.F. Sobrero, A. Pastorino

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): A.F. Sobrero, A. Pastorino, D.J. Sargent

Writing, review, and/or revision of the manuscript: A.F. Sobrero, D.J. Sargent, P. Bruzzi

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): A.F. Sobrero, A. Pastorino

Study supervision: A.F. Sobrero

We thank Lee M. Ellis, MD, for his review of the manuscript and insightful comments.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.