Toxicity and Efficacy Probability Interval Design for Phase I Adoptive Cell Therapy Dose-Finding Clinical Trials

In the past few years, promising antitumor effects have been seen in patients with late-stage cancer when they were treated with adoptive cell therapy (ACT), including chimeric antigen receptor (CAR) T cells, T-cell receptors (TCR), and tumor-infiltrating lymphocytes (TIL). Early data continue to warrant the expedited development of these therapies as a treatment option for patients with various cancers. ACT is a highly personalized cancer therapy that harnesses a patient's own immune cells (specifically, T lymphocytes) to recognize cancer-specific abnormalities, enabling them to target and attack malignant cells throughout the body. However, challenging questions remain regarding the immunobiology and development of these personalized therapies, such as the design of an appropriate phase I dose-escalation study that takes into consideration the unique properties of ACTs.

Traditional phase I dose-finding oncology trials aim to identify the maximum tolerated dose (MTD), which is the highest dose that has a dose-limiting toxicity (DLT) rate less than or close to a prespecified target rate, say 30%. The commonly used methods are the rule-based designs, such as the 3 + 3 design (1), and the model-based designs, such as the modified toxicity probability interval (mTPI) design (2–4) and the continual reassessment method (CRM; ref. 5).

Traditional designs considering the DLT data implicitly assume a monotonically increasing relationship between dose and response efficacy; otherwise, there is no justification to escalate the dose level when it is safe. Monotone efficacy may be a reasonable assumption for cytotoxic agents. However, this assumption may not hold for ACTs. In ACT trials, the MTD is not always optimal, as clinical response correlates less with dose level. For example, recent phase I trials of CAR T cells targeting CD19 suggest that a range of dose levels is generally safe and effective, and there is no clear correlation between T-cell dose and clinical response (6). In particular, two patients in the CAR T-cell trials reported in refs. 7 and 8 exhibited inferior efficacy outcomes to another patient despite receiving a 10-fold higher T-cell dose. A TIL study for metastatic cancer (9) found no correlation between the number of cells administered and the likelihood of a clinical response. Therefore, traditional algorithmic and model-based designs based on binary measures of toxicity may be inappropriate for identifying the optimal dose of ACTs. Given the complexity of an ACT product, preliminary dose exploration should aim to capture effective biologic activity rather than dose-limiting toxicity alone (10).

Emerging data suggest that CAR T-cell therapy in B-cell hematologic malignancies may induce rapid responses. Toxicity and efficacy of a biomarker (e.g., cell expansion) may, therefore, be measured in the same time frame (6, 11, 12). Model-based methods have been developed to model toxicity and efficacy data jointly in order to determine the acceptable dose level (13); these methods are powerful and effective, although they may require substantial investigation from trial statisticians to ensure proper implementation and calibration. To this end, we developed a practical dose-finding design for an ACT that (i) incorporates both toxicity and efficacy data; (ii) provides the same adaptive feature as model-based designs; and (iii) most importantly, is transparent to clinicians and as simple to implement as the 3 + 3 and mTPI designs. We propose a toxicity and efficacy probability interval (TEPI) design, which is based on a clinician-elicited decision table in terms of efficacy and toxicity probability intervals. This design is motivated for the conduct of a phase I CD19-targeted CAR T-cell therapy dose-finding trial.

Consider d ascending doses in a single-agent ACT phase I trial. Due to ethical considerations, it is always assumed that the toxicity probability p_i increases with dose level i. However, the efficacy probability q_i may increase initially and then reach a plateau from which minimal improvement or even decreasing efficacy may be seen with increasing dose. For this reason, we assume that q_i is not monotone with i, and that p_i and q_i are independent. Suppose that dose i is currently used in the trial and n_i patients have already been allocated to dose i, with x_i and y_i patients experiencing toxicity and efficacy outcomes. Aggregating across all the doses, the trial data are denoted as |$D = \{ {({n_i},{x_i},{y_i}),\;i = 1, \ldots ,d} \}$|⁠.

Similar to mTPI, we partition the unit intervals (0, 1) for p_i and q_i into subintervals. Denoting (a, b) and (c, d) a subinterval in the partition for p_i and q_i, respectively. The interval combination |(a,\,b) \times (c,\,d)$| forms the basis for dose-finding decisions, with each combination corresponding to a specific decision, such as dose escalation or de-escalation. A dose-finding decision table can then be elicited with clinicians for all interval combinations. An example of such a two-dimensional table is given in Table 1. We call this the “preset table" for the TEPI design, which is fixed and elicited prior to the trial. As illustrated in Table 1, there are four subintervals for toxicity (rows) and efficacy (columns), the intersection of which forms 16 interval combinations. Each of the 16 combinations corresponds to a specific dosing decision. Decision “E” denotes escalation (i.e., treating the next cohort of patients at the next higher dose). Decision “S” denotes staying at the current dose for treating the next cohort of patients. Decision “D” denotes de-escalation (i.e., treating the next cohort of patients at the next lower dose). These reflect practical clinical actions when the particular combination of toxicity and efficacy data are observed at a certain dose level. For example, Table 1 shows that the interval combination |(0,0.15) \times (0,0.2)$| for p_i and q_i corresponds to an action of “E”—escalation. This means that if the observed toxicity rate for a dose falls in (0, 0.15) and the observed efficacy rate is in (0, 0.2), the next patient cohort will be treated at the next higher dose level. In order to formulate this table, it is required to have determined: (i) the maximum tolerated toxicity rate, p_T, and (ii) the minimum acceptable efficacy rate, q_E, at which the clinician is willing to treat future patients at the current dose level.

Table 1 is elicited for an ACT trial based on |{p_T} = 0.4$|⁠, |{q_E} = 0.2$|⁠, historical data for the ACT, and clinician input. For other trials, the preset table can be modified accordingly. A more detailed description of Table 1 is given in Supplementary Material A.

Building upon the preset table, we set up a “local” decision-theoretic framework and derive a Bayes rule. Here, local means that the framework focuses on the optimal decision to be made for the current dose instead of the trial. We show that the Bayes rule is equivalent to computing the joint unit probability mass (JUPM) for the toxicity and efficacy probability intervals. For a given region A, the JUPM is defined as the ratio between the probability of the region and the size of the region (2, 14). Considering the two-dimensional unit square |(0,1) \times (0,1)$| in the real space, the JUPM for each interval combination |(a,b) \times (c,d)$| is

Here, the numerator, |Pr\{ {p_i} \in (a,b),\;{q_i} \in (c,d)|D\} $|⁠, is the posterior probability of p_i and q_i falling in the interval (a, b) and (c, d), respectively.

Assume the prior for each p_i follows an independent |beta({\alpha _p},{\beta _p})$|⁠, and the prior for each q_i follows an independent |beta({\alpha _q},{\beta _q})$|⁠, where |beta(\alpha ,\beta )$| denotes a beta distribution with mean |\alpha /(\alpha + \beta )$|⁠. The rationale of using independent priors follows the same argument in (2). Assume |{x_i}|{p_i}:Bin({n_i},{p_i})$| and |{y_i}|{q_i}:Bin({n_i},{q_i})$|⁠, where |Bin(n,q)$| denotes a binomial distribution with n trials and q probability of success. Then, the likelihood function for the observed toxicity data |({x_i},{n_i}),i = 1,...\,,d$|⁠, is a product of binomial densities |l(p) = \prod\nolimits_{i = 1}^d p_i^{{x_i}}{(1 - {p_i})^{{n_i} - {x_i}}}$|⁠, and the likelihood function for the efficacy data |({y_i},{n_i}),i = 1,...,d$| is a product of binomial densities |l(q) = \prod\nolimits_{i = 1}^d q_i^{{x_i}}{(1 - {q_i})^{{n_i} - {x_i}}}$|⁠. Thus, the posterior distributions for p_i and q_i are |beta({\alpha _p} + {x_i},{\beta _p} + {n_i} - {x_i})$| and |beta({\alpha _q} + {y_i},{\beta _q} + {n_i} - {y_i})$|⁠, respectively. Based on the posterior distributions, there exists a “winning” interval combination |({a^*},{b^*}) \times ({c^*},{d^*})$| that achieves the maximum JUPM among all the combinations in Table 1, and the corresponding decision for that combination is selected for treating the next cohort of patients. It can be shown that the decision is the Bayes rule under a balanced loss function (Supplementary Material B).

The basic dose-finding concept of TEPI is as follows. Assume that the current patient cohort is treated at dose i. After the current cohort completes DLT and response evaluation, compute the JUPMs for all the interval combinations in Table 1. The TEPI design recommends “E,” “S,” or “D” corresponding to the combination with the largest JUPM value. Because for a given trial there are a finite number of possible toxicity and efficacy outcomes as binomial counts, for any toxicity and efficacy counts that can be observed in the trial, the TEPI dose-finding decisions can be precalculated. For our ACT trial, based on Table 1, all the decisions have been precalculated and presented in Table 2. Similar to the decision table for mTPI, this table enables clinicians to conduct the trial with transparency.

In practice, the TEPI design needs to be calibrated according to physicians' needs. This is transparent and requires little effort. The tuning is for the intervals in Table 1 so that the decisions in Table 2 are satisfactory to the clinicians. Specifically, by modifying the interval combinations in Table 1, a new decision table can be generated in the form of Table 2. The tuning is completed once the desirable decisions are obtained.

To enable ethical constraints, we introduce two additional rules as part of the dose-finding algorithm. One is to exclude any dose with excessive toxicity, and the other is to exclude any dose with unacceptable efficacy.

• Safety rule: If |Pr({p_i} \gt {p_T}|D) \gt \eta $| for a |\eta $| close to 1 (say, 0.95), exclude dose |i,i + 1,\,_\cdots ,d$| from future use for this trial (i.e., these doses will never be tested again in the trial) and treat the next cohort of patients at dose |i - 1$|⁠. This corresponds to a dosing action of “DU_T”—de-escalate due to unacceptable high toxicity.

• Futility rule: If |Pr({q_i} \,\gt \,{q_E}|D) \lt \,\xi $| for a small |\xi $| (say 0.3), then exclude dose i from future use in the trial. This corresponds to a dosing action of “EU”—escalate and never return due to unacceptable low efficacy—or “DU_E”—de-escalate and never return this dose due to unacceptable low efficacy.

A dose level is considered “available" if it satisfies both the safety and futility rules, and only these doses can be used to treat subsequent patients.

At the end of the trial, we select the most desirable dose based on a utility score to balance the toxicity and efficacy tradeoff. Utility-based decision criteria have been adopted widely in recent dose-finding trials (15–17). An elicited utility function for safety and efficacy can be constructed based on p_T and q_E through discussions with clinicians. For example, Fig. 1 (left) shows the utility function f₁(p) for safety, where p denotes the toxicity rate. Utility f₁(p) is set to 1 if p ≤ 15%, 0 if p > 40%, and linearly decreasing with p if |p \in (15\% ,40\% ).$| Utility f₂(q) is set in a similar fashion. Figure 1 shows both utility functions. We assume a monotonic constraint on priors for p_i's while selecting the best dose (i.e., |{p_1} \le {p_2} \le \cdots {p_d}$|⁠). The utility score function is defined as |U(p,q) = {f_1}(p){f_2}(q)$|⁠, where p denotes the toxicity rate, and q denotes the efficacy rate. Both |{f_1}( \cdot )$| and |{f_2}( \cdot )$| are truncated linear functions, given by

where p*'s and q*'s are prespecified cutoff values. For each dose i, we use a simple numerical approximation approach to compute the posterior expected utility, |E[U({p_i},{q_i})|D]$|⁠. We generate a total of T random samples from the posterior distributions. For each sample t, we generate |{p^t} = (p_1^t, \ldots ,p_d^t)$| and |{q^t} = (q_1^t, \ldots ,q_d^t)$| as a random sample of d probabilities of toxicity and efficacy, respectively. We perform the isotonic transformation (2, 14) on p^t to obtain |{\hat{p}^t} = (\hat{p}_1^t, \ldots ,\hat{p}_d^t)$| where |\hat{p}_i^t \le \hat{p}_j^t$| if |i \lt j$|⁠. This ensures that |\hat{p}_i^t$| values are nondecreasing. For each dose i, based on the samples |q_i^t$| and |\hat{p}_i^t$|⁠, a corresponding utility score is |{U^t}(\hat{p}_i^t,q_i^t) = {f_1}(\hat{p}_i^t){f_2}(q_i^t)$|⁠. Then, the estimated posterior expected utility is given by

Finally, selected the optimal dose |\hat{d}$| given by

Trial conduct under TEPI is simple and transparent. During the trial, all dose-finding decisions follow Table 2 and the safety and futility rules. The steps of implementing TEPI design are as follows:

• Clinicians choose a starting dose, the maximum tolerable toxicity rate (p_T), and the minimum acceptable efficacy rate (q_E).

• Elicit the preset decision table (e.g., Table 1) and derive the dose-finding table (e.g., Table 2) to ensure that they reasonably reflect clinical practice during the trial (calibrate the intervals in the decision table as needed based on computer simulations).

• A dose is “available” if |Pr({p_i} \gt {p_T}|D) \lt \eta $| and |Pr({q_i} \gt {q_E}|D) \gt \xi .$| If no dose is available, terminate the trial.

• The “current” dose is the dose used to treat the current cohort of patients.

• If the current dose invokes the safety rule, de-escalate to the closest available dose below the current dose.

• If the current dose invokes the futility rule,

  • – If the decision is “E,” escalate to the closest available dose above the current dose. If no doses above the current dose are available, de-escalate to the closest available dose below the current dose. (This rule is justified because we do not assume a monotonic dose–efficacy relationship. That is, although the current dose is not effective, an effective dose could be either a higher dose or a lower dose).

  • – If the decision is “D,” de-escalate to the closest available dose below the current dose. If no doses below the current dose are available, terminate the trial.

  • – If the decision is “DU_T,” mark the current dose and all higher doses as unavailable. De-escalate to the closest available dose below the current dose. If no doses are available, terminate the trial.

  • – If the decision is “S,” de-escalate to the closest available dose below the current dose. If no dose below the current dose is available, terminate the trial.

• If the current dose invokes both safety and futility rules, de-escalate to the closest available dose below the current dose. If no doses below the current dose are available, terminate the trial.

• Do not skip untried doses.

• Stop the trial at a prespecified sample size if it has not been terminated by then.

• Select the optimal dose |\hat{d}$| in (2) as the final dose.

To characterize the TEPI design and compare the performance of TEPI with existing designs, we simulate clinical trials under the following six different scenarios. In most ACT dose-finding trials to date, four or fewer doses are investigated (12, 18–20). Therefore, we assume four doses in each scenario. For each scenario, we specify the true toxicity and efficacy probabilities for all four doses, and generate random binary toxicity and efficacy outcomes based on these probabilities. We compare with toxicity-based designs, such as the 3 + 3, mTPI, and CRM designs, and with toxicity–efficacy-based designs, such as the EffTox design (13). A full description of all the scenarios is provided in Supplementary Material C.

For TEPI, the simulation is based on the dose-finding decision described in Tables 1 and 2 using hyperparameters |{\alpha _p} = {\beta _p} = {\alpha _q} = {\beta _q} = 1$|⁠. For mTPI, the equivalence interval is set to [0.25, 0.35], and the dose-finding table is given in Table 3. We use 0.35 as the upper bound of the equivalence interval for mTPI instead of 0.4, because it is difficult to justify such a high-toxicity rate without considering efficacy data. NextGen-DF (4) is used to implement the standard 3 + 3 design and the CRM design (21). For the EffTox design, we use the EffTox software downloaded from https://biostatistics.mdanderson.org/softwaredownload/SingleSoftware.aspx?Software_Id=2 and set |{p_T} = 0.4$|⁠, |{q_E} = 0.4$|⁠, |{\pi _1} = (0.2,0)$|⁠, |{\pi _2} = (1,0.6)$|⁠, and |{\pi _3} = (0.5,0.5)$|⁠, which is compatible with the proposed utility function. Under each simulation scenario, we ran 1,000 simulated trials with a maximum sample size of 27 patients and cohort size of 3.

Table 4 summarizes the simulation results. In scenario 1, all four doses are tolerable but unacceptable due to low efficacy rates. Under the TEPI design, 35% of the trials terminate early with an average sample size of 21. In contrast, the mTPI and CRM designs rarely stop early and exhaust the sample size, whereas the 3 + 3 design stops early 22% of the time and EffTox stops early 92% of the time, which means that EffTox is most efficient at minimizing the number of patients treated on a drug (or the selected dose ranges of a drug) having minimal efficacy. Except for EffTox, TEPI is better than other designs in this scenario.

Scenario 2 is an extreme case where all doses are safe but have the same high efficacy rate, so the starting dose has the highest utility. TEPI selects this dose level 84% of the time compared with 6% to 66% for the other designs. Besides, TEPI puts more patients on average at dose levels 1 and 2 and fewer patients at dose levels 3 and 4 than the mTPI and CRM designs.

Under scenario 3, dose levels 1 to 3 are safe, dose level 4 is unsafe, and dose level 2 has the highest efficacy. TEPI selects dose level 2 88% of the time and allocates an average of 12 patients to it. In contrast, mTPI and CRM select this dose level with much lower frequencies and allocate fewer patients. Moreover, mTPI and CRM select dose level 3 (the MTD) 60% of the time, whereas TEPI selects this level only 0.3% of the time. A similar trend is observed for dose level 4. While EffTox selects dose level 2 mostly in trials that do not stop early, it stops early with a rate as high as 50%, which is not necessary because there is a desirable dose in this scenario.

In scenario 4, the two lower doses are safe and effective, while the two higher doses are efficacious but unsafe. There is a monotonically increasing relationship between efficacy and dose, which is the underlying assumption for the mTPI, 3 + 3, and CRM designs. In this scenario, dose level 2 has the highest utility. The four designs select this dose with similar frequency (∼40%), which demonstrates that TEPI has good performance characteristics even under the conventional assumptions. However, it is interesting to note that 54% of the time, TEPI selects dose level 1, which has slightly lower utility than dose level 2.

On the other hand, the mTPI, CRM, and EffTox designs are aggressive, selecting an unsafe dose (level 3 or 4) 46%, 49%, and 27% of the time, respectively; in comparison, the 3 + 3 design selects an unsafe dose 14% of the time, and TEPI only 5% of the time.

In scenario 5, all doses are tolerable, and efficacy increases from dose level 1 but plateaus at dose level 2. Dose level 2 is optimal. TEPI selects this dose level 65% of the time, allocating an average of 8.6 patients. In contrast, mTPI, CRM, and EffTox select dose level 2 32%, 25%, and 48% of the time, respectively, while selecting the suboptimal dose level 3 44%, 45%, and 38% of the time.

In scenario 6, all doses are too toxic despite having acceptable efficacy. In this case, TEPI terminates early 66% of the time, with an average sample size of 17, and mTPI, 3 + 3, and EffTox terminate early 76%, 89%, and 69% of the time, respectively, with average sample sizes of 15, 5, and 16. CRM does not stop early, with an average of 26 patients treated at dose levels 1 and 2. Despite its ability to stop early with a slightly higher rate than TEPI, EffTox aggressively selects dose level 2 12% of the time. Here, TEPI seems more aggressive than mTPI, which is expected because dose level 1 has acceptable efficacy.

In all scenarios where an acceptable dose exists (scenarios 2, 3, 4, 5), the 3 + 3 design is less likely to select the desirable dose compared with the TEPI. It appears that the 3 + 3 design is too conservative in that it is unable to escalate quickly, even when the doses are safe, consistent with previous finding (3). As CRM and mTPI do not incorporate efficacy data in the trial conduct, TEPI is superior in scenarios 1 to 3 and 5 and performs well in scenario 4. EffTox performs better than TEPI only in scenario 1, and in scenarios 2 to 6, TEPI is more desirable than EffTox.

We performed a sensitivity study to evaluate the impact of varying sample sizes on the ability of the TEPI design to identify the optimal dose. We arbitrarily selected scenario 3 and created a new scenario (scenario 7) with |p = (0.1,0.2,0.3,0.7)$| and |q = (0.05,0.2,0.5,0.6)$|⁠. Figure 2 plots the frequency of selecting the true optimal dose level against sample sizes of 15, 27, and 48. For each scenario and sample-size combination, we simulated 1,000 trials. In both scenarios, the selection probability increases with sample size. This observation is not surprising and supports the importance of considering the tradeoff between sample size and the precision of dose selection. However, it is worth noting that even for the small sample size of 15 or 27, the TEPI design performs reasonably well.

Traditional phase I designs use only toxicity data and may not be appropriate for ACT dose-finding trials, because efficacy may not increase with increasing dose. The proposed TEPI design attempts to address this problem by accounting for both efficacy and toxicity simultaneously. Typically in ACT trials, efficacy or activity biomarkers can be observed quickly, oftentimes as fast as the toxicity outcomes. This makes TEPI feasible. However, when efficacy outcome cannot be quickly observed, either delayed enrollment is required to use TEPI or statistical models must be modified to account for delayed efficacy outcomes.

The TEPI design is simple, transparent, and appealing because all dose-escalation decisions can be prespecified prior to the trial start. During the trial conduct, no design modification or “black-box” statistical calculation is required. Through simulation studies, we have demonstrated the superiority of the TEPI design in selecting and allocating more patients to the true optimal dose over the 3 + 3, mTPI, and CRM designs, especially when the assumption of monotone increasing efficacy is violated. Even if the monotonic relationship is true, the proposed design is still superior to the 3 + 3 design. More importantly, TEPI terminates the trial earlier when all tested doses are safe but no dose is likely to be efficacious. This is certainly more ethical than to continue to expose patients with cancer to a safe but ineffective drug.

The proposed TEPI design is a natural extension of mTPI by adding the efficacy interval into the dose-finding model. To use this design properly, we recommend a close collaboration between clinicians and statisticians to determine the initial design parameters, such as the interval combinations and corresponding dose-finding decisions in Table 1, the utility function, and the safety and futility stopping rules.

The TEPI design uses the utility function of safety and efficacy to choose the optimal dose. Depending on the clinical situation, one could use a different metric to make the final dose selection. For example, the dose with the highest probability |Pr({p_i} \lt {p_T},{q_i} \gt {q_E} + \delta |D)$| may be selected as the best dose, where δ is the expected increment over the minimum efficacy rate. A limitation of TEPI may be the assumption of independence of the safety and efficacy data. In general, the true relationship between safety and efficacy is complex, and it has been demonstrated that this independence assumption has negligible effects (22, 23). Finally, we reemphasize the importance of simplicity, flexibility, and transparency of using TEPI in designing a dose-finding trial.

D.H. Li holds equity interest (including patents) in Juno Therapeutics, Inc. J.B. Whitmore holds equity interest in Juno Therapeutics, Inc. Y. Ji is co-founder of and holds ownership interest (including patents) in Laiya Consulting, Inc. and is a consultant for Takeda Pharmaceuticals USA, Inc. No potential conflicts of interest were disclosed by the other author.

Conception and design: D.H. Li, J.B. Whitmore, W. Guo, Y. Ji

Development of methodology: D.H. Li, J.B. Whitmore, W. Guo, Y. Ji

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): D.H. Li, J.B. Whitmore, W. Guo, Y. Ji

Writing, review, and/or revision of the manuscript: D.H. Li, J.B. Whitmore, W. Guo, Y. Ji

Administrative, technical, or material support (i.e., reporting or organizing data, constructing databases): D.H. Li