Accuracy, Safety, and Reliability of Novel Phase I Designs—Response

We appreciate that Drs. Wages and Braun agree with us on the importance of studying the within-trial behavior of phase I methods, which is one of the primary objectives of our article and the reason why we proposed and investigated the rationality of dose assignments in various phase I methods (1). In our viewpoint, that is also one of the major contributions of our article: call researcher's attentions to within-trial behavior that has been largely ignored by existing phase I methodologic literature.

In the section “Performance metrics” in our article, we defined “irrational dose assignment” as the failure to deescalate the dose when 2 of 3 or ≥ 3 of 6 patients had dose-limiting toxicities (DLT) at a given dose. Given that definition, we showed that model-based methods may make irrational dose assignments due to model misspecification and that model-assisted designs (e.g., BOIN method) do not make such irrational dose assignments. Our definition of irrational dose assignment was chosen to represent some of the most obvious and intuitive cases in practice. It is certainly not the only possible definition and was not intended to cover all possible irrational dose assignments.

Drs. Wages and Braun provided a hypothetical trial example under the BOIN method and raised the concern that “no matter what DLT outcome was observed for patient 16, the observed DLT rate at dose level 2, either 0 of 6 or 1 of 6, would guarantee that the BOIN method would return to dose level 3 for patient 17. Therefore, the information obtained from the participation of patient 16 in the study does not contribute to the dose assignment algorithm for patient 17. Overall, data observed for 8 of 36 (22%) patients accrued to the study (patients 9, 13, 16, 17, 19, 21, 23, and 30) does not inform the dosing decision for the following patient accrued to the study.”

The trial example is interesting, but we respectfully disagree with the concerns. In the sequential decision-making process such as that in phase I trials, it is normal and common for a specific patient's outcome to have no impact on the decision making immediately after that patient (e.g., the dose assignment for the patient accrued immediately after that patient), especially when the decision is limited to multiple discrete choices. For example, at a given dose, if the first 5 patients do not experience DLT, the 6th patient's outcome (DLT/no DLT) will not change the decision of dose assignment (i.e., escalating the dose) for the very next patient, given the rule that we escalate the dose if ≤1 of 6 patients has DLT. This phenomenon also often occurs in model-based designs when the outcome of a patient does not change the model estimates enough to trigger the change in the decision of dose assignment for the very next patient (e.g., when the estimate of DLT rate for the current dose is closest to the target no matter whether the current patient has DLT or not). Figure 1 shows a trial example of the CRM constructed under the same setting as that used by Drs. Wages and Braun, in which the outcome (DLT/no DLT) of patients 11, 14, 17, 18, 19, 20, 22, 23, 24, 25, 26, 27, 29, 30, 31, 32, 33 (i.e., 17/36 or 47.2% patients) has no impact on the decision of dose assignment (i.e., stay at the same dose) for the patient accrued immediately after them.

A more important point is that although it is true that the outcome (DLT/no DLT) of some patients may not alter the dose assignment for the patient accrued immediately after them, that does not mean that the data have no impact on the dose assignment further downstream in the trial. Actually, these patients’ data (DLT/no DLT) have profound impact on and do contribute to the dose assignment for patients accrued thereafter, but not immediately after them. Using the trial example provided by Drs. Wages and Braun, suppose that patient 9 had DLT, rather than no DLT. This change has no impact on the dose assignment for patient 10 (still treated at dose 2), as noted by Drs. Wages and Braun, but it will lead to the elimination of dose 3 after patient 12 is treated, as 3 of 4 patients had DLTs at dose 3. The dose elimination/early stopping rule is described at the end of the fourth paragraph on page 4360, as an integrated component of the dose assignment rule of the BOIN design, which says that a dose will be eliminated if Pr (DLT rate of the dose > target | data) > 0.95. As a result, after patient 12, the dose will never be escalated back to dose 3, which completely changes the remaining course of the trial. More generally, as the estimate of the DLT rate of a dose is based on all accumulated data at that dose, and the decision of dose assignment for the BOIN method is based on that estimate, it is not difficult to see that even though a patient's data may have no impact on the dose assignment for the very next patient, the data may still change the dose assignment for patients accrued later and thus alter the course of the trial. Therefore, it is not true that “… data observed for 8 of 36 (22%) patients accrued to the study (patients 9, 13, 16, 17, 19, 21, 23, and 30) does not inform the dosing decision for the following patient accrued to the study.”

Finally, Drs. Wages and Braun commented that “It is not clinically sensible to view as irrational the failing to deescalate when ≥3 of 6 patients experienced DLTs, but then allow, as rational, returning to that dose when the very next patient has a DLT at the next lowest dose level.” We do not see escalating the dose when the very next patient experiences DLT at the next lowest-dose level as a problem. Again, when evaluating the safety of a dose, investigators will base the evaluation on all patients treated at that dose, rather than only the last treated patient. This is particularly true for phase I trials, where patients are highly heterogeneous, ranging from having very poor to good prognoses. It is difficult to imagine that clinicians would make their decisions based on only the very last patient, while ignoring all the patients treated previously at the same dose. For example, at a given dose, if the most recently treated patient experienced DLT (perhaps due to his/her poor prognosis), but the previous 5 patients treated at that same dose did not experience DLT, in light of the observed data that 1 of 6 patients experienced DLT, based on our experience, most clinicians would agree that the dose is safe and therefore would be comfortable with dose escalation when the target DLT rate is 25%. It is possible that after BOIN confirms that the current dose level d is very safe (e.g., 1/6 patients had DLT), it may escalate back to dose level d + 1, at which previously 2 of 3 patients had DLTs, as in Drs. Wages and Braun's example. That action is not unreasonable because if the observed data indicate that d is very safe, it is likely that the relative high toxicity rate (2/3 DLTs) previously observed at d + 1 may be due to the randomness of small sample size and it is reasonable to “revisit” d + 1 to collect more data. If the new data observed at d + 1 confirm that d + 1 is indeed overly toxic, for example, we observe 2/3 DLTs again and hence cumulatively 4 of 6 patients had DLTs at d + 1, BOIN will eliminate that dose and prohibit any future revisit to d + 1. It is important to note that when the target DLT rate is slightly lower, such as 24% or lower, BOIN will immediately eliminate any dose with 2 of 3 DLTs and the phenomenon described by Drs. Wages and Braun will never occur. In contrast, model-based designs still have substantial chances to continue treating the next cohort of patients at (or escalate back to) a dose where 2 of 3 or ≥ 3 of 6 patients have experienced DLTs, even when the target DLT rate is 20%, see Figure S8 of Zhou and colleagues (2). In the case that investigators concern about the dose revisit as Drs. Wages and Braun did, it can be easily addressed by lightly tightening the BOIN's dose elimination rule by decreasing its cutoff from 0.95 to 0.94, that is, a dose will be eliminated if Pr (DLT rate > target | data) > 0.94. This will eliminate any dose with 2 of 3 DLTs and thus prohibit revisiting that dose.

See the original Letter to the Editor, p. 5482