Purpose:

Integration of evolutionary dynamics into systemic therapy for metastatic cancers can prolong tumor control compared with standard maximum tolerated dose (MTD) strategies. Prior investigations have focused on monotherapy, but many clinical cancer treatments combine two or more drugs. Optimizing the evolutionary dynamics in multidrug therapy is challenging because of the complex cellular interactions and the large parameter space of potential variations in drugs, doses, and treatment schedules. However, multidrug therapy also represents an opportunity to further improve outcomes using evolution-based strategies.

Experimental Design:

We examine evolution-based strategies for two-drug therapy and identify an approach that divides the treatment drugs into primary and secondary roles. The primary drug has the greatest efficacy and/or lowest toxicity. The secondary drug is applied solely to reduce the resistant population to the primary drug.

Results:

Simulations from the mathematical model demonstrate that the primary–secondary approach increases time to progression (TTP) compared with conventional strategies in which drugs are administered without regard to evolutionary dynamics. We apply our model to an ongoing adaptive therapy clinical trial of evolution-based administration of abiraterone to treat metastatic castrate-resistant prostate cancer. Model simulations, parameterized with data from individual patients who progressed, demonstrate that strategic application of docetaxel during abiraterone therapy would have significantly increased their TTP.

Conclusions:

Mathematical models can integrate evolutionary dynamics into multidrug cancer clinical trials. This has the potential to improve outcomes and to develop clinical trials in which these mathematical models are also used to estimate the mechanism(s) of treatment failure and explore alternative strategies to improve outcomes in future trials.

Translational Relevance

Most current cancer treatments use multiple drugs with different mechanisms of action, but the strategy for optimally combining them is typically nonquantitative and does not integrate evolutionary dynamics of resistance. Furthermore, trial analysis focuses on outcomes based on survival and response but not mechanisms of treatment failure or alternative treatment strategies. Here, we present a multidisciplinary study analyzing the evolutionary dynamics of multidrug cancer therapy in the context of an ongoing evolution-based (adaptive therapy) trial treating metastatic castrate-resistant prostate cancer with two translational goals: (i) demonstrate a strategy by which two drugs can be administered to delay onset of progression and (ii) demonstrate a strategy for cancer clinical trial design that includes a mathematically framed resistance management plan and uses computational analysis in each patient who progresses to understand the mechanism(s) of failure and determine treatment strategies that would have improved the outcome in that individual.

Successful cancer therapies that kill malignant cells also apply strong selection pressure for resistance, leading to treatment failure and tumor progression (1, 2). For example, men presenting with metastatic prostate cancer (mPC) therapy are almost always treated with androgen deprivation therapy (ADT; refs. 3, 4) by administering a drug that blocks normal cell testosterone production. ADT results in a significant tumor response in the vast majority of men, and the serum biomarker [prostate-specific antigen (PSA)] returns to the normal range or becomes undetectable in >80% of patients (4). Despite this high level of efficacy, ADT is virtually never curative, and invariably resistant populations emerge, leading to progression within 1 to 3 years (4).

One recent literature survey showed that while evolutionary terms rarely appeared in papers studying cancer therapeutic relapse before 1980 (<1%; ref. 5), the use of evolutionary terms has steadily increased more recently, due to the potential benefits of studying therapeutic relapse from an evolutionary perspective. The field of evolutionary dynamics encompasses three fundamental concepts (6). First, heritable variation allows individuals to pass on traits to descendants. Second, there is a struggle for existence, competition between individuals for survival and proliferation. Lastly, the variation in inheritance influences the competition. Leveraging the power of mathematical models in evolutionary dynamics is particularly useful in the study of the emergence of cancer resistance.

The evolutionary dynamics leading to resistance in ADT in prostate cancer apply equally well to most cancers and most cancer therapies. Standard-of-care (SOC) practice typically applies drugs at the maximum tolerated dose (MTD) continuously until progression (7). The goal of therapy, even in a clinical setting in which cure is not possible, is to kill as many cancer cells as possible, reasoning that this will produce the best patient outcome. However, MTD treatment strategy has been questioned based on evolutionary principles because it places strong selective pressure for the evolution of treatment resistance while eliminating all treatment-sensitive phenotypes that could potentially compete with and suppress the proliferation of the resistant populations (see Fig. 1A). These evolutionary dynamics are well recognized in controlling invasive pests and infectious disease therapies as “competitive release” (8). Thus, the evolutionary dynamics of MTD therapy may actually accelerate the proliferation of resistant phenotypes and subsequent treatment failure.

Figure 1.

An illustration of evolutionary dynamics of abiraterone and docetaxel therapy. A, MTD administers abiraterone continuously. Blue cells are treatment sensitive (T+ and TP), and yellow cells are treatment resistant (T−). Treatment-sensitive cells are rapidly killed, leaving an absence of competition for space and resources for resistant populations to emerge, known as “competitive release.” B, Adaptive abiraterone (AT) treatment is an evolutionary-minded strategy to leave a controllable sensitive cell population (blue), which suppresses a smaller resistant population (yellow). Treatment holidays allow the sensitive cells to grow back, suppressing resistant growth for a longer period of time until inevitable treatment failure. C, A P-S approach to adaptive therapy. Docetaxel (our secondary treatment) is used to control the cells resistant to abiraterone (our primary treatment). As adaptive abiraterone therapy (AT) progresses, docetaxel is administered during later cycles to control emergence of proliferating resistant T− cells (yellow). Inevitably resistance to docetaxel will evolve, leading to treatment failure and relapse of T−/− cells (purple).

Figure 1.

An illustration of evolutionary dynamics of abiraterone and docetaxel therapy. A, MTD administers abiraterone continuously. Blue cells are treatment sensitive (T+ and TP), and yellow cells are treatment resistant (T−). Treatment-sensitive cells are rapidly killed, leaving an absence of competition for space and resources for resistant populations to emerge, known as “competitive release.” B, Adaptive abiraterone (AT) treatment is an evolutionary-minded strategy to leave a controllable sensitive cell population (blue), which suppresses a smaller resistant population (yellow). Treatment holidays allow the sensitive cells to grow back, suppressing resistant growth for a longer period of time until inevitable treatment failure. C, A P-S approach to adaptive therapy. Docetaxel (our secondary treatment) is used to control the cells resistant to abiraterone (our primary treatment). As adaptive abiraterone therapy (AT) progresses, docetaxel is administered during later cycles to control emergence of proliferating resistant T− cells (yellow). Inevitably resistance to docetaxel will evolve, leading to treatment failure and relapse of T−/− cells (purple).

Close modal

An alternative strategy, termed “metronomic therapy” (9, 10), typically administers drugs continuously or at frequent intervals. This approach attempts to reduce drug toxicity, and the continuous or frequent administration is thought to inhibit tumor angiogenesis (11). Although the individual doses may be lower, metronomic therapy retains the goal of MTD treatment to kill as many cancer cells as possible by maximizing the total drug dose and reducing tumor blood flow.

Intermittent therapy has been used in ADT treatment (12). However, the protocol design typically requires a 7- or 8-month “induction period” in which ADT is administered continuously at MTD. This approach, primarily because of the induction period, does not take into consideration the predicted Darwinian dynamics of tumor and evolution-based mathematical models (13), which show that intermittent therapy produces the same outcomes as ADT—a result that was, in fact, observed in the clinical trials (12, 14).

Despite this central role of evolution as a proximate cause of death in patients treated for disseminated cancer, a recent study found that evolutionary principles are rarely considered in current cancer treatment trials. However, recently, a number of theoretical (13, 15), preclinical (8, 16), and clinical studies (17) have now suggested that explicit integration of evolutionary dynamics in cancer treatment protocols can significantly improve tumor control with increased time to progression (TTP) often while reducing the cumulative drug dose (17).

This adaptive approach means that each patient's treatment is truly personalized based on the tumor's state and response rather than a one-size-fits-all fixed treatment regime. Personalized treatment of prostate cancer has benefited from a plethora of computational modeling approaches, including intermittent androgen deprivation (18), introducing evolutionary cycles of treatment (19), optimizing treatment using control theoretic approaches (20), searching for evolutionary double-binds (2), or evolutionarily stable control (19, 21, 22). However, it is not yet clear how to extend these evolutionarily enlightened therapeutic concepts to multiple treatments (19).

For simplicity, all of these prior investigations have focused on monotherapies, but many (perhaps most) current cancer protocols administer two or more drugs. Integrating evolutionary dynamics in multidrug therapy is quite challenging. In addition to the overlapping pharmacodynamics and mechanisms of antitumor activity and toxicity, the molecular mechanisms of resistance and the evolutionary dynamics governing proliferation of resistant phenotypes are complex. Furthermore, metastatic cancer therapy can include chemotherapies, hormone therapy, angiogenesis inhibitors, and immunotherapy. Often, there are multiple available drugs within each treatment category [e.g., there are currently 52 (ref. 23) drugs approved for treatment of mPC]. Thus, potential combinations of cancer treatment types, specific drugs, and dosing schedules are enormous, such that defining optimal strategies is a daunting task. On the other hand, multidrug therapy offers an opportunity to further improve outcomes by harnessing these complex evolutionary dynamics.

Here, we have elected to focus on our prior experience in modeling an ongoing evolution-based clinical trial using abiraterone to treat men with metastatic castrate-resistant prostate cancer (mCRPC). Briefly, there are two general mechanisms that allow prostate cancer cells to become resistant to ADT. Some cancer cells upregulate CYP17A, an enzyme that allows the cell to produce testosterone (24). This autostimulatory loop allows these cancer cells to proliferate ADT. Alternatively, prostate cancer cells can become independent of androgen so that they can proliferate in the absence of testosterone (see Fig. 2 for a schematic of the interactions between cell types and treatments considered here). Abiraterone (25–27) blocks CYP17A and is commonly used for the initial treatment of mCRPC. About 60% of men with mCRPC respond to abiraterone, but evolution of resistance occurs fairly quickly with median TTP ranging from 7 to 15 months (28). In the ongoing trial (NCT02415621), abiraterone is administered only until the PSA declines to 50% of the pretreatment value. Withdrawal of abiraterone allows the CYP17A prostate cancer cells to proliferate but, in doing so, they inhibit proliferation of the androgen-independent cells (see Fig. 1B). Thus, the ratio of the two phenotypes remains similar at the end of the cycle so that restarting abiraterone produces a response virtually identical to the first cycle. By cycling abiraterone based on these underlying evolutionary dynamics, the TTP has been increased (compared with MTD abiraterone) by about 20 months while the cumulative drug dose has been reduced by half (17).

Figure 2.

Model schematic of cell types and treatments. The current adaptive therapy paradigm in mCRPC considers three cell types. Testosterone-producing cells (TP) support testosterone-dependent cells (T+), shown by green arrow. Both are targeted by abiraterone treatment (light blue), while testosterone-independent cells (T−) are resistant to abiraterone. Secondary docetaxel treatment targets each cell type in proportion to type-specific growth rate, ri. A fourth doubly resistant cell type, testosterone-independent cells that are also resistant to docetaxel (T−/−), emerges under combination therapy.

Figure 2.

Model schematic of cell types and treatments. The current adaptive therapy paradigm in mCRPC considers three cell types. Testosterone-producing cells (TP) support testosterone-dependent cells (T+), shown by green arrow. Both are targeted by abiraterone treatment (light blue), while testosterone-independent cells (T−) are resistant to abiraterone. Secondary docetaxel treatment targets each cell type in proportion to type-specific growth rate, ri. A fourth doubly resistant cell type, testosterone-independent cells that are also resistant to docetaxel (T−/−), emerges under combination therapy.

Close modal

Here, we begin our investigation of evolutionary dynamics in multidrug therapy by examining the addition of docetaxel to abiraterone for treatment of mCRPC (see Fig. 1C). Briefly, docetaxel is a taxane drug that is active in mCRPC. It binds preferentially and reversibly to the β-subunit of tubulin and disrupts microtubules during mitosis inducing cell-cycle arrest and apoptosis (29). In addition, docetaxel inhibits intracellular AR trafficking, generating some synergy with hormone therapy (30). SOC treatment for abiraterone monotherapy is a daily oral dose of 1,000 mg, and the protocol for docetaxel monotherapy is intravenous infusion 60 to 100 mg/m2 every 3 weeks (31). Prior studies have administered abiraterone and docetaxel simultaneously and sequentially but used a “one-size-fits-all” approach that did not use patient-specific evolutionary dynamics.

Background

Estimating the cost of resistance to cancer therapy.

Acquired drug resistance can develop due to mutation or through upregulation of molecular machinery (such as MDR protein family) encoded in the human genome (32, 33). Here our focus is less on the molecular events that lead to resistance than on the evolutionary cost of the resistance mechanism. Typically, an evolutionary cost is realized by a decreased growth rate, resulting as the consequence of acquiring resistance mechanisms. Expression, maintenance, and utilization of these pathways require resources that, in the limited-substrate tumor microenvironment, must be diverted from functions not necessary for survival including proliferation and invasion (34). These dynamics are perhaps most clear in upregulation of xenobiotic pathways such as P-glycoprotein (PgP), a membrane transporter that extrudes lipophilic cationic drugs (35). PgP hydrolyzes two ATP for every transported molecule. In experimental studies, synthesis, maintenance, and operation of PgP can consume up to 50% of the cell's energy budget (35).

The evolutionary cost of resistance in the CYP17A and androgen-independent phenotypes to be modeled in our clinical trial is (unlike PgP expression) not obvious. However, for clinical translation we estimate the cost of resistance using an “inverse problem” approach, based on the principle of “evolutionary triage” (36) in which fitter cells proliferate at the expense of those less fit. This allows a simple generality—the relative abundance of each tumor subpopulation will usually be a reasonable estimate of its fitness. Thus, because a reduction of PSA by at least 50% following abiraterone administration (17) is necessary to be included in our trial (NCT02415621), we can generally estimate that the CYP17A phenotype is the most prevalent and, therefore, in the absence of abiraterone, fitter than the androgen-independent phenotype. Although this approach is imperfect, we have found it provides a reliable estimate of intratumoral evolutionary dynamics in patients enrolled in the clinical trial (NCT02415621) and who will be evaluated in this study.

A primary–secondary (P-S) adaptive therapy.

Multidrug therapy in cancer is common, but most are administered more or less simultaneously with doses and timing largely dictated by prior monotherapy trials. Here, we investigate the addition of docetaxel in the context of a clinical trial in which abiraterone is administered depending on patient response and driven by an evolution-based mathematical model, to reduce the proliferation of the resistant androgen-independent phenotype. To address this specific clinical problem, we developed an evolution-based hypothesis that we termed P-S therapy. Briefly, we began with the recognition that the primary goal of the patient-specific variations in abiraterone administration within our trial is to suppress proliferation of treatment-resistant prostate cancer cells. Thus, any additional therapy should focus on reducing the resistant population. Specifically targeting a mechanism of resistance is appealing, but there appears to be myriad molecular pathways that result in androgen-independent survival and proliferation. Thus, using chemotherapy such as docetaxel that broadly targets proliferating cells and has been shown to be active in prostate cancer was selected as “secondary” drug. However, in this context, docetaxel should not be administered to indiscriminately kill all proliferating cancer cells because that would eliminate both abiraterone-resistant and -sensitive phenotypes. Thus, we reasoned that optimal integration of docetaxel into abiraterone therapy requires it to be administered when only resistant cells are proliferating—that is, during the administration of abiraterone. This then constituted our P-S strategy—abiraterone was designated the primary therapy because of its high efficacy and low toxicity while docetaxel is designated the secondary therapy with explicit role of eliminating cancer cells resistant to abiraterone. The focus of our modeling efforts, therefore, is quantitatively investigating these hypothesized evolutionary dynamics and designing a feasible clinical protocol that can translate these principles.

Evolving subpopulations in hormone therapy of mPC.

In prior investigations (13, 17, 37), we characterized the mPC subpopulations based entirely on their interactions with testosterone. We use as a point of departure the Lotka–Volterra mathematical model developed in ref. 15, described below. Here, we retain that basic model of three competing subpopulations, but integrate sensitivity to docetaxel: (i) T+ (androgen dependent), (ii) TP (androgen producing), and (iii) T−/+ (androgen independent). In adding docetaxel as our secondary drug, we add a fourth competing phenotype: (iv) T−/− (androgen independent and resistant to docetaxel). To summarize, the T+ and TP cells are targeted by our current clinical abiraterone trial (ADT is routinely included when abiraterone is administered). The T− cells are resistant to the trial drugs, and reducing their proliferation to maximize tumor control and time to progression is the primary goal of the abiraterone clinical trial. Here, we investigate the possibility of further reducing proliferation of the resistant (T−) cell using docetaxel. However, because subpopulations of the T− cells can become resistant (to docetaxel), we divide this population into T−/+, which is sensitive to docetaxel, and T−/−, which is resistant.

The mathematical model

Competitive interactions between these three phenotypes (T+, TP, and T−) were previously described by a Lotka–Volterra competition model (8). As a point of departure, the mathematical model developed in ref.15 was extended to include resistance to docetaxel. For simplicity, resistance to docetaxel is considered only in the T− population and neglected in both T+ and TP. Docetaxel resistance is expected to confer some phenotypic cost, allowing us to neglect the docetaxel-resistant T+ and TP populations during docetaxel treatment only. During combination of abiraterone and docetaxel, both T+ and TP populations are decaying, leading us to consider a four-population model (T+, TP, T−, and the doubly resistant T−/−) as follows:

formula

where xi is the population size of each phenotype, ri is the intrinsic growth rate of each phenotype, aij is the competition coefficient of phenotype i competing with phenotype j, and Ki is the carrying capacity. This model describes a wide range of patient-specific dynamics based on the competition coefficients (aij) and the initial conditions of the abiraterone-resistant T− population (see ref. 8).

The Lotka–Volterra mathematical model of competition has three essential features: intrinsic subpopulation growth rates, competition through interactions, and subpopulation carrying capacities. We now reiterate the assumptions made in ref. 15, and subsequently describe the departure from previous modeling approaches. Intrinsic growth rates, ri, were parameterized using measured doubling times of corresponding cell lines: ATCC@CRL-1740 LNCaP cell line, ATCC@CRL- 2128 H295R cell line, and ATCCCRL-1435 PC-3 cell lines for T+, TP, and T− cells, respectively. In vivo, these growth rates are likely to be unrealistic, but represent upper bounds on growth (for simulations herein, we scaled these growth rates by 10%). The carrying capacity of each subpopulation, Ki, represents the maximum population size that is sustainable in a given environment (e.g., with or without drug). A single TP cell can support the growth of 1.5 T+ cells and TP and T− cells have carrying capacities of 104. T−/− cells are assumed to have a decreased carrying capacity (K4 = 6 × 103). During abiraterone treatment, the carrying capacity of TP cells drops to 102, where TP cells support only 0.5 T+ cells. Carrying capacities are effectively modulated by competition between the three subpopulations, described by the competition matrix values, aij. The competition matrix below is built from a set of inequalities governed by two assumptions: (i) T+ cells without testosterone are the least competitive cell type and (ii) T− is more competitive toward TP than toward T+ cells. The inequalities reported in ref. 15 are as follows: a31 > a21, a32 > a12, a13 > a23, a13 > a12, a23 > a21, and a32 > a31 (17).

 T+ TP T− T−/− 
T+ a12 a13 a14 
TP a21 a23 a24 
T− a31 a32 a34 
T−/− a41 a42 a43 
 T+ TP T− T−/− 
T+ a12 a13 a14 
TP a21 a23 a24 
T− a31 a32 a34 
T−/− a41 a42 a43 

To account for the phenotypic cost of resistance to docetaxel, we calculate the values for T−/− (fourth row) as a function of the competition values of T− (third row) as follows: a41 = a31 − δ, a42 = a32 − δ, a43 = a33 − δ, a44 = 1, a14 = a13 − 2δ, a24 = a23 − 2δ, a34 = a33 − 2δ, where δ is the cost of resistance to docetaxel.

Following standard mathematical models of chemotherapy (7, 38), the cytotoxic docetaxel targets proliferating cell types in proportion to the growth rate (given in equation (1)):

formula

The docetaxel dose parameter, ci, is modulated to kill only proliferating cell types (positive growth) such that ci = 0 when |$( {1 - \ {\frac{{\sum {a_{ij}}{x_j}}}{{{K_i}}}}} ) \ \lt, 0$|⁠. The docetaxel dose parameter is greater than 1 during therapy (c1 = c2 = c3 = 1.5, held constant unless otherwise noted) and 0 during no treatment. T−/− cells are assumed to be fully resistant to docetaxel (c4 = 0). A small death rate, d, is also introduced (d = 5 × 10−4).

Under the assumption that each cell, regardless of its type, produces one unit of PSA per unit time, and that 50% of the PSA decays out of the serum in each time step, the PSA is given by

formula

This simulated serum PSA dynamics provides a direct correlation and is used to inform on/off treatment in the clinic. Below, we have used the nonlinear constrained optimization toolbox “lsqcurvefit” from Matlab to find the best parameter fit of equation (2) to sample patient data. By the theorem by Sontag and colleagues (39), 2n + 1 observed data points are needed to properly determine n model parameters. As such, most model parameters were kept constant and are identical to Figs. 3 and 4. Growth rates were taken as 10% of in vitro values (described above, |$\vec{r} = [ {0.00278,0.00355,0.00665,0.00665} ]$|⁠). Competition parameters are likewise held constant between patient simulations (a12 = 0.7; a13 = 0.8; a21 = 0.4; a23 = 0.5; a31 = 0.6; a32 = 0.9; |$\delta = 0.05$|⁠). For more discussion on the robustness and sensitivity of chosen competition parameters, we direct the curious reader to ref. 30. The driving assumption is that initial conditions (⁠|${\vec{x}_0}$|⁠) are subject to the greatest variance between patients. These were found by the least-squares optimization toolbox, using identical treatment times from patient clinical trial data (time in days).

Figure 3.

Simulation results. Six example therapy simulations of a representative virtual patient with identical initial parameters (⁠|$\vec{r}$| = [0.00278, 0.00355, 0.00665, 0.00665], |$\vec{x}_0^{}$| = [2062, 2617, 3.97 × 10−11, 3.97 × 10−12], a12 = 0.7; a13 = 0.8; a21 = 0.4; a23 = 0.5; a31 = 0.6; a32 = 0.9; δ = 0.05; c1 = c2 = c3 = 1.5; c4 = 0). The green dotted line shows the initial population size. A, No treatment leads to unhindered tumor growth to carrying capacity consisting primarily of TP cells. B, MTD of abiraterone results in competitive release of the T− resistant population. C, Maximum tolerated dose of docetaxel administered after abiraterone results in a good short-term response, with a relapse of the doubly resistant T−/− population. D, Metronomic abiraterone (LDM) also results in competitive release of T−, although delayed compared with MTD. E, An adaptive dose schedule of abiraterone (primary) is a personalized approach responsive to the tumor, which delays the onset of resistance, which may be extended by well-timed combination of docetaxel (secondary), shown in F.

Figure 3.

Simulation results. Six example therapy simulations of a representative virtual patient with identical initial parameters (⁠|$\vec{r}$| = [0.00278, 0.00355, 0.00665, 0.00665], |$\vec{x}_0^{}$| = [2062, 2617, 3.97 × 10−11, 3.97 × 10−12], a12 = 0.7; a13 = 0.8; a21 = 0.4; a23 = 0.5; a31 = 0.6; a32 = 0.9; δ = 0.05; c1 = c2 = c3 = 1.5; c4 = 0). The green dotted line shows the initial population size. A, No treatment leads to unhindered tumor growth to carrying capacity consisting primarily of TP cells. B, MTD of abiraterone results in competitive release of the T− resistant population. C, Maximum tolerated dose of docetaxel administered after abiraterone results in a good short-term response, with a relapse of the doubly resistant T−/− population. D, Metronomic abiraterone (LDM) also results in competitive release of T−, although delayed compared with MTD. E, An adaptive dose schedule of abiraterone (primary) is a personalized approach responsive to the tumor, which delays the onset of resistance, which may be extended by well-timed combination of docetaxel (secondary), shown in F.

Close modal
Figure 4.

Optimal timing of secondary docetaxel dose. A, An adaptive dose of abiraterone administered for two cycles (light blue bars) to investigate optimal timing of docetaxel (initial cell population counts are taken from cycle 11 of figure 3F). Docetaxel (red bar) may be administered at the exact time as abiraterone (Δt = 0), slightly before (Δt < 0) or slightly after (Δt > 0). The first cycle (abiraterone only) shows a decaying T+ and TP population with rising T− and T−/− resistant populations. The second cycle includes the addition of delayed (Δt = 100) docetaxel which targets the proliferating T− population. B, Varying the docetaxel timing shows that the minimum resistant T− population (yellow dashed line) is slightly after the start of abiraterone (Δt = 100). This same optimal docetaxel timing results in the minimization of T−/− cell population too (purple dashed line). Because abiraterone eliminates the T+ and TP populations, docetaxel is most effective at the end of the abiraterone cycle, where the resistant T− population is proliferating rapidly in the absence of competition.

Figure 4.

Optimal timing of secondary docetaxel dose. A, An adaptive dose of abiraterone administered for two cycles (light blue bars) to investigate optimal timing of docetaxel (initial cell population counts are taken from cycle 11 of figure 3F). Docetaxel (red bar) may be administered at the exact time as abiraterone (Δt = 0), slightly before (Δt < 0) or slightly after (Δt > 0). The first cycle (abiraterone only) shows a decaying T+ and TP population with rising T− and T−/− resistant populations. The second cycle includes the addition of delayed (Δt = 100) docetaxel which targets the proliferating T− population. B, Varying the docetaxel timing shows that the minimum resistant T− population (yellow dashed line) is slightly after the start of abiraterone (Δt = 100). This same optimal docetaxel timing results in the minimization of T−/− cell population too (purple dashed line). Because abiraterone eliminates the T+ and TP populations, docetaxel is most effective at the end of the abiraterone cycle, where the resistant T− population is proliferating rapidly in the absence of competition.

Close modal

Major assumptions of the mathematical model

  • CYP17A phenotype (TP) is fitter than androgen independent (T−) during no treatment

  • Abiraterone-sensitive (T+ and TP) cells suppress the growth of abiraterone-resistant (T−) cells during no treatment

  • Docetaxel targets proliferating cells, in proportion to growth rate

In the next section, we discuss the implications of this model on the design of P-S adaptive therapy with abiraterone and docetaxel.

Simulations of population density, xi, for a representative virtual patient are shown in Fig. 3. The simulations assume that the starting point is development of mCRPC. Therefore, we assume that prior ADT eliminated T+ population but that some T+ cells may survive by using local testosterone produced by the TP cells as a “common good.” Left untreated, competition between all four phenotypes results in abiraterone-sensitive T+ (blue) and TP (red) populations outcompeting their resistant counterparts due to the cost of developing resistance (Fig. 3A). The untreated population eventually reaches a carrying capacity with a roughly equal mix of “freeloading” T+ cells utilizing the testosterone-producing TP cells to provide exogenous testosterone.

An MTD of abiraterone (see Fig. 3B) targets these two treatment-sensitive populations, giving rise to a relapse of T− cells (yellow). At the point of relapse, a secondary therapy of docetaxel may be given (Fig. 3C). This results in a short-term response, but resistance still occurs in the doubly resistant T−/− population. Alternative fixed treatment schedules may be considered, such as a low-dose metronomic schedule of abiraterone, which gives a similar time to the onset of resistance (Fig. 3D). Drug doses (binary of on treatment or off) are shown by solid bars of light blue (abiraterone) and dark red (docetaxel) at the top of each subfigure.

Adaptive therapy does not rely on an a priori fixed time schedule, but rather adjusts the timing of drug administration depending on the patient-specific changes in PSA in response to treatment. Adaptive abiraterone predicts prolonged tumor control (Fig. 3E), albeit with eventual resistance as the T− population continues to grow slowly. This adaptive approach shows further improvement when using the proposed P-S approach: adaptively administering abiraterone according to the given protocol (administer until a threshold of 50% of initial PSA is reached and restart therapy on return to initial PSA value), combined with a well-timed administration of docetaxel during each dose of abiraterone. Shown in Fig. 3F is the patient's improved response with adaptive abiraterone in combination with secondary docetaxel at the beginning of each adaptive cycle. The emergent resistant population on relapse is now the T−/− (Fig. 3F, dashed purple) population. In the next section, we investigate the timing of docetaxel treatment on delaying the onset of resistance.

Timing of docetaxel on resistance

It is clear that the addition of docetaxel second-line therapy extends the relapse time and alters the evolutionary trajectory of the tumor. The hypothesis is that the primary therapy (abiraterone) is best utilized for tumor burden control by altering the competition between TP and T+ populations. The purpose of the secondary therapy (docetaxel) is to target the cancer cells that are resistant to the primary treatment agent. Here, the goal is to suppress proliferation of the abiraterone-resistant T− population.

Docetaxel will be most effective at targeting proliferating resistant T− cells while their growth rate is the highest during abiraterone treatment. At this time, most of the treatment-sensitive T+ and TP population will be decaying, so the remaining proliferating fraction is likely to be treatment-resistant T−. In this way, docetaxel's cytotoxic cell kill targeting proliferating cells can selectively target proliferating resistant T− cells during abiraterone treatment, while competition from T+ and TP cells suppresses resistant T− cells during abiraterone treatment holidays.

This hypothesis is confirmed by varying the administration of docetaxel relative to the start of abiraterone for a single cycle of adaptive therapy (Fig. 4). Figure 4A shows the optimal scenario: treating with the secondary therapy a short time after the start of abiraterone, but before the end of the abiraterone washout period. The full range of docetaxel start times are shown in Fig. 4B. Administering docetaxel before abiraterone (Δt < 0) or well after (Δt >> 0) results in a highly resistant T− fraction (dashed yellow). The optimal time difference (Δt = 100) pushes the second-line therapy to end of the effective treatment period of abiraterone (blue bar, Fig. 4A, top). This placement has a similar effect on the T−/− population (purple dashed).

Retrospective personalized adaptive P-S therapy

At the time of writing, 4 patients (of 18 patients accrued) on the adaptive therapy trial (NCT02415621) have progressed by radiographic and PSA criteria. Following the trial protocol, none of these patients received docetaxel (ci = 0); therefore, by using the patient-specific PSA data of these relapsed patients (Fig. 5, black dots), we can use the model (40) to recapitulate these data and capture patient-specific model parameters. Subsequent prediction of the effectiveness of our P-S adaptive approach can now be tested for each patient in a retrospective manner.

Figure 5.

Days gained via P-S therapy. Data from two of the four patients with PSA and radiographic progression. Model fit (dashed red line) shown for identical parameters as Fig. 3 except for initial conditions (⁠|${\vec{x}_0}$|⁠). The model accurately recapitulates time to relapse due to resistant T− population. Parameters from patient-specific model fitting are used to test the efficacy of the addition of secondary docetaxel treatment at the optimal time of each abiraterone cycle (blue dashed line; Δt = 30). We use a “days gained” metric to note the extension in days the model predicts a 3-fold (PSA = 3) increase in PSA from the initial value for abiraterone–docetaxel adaptive therapy compared with adaptive abiraterone only. A, Patient 1010 relapses after two cycles of abiraterone. Best model fit parameters: x1(0) = 1.4 × 102; x2(0) = 6.9 × 102; x3(0) = 9.2 × 10−3. Docetaxel as a secondary therapy predicts an average of 48 days gained (blue dashed line). B, Patient 1006 relapses after eight cycles of abiraterone. Best model fit parameters: x1(0) = 8.7 × 102; x2(0) = 1.6 × 10−5; x3(0) = 4.0 × 10−25. Docetaxel as a secondary therapy predicts an average of 132 days gained (blue dashed line). This “days gained” metric depends on the initial condition of T−/−, but the model predicts a positive value for days gained for a range of T−/− values. Purple-shaded region shows P-S deviation for a range of 5 orders of magnitude of the initial T−/− population such that x4(0) ∈ [x3(0)/105, x3(0)].

Figure 5.

Days gained via P-S therapy. Data from two of the four patients with PSA and radiographic progression. Model fit (dashed red line) shown for identical parameters as Fig. 3 except for initial conditions (⁠|${\vec{x}_0}$|⁠). The model accurately recapitulates time to relapse due to resistant T− population. Parameters from patient-specific model fitting are used to test the efficacy of the addition of secondary docetaxel treatment at the optimal time of each abiraterone cycle (blue dashed line; Δt = 30). We use a “days gained” metric to note the extension in days the model predicts a 3-fold (PSA = 3) increase in PSA from the initial value for abiraterone–docetaxel adaptive therapy compared with adaptive abiraterone only. A, Patient 1010 relapses after two cycles of abiraterone. Best model fit parameters: x1(0) = 1.4 × 102; x2(0) = 6.9 × 102; x3(0) = 9.2 × 10−3. Docetaxel as a secondary therapy predicts an average of 48 days gained (blue dashed line). B, Patient 1006 relapses after eight cycles of abiraterone. Best model fit parameters: x1(0) = 8.7 × 102; x2(0) = 1.6 × 10−5; x3(0) = 4.0 × 10−25. Docetaxel as a secondary therapy predicts an average of 132 days gained (blue dashed line). This “days gained” metric depends on the initial condition of T−/−, but the model predicts a positive value for days gained for a range of T−/− values. Purple-shaded region shows P-S deviation for a range of 5 orders of magnitude of the initial T−/− population such that x4(0) ∈ [x3(0)/105, x3(0)].

Close modal

Two representative patients are shown in Fig. 5: one patient who relapsed quickly after the second cycle (patient 1010 in Fig. 5A) and one patient who relapsed after 8 cycles (patient 1006, Fig. 5B). Model fitting of adaptive abiraterone (dashed red lines) recapitulates the PSA data (black dots) in each patient by varying only initial conditions (patient 1006: x1(0) = 8.7 × 102; x2(0) = 1.6 × 10−5; x3(0) = 4.0 × 10−25; patient 1010: x1(0) = 1.4 × 102; x2(0) = 6.9 × 102; x3(0) = 9.2 × 10−3). The best fit for patient 1010 resulted in a much higher initial population of T− cells, leading to a faster relapse.

Next, we test the efficacy of adding the secondary docetaxel treatment at the beginning of each abiraterone cycle. We use a “days gained” metric to note the extension in days; the model predicts a 3-fold (PSA = 3) increase in PSA from the initial value for abiraterone–docetaxel adaptive therapy compared with adaptive abiraterone only (Fig. 5). This value depends on the initial level of the T−/− population, but the model predicts a positive value for days gained for a range of T−/− values. The blue dashed lines show patient response when initial T−/− is equivalent to initial T− predicted by model fit. With a cost to resistance, it is unlikely that the initial T−/− population will be higher than T−, so the standard deviation in simulated response to a range of T−/− (up to 5 orders of magnitude smaller than T−) is shown in purple shading.

Recent successes in preclinical studies (8, 16, 17) and a clinical trial in evolution-based administration of abiraterone in patients with mCRPC have supported the hypothesis that explicit inclusion of evolutionary dynamics in clinical cancer therapy can prolong tumor control with existing drugs. For simplicity, these clinical trials have focused on alternating between treatment and treatment holidays for monotherapies or a single combination drug cocktail because of the challenges and complexity of administering multiple therapies adaptively or otherwise. Here, we investigate the possibility of enhancing these evolutionary dynamics through the addition of another drug.

Because variations in doses of drugs and time of administration within and between treatment cycles result in a vast range of therapeutic strategies, we began with a hypothesis-driven study we termed P-S adaptive therapy and specifically focused on adding a chemotherapy (docetaxel) to our ongoing monotherapy clinical trial administering abiraterone to men with mCRPC.

We framed the P-S hypothesis mathematically and model simulations predicted generally improved outcomes for primary adaptive abiraterone in combination with secondary docetaxel administered on a short time delay during each cycle. We then parameterized our model to fit the observed dynamics in 2 patients who had progressed on the clinical trial. Simulations of the combination therapy predicted that each patient would have benefited by the addition of docetaxel using the P-S strategy.

In general, this approach is consistent with prior studies emphasizing the value of continuous monitoring of evolving populations (41) and a proposal by Stankova and colleagues (42) that clinical trials should explicitly include a resistance management plan (RMP; refs. 43, 44) and should perform detailed after-action analyses (AAA; refs. 45, 46) on all patients who progressed to understand the mechanisms of treatment failure and identify strategies that would have improved outcomes. Here, our AAA in the 4 patients (in a cohort of 18) who have progressed in the abiraterone adaptive trial found the likely cause was a relatively large pretreatment fraction of resistant phenotypes. We then demonstrate an alternative RMP with inclusion of docetaxel that would have improved outcomes. This has now motivated ongoing preclinical studies to investigate the P-S strategy in vivo. If the preclinical studies are supportive, clinical translation will require specific pretreatment biopsies with application of IHC and image analytic techniques to quantify the pretreatment subpopulations.

Finally, we note the P-S strategy is simply one of many ways to combine two or more drugs to improve patient outcomes such that considerable theoretical and quantitative modeling will be necessary to establish optimal treatment methods.

J. Zhang reports receiving speakers bureau honoraria from Sanofi. No potential conflicts of interest were disclosed by the other authors.

Conception and design: J.B. West, M.N. Dinh, J.S. Brown, A.R. Anderson, R.A. Gatenby

Development of methodology: J.B. West, M.N. Dinh, J.S. Brown, A.R. Anderson, R.A. Gatenby

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): J. Zhang

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): J.B. West, M.N. Dinh, J.S. Brown, R.A. Gatenby

Writing, review, and/or revision of the manuscript: J.B. West, M.N. Dinh, J.S. Brown, J. Zhang, A.R. Anderson, R.A. Gatenby

Study supervision: A.R. Anderson

This work was supported by the following grants from the NIH NCI: U54CA143970-01, R01CA187532, RO1CA077575, and R01CA170595.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

1.
Gatenby
RA
. 
A change of strategy in the war on cancer
.
Nature
2009
;
459
:
508
9
.
2.
Gatenby
RA
,
Brown
J
,
Vincent
T
. 
Lessons from applied ecology: cancer control using an evolutionary double bind
.
Cancer Res
2009
;
69
:
7499
502
.
3.
Sweeney
CJ
,
Chen
YH
,
Carducci
M
,
Liu
G
,
Jarrard
DF
,
Eisenberger
M
, et al
Chemohormonal therapy in metastatic hormone-sensitive prostate cancer
.
N Engl J Med
2015
;
373
:
737
46
.
4.
Harshman
LC
,
Chen
YH
,
Liu
G
,
Carducci
MA
,
Jarrard
D
,
Dreicer
R
, et al
Seven-month prostate-specific antigen is prognostic in metastatic hormone-sensitive prostate cancer treated with androgen deprivation with or without docetaxel
.
J Clin Oncol
2018
;
36
:
376
82
.
5.
Aktipis
CA
,
Kwan
VS
,
Johnson
KA
,
Neuberg
SL
,
Maley
CC
. 
Overlooking evolution: a systematic analysis of cancer relapse and therapeutic resistance research
.
PLoS One
2011
;
6
:
e26100
.
6.
Brown
JS
. 
Why Darwin would have loved evolutionary game theory
.
Proc Biol Sci
2016
;
283
(1838). pii: 20160847.
7.
Norton
L
,
Simon
R
. 
The Norton-Simon hypothesis revisited
.
Cancer Treat Rep
1986
;
70
:
163
9
.
8.
Gatenby
RA
,
Silva
AS
,
Gillies
RJ
,
Frieden
BR
. 
Adaptive therapy
.
Cancer Res
2009
;
69
:
4894
903
.
9.
Benzekry
S
,
Pasquier
E
,
Barbolosi
D
,
Lacarelle
B
,
Barlesi
F
,
Andre
N
, et al
Metronomic reloaded: theoretical models bringing chemotherapy into the era of precision medicine
.
Semin Cancer Biol
2015
;
35
:
53
61
.
10.
Pramanik
R
,
Agarwala
S
,
Gupta
YK
,
Thulkar
S
,
Vishnubhatla
S
,
Batra
A
, et al
Metronomic chemotherapy vs best supportive care in progressive pediatric solid malignant tumors: a randomized clinical trial
.
JAMA Oncol
2017
;
3
:
1222
7
.
11.
Browder
T
,
Butterfield
CE
,
Kraling
BM
,
Shi
B
,
Marshall
B
,
O'Reilly
MS
, et al
Antiangiogenic scheduling of chemotherapy improves efficacy against experimental drug-resistant cancer
.
Cancer Res
2000
;
60
:
1878
86
.
12.
Hussain
M
,
Tangen
CM
,
Berry
DL
,
Higano
CS
,
Crawford
ED
,
Liu
G
, et al
Intermittent versus continuous androgen deprivation in prostate cancer
.
N Engl J Med
2013
;
368
:
1314
25
.
13.
Cunningham
JJ
,
Brown
JS
,
Gatenby
RA
,
Stankova
K
. 
Optimal control to develop therapeutic strategies for metastatic castrate resistant prostate cancer
.
J Theor Biol
2018
;
459
:
67
78
.
14.
Crook
JM
,
O'Callaghan
CJ
,
Duncan
G
,
Dearnaley
DP
,
Higano
CS
,
Horwitz
EM
, et al
Intermittent androgen suppression for rising PSA level after radiotherapy
.
N Engl J Med
2012
;
367
:
895
903
.
15.
Cunningham
JJ
,
Gatenby
RA
,
Brown
JS
. 
Evolutionary dynamics in cancer therapy
.
Mol Pharm
2011
;
8
:
2094
100
.
16.
Enriquez-Navas
PM
,
Kam
Y
,
Das
T
,
Hassan
S
,
Silva
A
,
Foroutan
P
, et al
Exploiting evolutionary principles to prolong tumor control in preclinical models of breast cancer
.
Sci Transl Med
2016
;
8
:
327ra24
.
17.
Zhang
J
,
Cunningham
JJ
,
Brown
JS
,
Gatenby
RA
. 
Integrating evolutionary dynamics into treatment of metastatic castrate-resistant prostate cancer
.
Nat Commun
2017
;
8
:
1816
.
18.
Morken
JD
,
Packer
A
,
Everett
RA
,
Nagy
JD
,
Kuang
Y
. 
Mechanisms of resistance to intermittent androgen deprivation in patients with prostate cancer identified by a novel computational method
.
Cancer Res
2014
;
74
:
3673
83
.
19.
West
J
,
You
L
,
Brown
J
,
Newton
PK
,
Anderson
AR
. 
Towards multi-drug adaptive therapy
.
bioRxiv
2018
;
p.476507
.
20.
Gluzman
M
,
Scott
JG
,
Vladirirsky
A
. 
Optimizing adaptive cancer therapy: dynamic programming and evolutionary game theory
.
arXiv
2018
;
01805
.
21.
Archetti
M
. 
Evolutionary game theory of growth factor production: implications for tumour heterogeneity and resistance to therapies
.
Br J Cancer
2013
;
109
:
1056
62
.
22.
Archetti
M
. 
Cooperation between cancer cells
.
Evol Med Public Health
2018
;
2018
:
1
.
23.
Drug.com [homepage on the Internet]
. 
Medications for prostate cancer;
2019
[cited 2019]. Available from
: www.drugs.com/condition/prostate-cancer.html.
24.
Kmetova Sivonova
M
,
Jurecekova
J
,
Tatarkova
Z
,
Kaplan
P
,
Lichardusova
L
,
Hatok
J
. 
The role of CYP17A1 in prostate cancer development: structure, function, mechanism of action, genetic variations and its inhibition
.
Gen Physiol Biophys
2017
;
36
:
487
99
.
25.
Fizazi
K
,
Tran
N
,
Fein
L
,
Matsubara
N
,
Rodriguez-Antolin
A
,
Alekseev
BY
, et al
Abiraterone plus prednisone in metastatic, castration-sensitive prostate cancer
.
N Engl J Med
2017
;
377
:
352
60
.
26.
Ryan
CJ
,
Smith
MR
,
de Bono
JS
,
Molina
A
,
Logothetis
CJ
,
de Souza
P
, et al
Abiraterone in metastatic prostate cancer without previous chemotherapy
.
N Engl J Med
2013
;
368
:
138
48
.
27.
Chi
KN
,
Protheroe
A
,
Rodriguez-Antolin
A
,
Facchini
G
,
Suttman
H
,
Matsubara
N
, et al
Patient-reported outcomes following abiraterone acetate plus prednisone added to androgen deprivation therapy in patients with newly diagnosed metastatic castration-naive prostate cancer (LATITUDE): an international, randomised phase 3 trial
.
Lancet Oncol
2018
;
19
:
194
206
.
28.
Mostaghel
EA
,
Marck
BT
,
Plymate
SR
,
Vessella
RL
,
Balk
S
,
Matsumoto
AM
, et al
Resistance to CYP17A1 inhibition with abiraterone in castration-resistant prostate cancer: induction of steroidogenesis and androgen receptor splice variants
.
Clin Cancer Res
2011
;
17
:
5913
25
.
29.
Ploussard
G
,
Terry
S
,
Maille
P
,
Allory
Y
,
Sirab
N
,
Kheuang
L
, et al
Class III beta-tubulin expression predicts prostate tumor aggressiveness and patient response to docetaxel-based chemotherapy
.
Cancer Res
2010
;
70
:
9253
64
.
30.
Darshan
MS
,
Loftus
MS
,
Thadani-Mulero
M
,
Levy
BP
,
Escuin
D
,
Zhou
XK
, et al
Taxane-induced blockade to nuclear accumulation of the androgen receptor predicts clinical responses in metastatic prostate cancer
.
Cancer Res
2011
;
71
:
6019
29
.
31.
de Bono
JS
,
Oudard
S
,
Ozguroglu
M
,
Hansen
S
,
Machiels
JP
,
Kocak
I
, et al
Prednisone plus cabazitaxel or mitoxantrone for metastatic castration-resistant prostate cancer progressing after docetaxel treatment: a randomised open-label trial
.
Lancet
2010
;
376
:
1147
54
.
32.
Longley
DB
,
Johnston
PG
. 
Molecular mechanisms of drug resistance
.
J Pathol
2005
;
205
:
275
92
.
33.
Holohan
C
,
Van Schaeybroeck
S
,
Longley
DB
,
Johnston
PG
. 
Cancer drug resistance: an evolving paradigm
.
Nat Rev Cancer
2013
;
13
:
714
26
.
34.
Wishart
GC
,
Plumb
JA
,
Going
JJ
,
McNicol
AM
,
McArdle
CS
,
Tsuruo
T
, et al
P-glycoprotein expression in primary breast cancer detected by immunocytochemistry with two monoclonal antibodies
.
Br J Cancer
1990
;
62
:
758
61
.
35.
Broxterman
HJ
,
Pinedo
HM
,
Kuiper
CM
,
Kaptein
LC
,
Schuurhuis
GJ
,
Lankelma
J
. 
Induction by verapamil of a rapid increase in ATP consumption in multidrug-resistant tumor cells
.
FASEB J
1988
;
2
:
2278
82
.
36.
Gatenby
RA
,
Cunningham
JJ
,
Brown
JS
. 
Evolutionary triage governs fitness in driver and passenger mutations and suggests targeting never mutations
.
Nat Commun
2014
;
5
:
5499
.
37.
You
L
,
Brown
JS
,
Thuijsman
F
,
Cunningham
JJ
,
Gatenby
RA
,
Zhang
J
, et al
Spatial vs. non-spatial eco-evolutionary dynamics in a tumor growth model
.
J Theor Biol
2017
;
435
:
78
97
.
38.
Simon
R
,
Norton
L
. 
The Norton-Simon hypothesis: designing more effective and less toxic chemotherapeutic regimens
.
Nat Clin Pract Oncol
2006
;
3
:
406
7
.
39.
Sontag
Eduardo D
. 
For differential equations with r parameters, 2r+ 1 experiments are enough for identification
.
Journal of Nonlinear Science 12.6
( 
2002
):
553
83
.
40.
Wilkerson
J
,
Abdallah
K
,
Hugh-Jones
C
,
Curt
G
,
Rothenberg
M
,
Simantov
R
, et al
Estimation of tumour regression and growth rates during treatment in patients with advanced prostate cancer: a retrospective analysis
.
Lancet Oncol
2017
;
18
:
143
54
.
41.
Fischer
A
,
Vazquez-Garcia
I
,
Mustonen
V
. 
The value of monitoring to control evolving populations
.
Proc Natl Acad Sci U S A
2015
;
112
:
1007
12
.
42.
Stankova
K
,
Brown
JS
,
Dalton
WS
,
Gatenby
RA
. 
Optimizing cancer treatment using game theory: a review
.
JAMA Oncol
2018
.
doi: 10.1001/jamaoncol.2018.3395. [Epub ahead of print]
.
43.
Hemingway
J
,
Vontas
J
,
Poupardin
R
,
Raman
J
,
Lines
J
,
Schwabe
C
, et al
Country-level operational implementation of the Global Plan for Insecticide Resistance Management
.
Proc Natl Acad Sci U S A
2013
;
110
:
9397
402
.
44.
Mnzava
AP
,
Knox
TB
,
Temu
EA
,
Trett
A
,
Fornadel
C
,
Hemingway
J
, et al
Implementation of the global plan for insecticide resistance management in malaria vectors: progress, challenges and the way forward
.
Malar J
2015
;
14
:
173
.
45.
Ross
MC
,
Smith
KK
,
Smith
A
,
Ryan
R
,
Webb
L
,
Humphreys
S
. 
Analysis of after-action reporting by deployed nurses
.
Mil Med
2008
;
173
:
210
6
.
46.
Singleton
CM
,
Debastiani
S
,
Rose
D
,
Kahn
EB
. 
An analysis of root cause identification and continuous quality improvement in public health H1N1 after-action reports
.
J Public Health Manag Pract
2014
;
20
:
197
204
.

Supplementary data