Evolutionary Dynamics of Mutator Phenotypes in Cancer: Implications for Chemotherapy¹ Free

The genetic material of cells is subject to stress and damage as a result of environmental agents or the production of oxidative radicals during metabolism. However, eukaryotic cells tend to maintain stable genomes. Maintenance of this stability requires the presence of so-called checkpoint genes (1, 2). When the DNA or chromosome structure becomes damaged, the checkpoints delay the cell cycle and correct the damage (1, 3, 4). On the other hand, many cancers are characterized by a phenomenon called genetic instability (5). This can be defined as an elevated rate at which cell genomes acquire changes. It is related to the corruption of checkpoint genes that are responsible for detecting and repairing damage (6, 7, 8, 9). Different types of instability can be distinguished (5, 10). Some result in the accumulation of relatively small sequence changes, brought about by deletions and insertions of a few nucleotides or base substitutions. Other types of instabilities involve larger scale genomic changes. This can involve the loss or gain of whole chromosomes or parts of chromosomes, chromosome translocations, and gene amplifications. We will refer to cells with genetic instability as mutator phenotype (11, 12, 13, 14, 15).

Because in many cases, the presence or absence of genetic instability is a major difference between cancerous and healthy tissue (5), an understanding of cancer progression, as well as insights into therapeutic approaches, requires an understanding of the evolutionary dynamics of mutator phenotypes and stable cell populations. It is crucial to understand the conditions under which selection favors mutator phenotypes, and when stable cells prevail (16). Both genetic stability and instability can confer advantages and disadvantages for the cells (17, 18). Faithful replication ensures that the phenotype of the cell is preserved and that deleterious alterations are avoided; at the same time, induction of the checkpoints can result in cell cycle arrest, which is a cost. Absence of checkpoints avoids cell cycle arrest in the face of damage but can result in deleterious mutants that reduce the overall fitness of the cell. Here, we use mathematical models to study the dynamic interactions between stable and unstable cell populations. In particular, we seek to understand how the amount of DNA damage experienced by tissue influences the relative fitness of mutator phenotypes. This has significance for understanding the processes underlying cancer initiation, cancer progression, and chemotherapeutic and radiation approaches to treatment.

In this section, we explore the competition dynamics between a stable and a mutator cell population. They differ in the probability with which they repair genetic damage. Stable cells repair damage with a probability ε_s, and mutator cells repair damage with a probability ε_m, where ε_s > ε_m. We further assume that these cell populations differ in their intrinsic rate of replication. The stable cells replicate at a rate r_s, and the mutator cells replicate at a rate r_m. Let us denote the abundance of stable and mutator cells as S and M, respectively. The competition dynamics are given by the following pair of differential equations that describe the development of the cell populations over time.

The model is explained graphically in Fig. 1. The cells replicate at a rate r_s or r_m. These parameters reflect how often cells reproduce and die; we will call this the intrinsic replication rate of the cells. The two cell populations compete for a shared resource. Competition is captured in the expressions φS and φM, which is specified in the “Appendix.” During replication, a genetic alteration can occur with a probability u. We call this the DNA hit rate. DNA damage can occur spontaneously (most likely at low levels), or it can be induced by DNA-damaging agents, which corresponds to a high value of u. If a genetic alteration has occurred, it gets repaired with a probability ε_s or ε_m. During repair, there is cell cycle arrest, and this is captured in the parameter β. The value of β can lie between 0 and 1 and thus reduces the rate of cell division (given by βr). If β = 0, the repairing cells never replicate, and this is the maximal cost. If β = 1, there is no cell cycle arrest and no cost associated with repair. With a probability (1 − ε_s or 1 − ε_m) the genetic alteration does not get repaired. If the alteration is not repaired, a mutant is generated. A mutation is therefore the result of the occurrence of DNA damage combined with the absence of repair. The mutant is viable (and neutral) with a probability α, whereas it is nonviable with a probability 1 − α. Therefore, the model captures both the costs and benefits of repair: efficient repair avoids deleterious mutations but is associated with cell cycle arrest. Absence of efficient repair can result in the generation of deleterious mutants but avoids cell cycle arrest.

Note that in this first model, we assume that the mutants that are created are either nonviable (and thus do not participate in the competition dynamics) or neutral (and thus have the same intrinsic reproductive rate as the wild type). We will include the possibility of advantageous and disadvantageous (but viable) mutants later.

In the following, we investigate how the competition dynamics depend on the rate at which cells acquire genetic alterations (DNA hit rate, u). In general, if two cell populations compete, the cells with the higher fitness win. The fitness of the cells is given by r_s,m[1 − u[1 − α + ε_s,m(α − β)]]. Note that the quantity 1 − α has the meaning of the cost of production of deleterious mutants; we will refer to it as C_del = 1 − α. Similarly, the quantity 1 − β is the cost of cell cycle arrest, C_arr = 1 − β. In these notations, we can rewrite the expression for the fitness in a more intuitive way, as shown below.

If the DNA hit rate is low (low value of u), the fitness of the cells is approximately given by their intrinsic rate of replication (r_s and r_m). Thus, the cell population with the higher intrinsic replication rate has a higher fitness than the cell population with the lower intrinsic replication rate. On the other hand, when the DNA hit rate, u, is increased, the fitness depends more strongly on other parameters. In particular, the fitness of both populations can depend on the DNA hit rate, u. Notably, an increase in the value of u may result in a stronger decline in fitness of the cell population with the faster intrinsic rate of replication relative to the slower cell population (Fig. 2). Therefore, if the DNA hit rate crosses a critical threshold, u > u_c, the outcome of competition can be reversed. We are interested in discovering under what circumstances this can occur (for a complete mathematical definition, see Supplemental Data). One condition required for the reversal of competition is that the stable and mutator cells are characterized by a sufficient difference in the repair rate (ε_m − ε_s; Fig. 2). Furthermore, we need to distinguish between two scenarios.

(a) In the first case, we assume that the stable cells have a faster intrinsic rate of replication than the mutator cells (i.e., r_s > r_m). Therefore, at low DNA hit rates, the stable cells win. An increased DNA hit rate, u, can shift the competition dynamics in favor of the unstable cells. In other words, unstable cells gain a selective advantage as the DNA hit rate becomes large. This is because the population of stable cells frequently enters cell cycle arrest when repairing genetic damage, and this slows down the overall growth rate. For this outcome to be possible, the following condition has to be fulfilled (for mathematical details, see Supplemental Data): the cost of cell cycle arrest, C_arr, must be greater than the cost of producing nonviable mutants, C_del. If this condition is not fulfilled, reversal of competition at high DNA hit rates is not observed.

(b) In the second case, we assume that the stable cells have a slower intrinsic replication rate than the mutator cells (i.e., r_s < r_m). Therefore, at low DNA hit rates, the unstable cells win. An increased DNA hit rate, u, can shift the competition dynamics in favor of the stable cells. In other words, a high DNA hit rate selects against genetic instability. This is because the unstable cells produce more nonviable mutants, and this reduces the effective growth rate significantly. In contrast to the previous scenario, this requires that the cost of producing nonviable mutants, C_del, must be higher than the cost of cell cycle arrest, C_arr. If this condition is not fulfilled, reversal of competition at high DNA hit rates is not observed. For mathematical details, see Supplemental Data.

To summarize, this analysis gives rise to the following results (Table 1A). A high DNA hit rate, u, can reverse the outcome of competition in favor of the cell population characterized by a slower intrinsic growth rate if the competing populations are characterized by a sufficient difference in their repair rates. The higher the difference in the intrinsic replication rate of the two cell populations, the higher the difference in repair rates required to reverse the outcome of competition. If the intrinsic replication rate of the genetically unstable cell is slower, a high DNA hit rate can select in favor of genetic instability. On the other hand, if the intrinsic growth rate of the genetically unstable cell is faster, a high DNA hit rate can select against genetic instability.

In the previous section, we considered the competition dynamics between stable and unstable populations of cells, assuming that they are characterized by different and fixed rates of replication. We further assumed that mutations are either nonviable or neutral. However, mutations are unlikely to be neutral and will change the replication rate of the cells. In other words, cells may evolve to grow either at a faster or a slower rate, depending on the mutations generated. Here, we extend the above model to take into account such evolutionary dynamics. The model is written out mathematically in the “Appendix” and explained schematically in Fig. 1 b. In the following, we outline the assumptions. We again consider two competing cell populations: a genetically stable population (S); and a mutator population (M). We start with unaltered cells that have not accumulated any mutations. They are denoted by S₀ and M₀, respectively. Both are assumed to replicate at the same rate, r₀. When the cells become damaged, and this damage is not repaired, mutants are generated. If the mutants are viable, they can continue to replicate. During these replication events, further mutations can be accumulated if genetic alterations are not repaired. We call the process of accumulation of mutations the mutational cascade. Cells that have accumulated i mutations are denoted by S_i and M_i, respectively, where i = 1,… , n. They are assumed to replicate at a rate r_i. Stable and unstable cells differ in the rate at which they proceed down the mutational cascade. In addition to the basic dynamics of cell replication described in the previous section, we assume that during cell division, mutated cells can undergo apoptosis because oncogenic mutations can induce apoptotic checkpoints (19, 20). Thus, the intrinsic replication rate of mutated cells is given by r_i(1 − a), where a denotes the probability to undergo apoptosis upon cell division.

To analyze this model, we have to assume a fitness landscape for the consecutive mutants (Fig. 3). Because we are interested in cancer progression, we assume that the intrinsic rate of cell division of the consecutive mutants, r_i, increases (r_i+1 > r_i). Such mutations could correspond to alterations in oncogenes or tumor suppressor genes. Because an accumulation of mutations cannot result in an infinite increase in the division rate of cells, we assume that the division rate plateaus. Once the cells have accumulated n mutations, we assume that further viable mutants are neutral because the division rate cannot be increased further (this end stage of the mutational cascade is thus mathematically identical to the simple model discussed in the last section). Whereas we assume that the consecutive mutants can divide faster, they can also carry a disadvantage: the mutations can be recognized by the appropriate checkpoints that induce apoptosis. With this in mind, we will consider two basic types of fitness landscapes. If r₀ > r_n(1 − a), the intrinsic growth rate of the mutated cells, S_i and M_i, will be less than that of the unaltered cells, S₀ and M₀ (Fig. 3). Whereas the mutations allow the cells to escape growth control, the mutated cells are killed at a fast rate by apoptosis upon cell division. This scenario corresponds to the presence of efficient apoptotic mechanisms in cells. On the other hand, if r₀ < r_n(1 − a), the accumulation of mutations will eventually result in an intrinsic growth rate that is larger than that of unaltered cells (Fig. 3). Whereas mutated cells can still undergo apoptosis upon cell division, apoptosis is not strong enough to prevent an increase in the intrinsic growth rate. Hence, this scenario corresponds to impaired apoptosis in cells. In the following sections, we study the competition dynamics between stable and mutator cells in an evolutionary setting, assuming the presence of relatively strong and weak apoptotic responses.

Here we assume that the apoptotic mechanisms in cells are strong. That is, r₀ > r_n(1 − a) (Fig. 3). This means that although the successive mutations will allow the cell to divide faster, the induction of apoptosis ensures that the intrinsic growth rate of the mutants is lower compared with that of unaltered cells. Note that it is not necessary to assume that oncogenic mutations allow cells to divide faster. Indeed, some cancer cells may progress more slowly through the cell cycle than healthy cells. The important assumption is that accumulation of mutations lowers the intrinsic growth rate of the cells. In this scenario, the intrinsic growth rate of the stable cells, S, is higher than that of the unstable cells, M. The reason is as follows. The population of stable cells, S, has efficient repair mechanisms. Thus, most cells will remain at the unaltered stage, S₀. Because population M is unstable, a higher fraction of this cell population will contain mutations. Because these mutations impair reproduction (e.g., because of induction of apoptosis), the intrinsic growth rate of the unstable cells, M, is lower than that of the stable population, S.

According to the above arguments, at low DNA hit rates, the cells with the faster intrinsic growth rate win the competition. Thus, at low DNA hit rates (low value of u), the stable phenotype, S, wins (Fig. 4). On the other hand, at higher DNA hit rates (high value of u), the outcome of competition can be reversed because frequent cell cycle arrest significantly reduces growth. That is, the genetically unstable cells, M, may win and take over the population (Fig. 4). As in the simple model discussed above, it requires that the cost of cell cycle arrest is higher than the cost associated with the generation of deleterious mutants (i.e., C_arr > C_del). Furthermore, reversal of competition may require that the repair rate of stable cells (ε_s) lies below a threshold and that there is a sufficient difference in the repair rate between stable and unstable cells. Mathematical conditions are given in the Supplemental Data. As the population of unstable cells wins, they accumulate mutations. Even if the sequential mutants are disadvantageous because of the induction of apoptosis, the high mutation rate pushes the population down the mutational cascade. While all variants, M_i, persist, the distribution of the variants becomes skewed toward M_n as the DNA hit rate is increased.

These results have important practical implications. The model tells us that in the presence of intact apoptotic mechanisms, a high DNA hit rate selects in favor of genetic instability, whereas the tissue remains stable and unaltered if the DNA hit rate is low. A high DNA hit rate can be brought about by both the presence of carcinogens and chemotherapy. Therefore, if healthy tissue is exposed to carcinogens, we expect genetic instability to rapidly emerge, and this can give rise to cancer progression. In the same way, chemotherapy can select for genetic instability in otherwise healthy tissue and thus induce new tumors as a side effect.

Now we assume that the apoptotic mechanisms in cells are impaired. That is, r₀ < r_N (1 − a) (Fig. 3). This means that accumulation of mutations will eventually result in the generation of variants that have a faster intrinsic growth rate compared with unaltered cells. Thus, in principle, both populations are expected to eventually evolve toward the accumulation of mutations and progression to cancer. Hence, both stable and unstable cancers can be observed. However, this process can occur over different time scales for the two populations of cells because they are assumed to differ in the repair rate and thus in the rate at which mutations can develop. Obviously, the mutator phenotype can accumulate mutations faster than the stable population of cells. If there is a sufficient difference in the repair rate of the two populations, the mutator phenotype will accumulate mutations and progress toward M_n during a period of time in which the population of stable cells will remain unaltered at stage S₀. In this case, genetic instability can drive cancer progression. A mathematical condition for this scenario is given in Supplemental Data.

If the stable and unstable populations compete, the unstable population will have a higher intrinsic growth rate than the stable population (because the induction of apoptosis in response to mutation is inefficient). Therefore, at low DNA hit rates, u, the mutator phenotype, M, wins the competition (Fig. 4). If the DNA hit rate is increased, the competition can be reversed in favor of the stable cell population, S (Fig. 4). This requires that the cost of generating deleterious mutants is greater than the cost of cell cycle arrest (i.e., C_del > C_arr). Furthermore, a sufficient difference in the repair rate of stable and unstable cells is required to reverse the outcome of competition. Exact mathematical conditions are given in Supplemental Data.

This again has practical implications. If cells develop a mutation resulting in impaired apoptotic responses, then genetic instability has a selective advantage if the DNA hit rate is low. Therefore, even if there is no exposure to carcinogens, a chance loss of apoptosis can result in the outgrowth of genetic instability and thus progression of cancer. On the other hand, if there is a growing cancer with impaired apoptotic responses, our results suggest that an elevation of the DNA hit rate by chemotherapeutic agents can reverse the relative fitness in favor of stable cells, and this can result in cancer reduction or slower progression.

A note of clarification: in the above arguments, we assumed for simplicity that apoptosis is inefficient in both the unstable and stable cells. The arguments about chemotherapy, however, remain robust, even if we assume that only the mutator phenotypes have impaired apoptosis, whereas the stable and healthy population of cells has intact apoptotic responses. The reason is that over the time frame considered, the population of stable cells remains genetically unaltered (i.e., at stage S₀). Because the cells are unaltered, the presence or absence of apoptosis does not change the dynamics.

This paper has examined the competition dynamics between genetically stable and unstable populations of cells. We used mathematical models to identify under which circumstances genetic instability is selected for or against in the context of cancer progression. In particular, we examined the role of the rate at which DNA is damaged.

We found that a change in the DNA hit rate can reverse the outcome of competition. In the simplest setting, an increase in the DNA hit rate can switch the outcome of competition in favor of cells characterized by a slower intrinsic growth rate. This requires a sufficient difference in the repair rate between the stable and mutator cells and a condition on the relative values of costs associated with cell cycle arrest and creation of deleterious mutants. The conditions under which genetic instability is selected for depend on the efficacy of apoptosis. In terms of cancer evolution and progression, this gave rise to the following insights (Table 1B).

(a) If apoptosis is strong, accumulation of mutations by unstable cells slows down the intrinsic growth rate because of the frequent induction of cell death. Thus, stable cells have a higher intrinsic growth rate than mutators. Consequently, at low DNA hit rates, the stable cells win. The presence of high DNA hit rates can, however, result in the selection and emergence of the genetically unstable cells. This occurs if the cost of cell cycle arrest upon repair is higher than the cost of creating deleterious mutations.

(b) On the other hand, if apoptotic responses are impaired, accumulation of mutations by unstable cells will not result in frequent cell death upon division. Therefore, the intrinsic growth rate of unstable cells can be higher than that of stable cells if adaptive mutations are acquired. In this case, genetic instability is expected to emerge at low DNA hit rates. At high DNA hit rates, however, genetic instability can be selected against, and mutators can go extinct. This occurs if the cost of creating deleterious mutations is higher than the cost of cell cycle arrest.

A fascinating question is how much genetic instability can contribute to faster adaptation and evolution of cancer cells (11, 12, 13, 14, 15, 21, 22, 23). It can be argued that genetic instability can be selected for due to the following two factors.

(a) Genetic instability can be advantageous if it results in a faster accumulation of adaptive mutations compared with stable cells (14). This could allow the cancer to evolve faster and to overcome selective barriers and host defenses. An example are tumor suppressor genes where both copies have to be mutated. Instead of the occurrence of two independent point mutations, loss of heterozygosity in genetically unstable cells can result in the loss of suppressor function if one copy has been mutated.

(b) Genetic instability can be advantageous simply because cells avoid delay in reproduction associated with repairing and maintaining the genome (17, 18). When genetic damage is detected, the relevant checkpoints induce cell cycle arrest, during which the damage is repaired. If genetic damage occurs often, frequent arrest significantly slows down the replication rate of the cells, and loss of repair can be advantageous. Experimental evidence supports this notion. Bardelli et al. (24) have shown that exposure to specific carcinogens can result in the loss of the checkpoint that was induced by the carcinogenic agent used.

At this stage, it is unclear what selective mechanism is responsible for the emergence of genetic instability (or in fact whether genetic instability appears simply as a side effect of other genetic alterations on the way to cancer). It is possible that different types of genetic instability can have different effects on the evolution of the cell populations. The increased rate at which the quasispecies travel up the fitness landscape may or may not be outweighed by the costs associated with creating deleterious mutations. This, in turn, may depend on the nature of the instability. In particular, it may be determined by whether the genetic changes are relatively small (such as in microsatellite instability) or larger (such as in chromosomal instability).

If the main driving force for the emergence of genetic instability is avoidance of cell cycle arrest (rather than faster adaptation), this could contribute to explaining why certain instabilities are specific to certain types of cancers or tissues. Different environments can cause different types of genetic alterations that induce separate checkpoints (24). The checkpoints that are lost in the cancer would be the ones that are most often induced in the appropriate environment and tissue surroundings. On the other hand, if genetic instability emerges mainly because it allows the cells to adapt faster, we expect that instability is lost at later stages of cancer progression. This is because the cancer has evolved to an optimal phenotype, and now stability avoids deleterious mutations and thus increases fitness (16).

If genetic instability can result in a faster accumulation of adaptive mutations [case (a) above], it could in principle be the driving force of cancer progression. As pointed out in the previous section, it is unclear whether this is the case, or whether alternative selection pressures are responsible for the emergence of genetic instability. Here, we assume that instability can result in the accumulation of adaptive mutations and explore possible pathways to the emergence of genetic instability and cancer progression. Assume we start from a wild-type cell that is stable and has intact apoptotic mechanisms. The mathematical model suggests that genetic instability can only drive progression toward fitter phenotypes if apoptosis is impaired. This is because in the presence of intact apoptosis, accumulation of mutations results in elevated levels of cell death that slow down the intrinsic growth rates. Thus, to gain a fitness advantage, both apoptosis and stability genes have to be mutated. This can occur via two pathways (Fig. 5).

(a) In the first pathway, the initial mutation impairs the apoptotic response in the cell. This variant is selectively neutral compared with the wild type. The reason is that the cell still has intact repair mechanisms. Therefore, mutations are unlikely to be generated in the time frame considered. As long as mutations do not accumulate, the presence or absence of apoptotic mechanisms does not change the dynamics of cell growth. Following this mutation, a second mutation is generated that confers genetic instability. This mutant will be selected for instantly.

(b) In the second pathway, the initial mutation confers genetic instability to the cell. Because apoptotic responses are still intact, the model analysis tells us that this variant will have a lower fitness compared with the wild type and will be on its way to extinction. However, because the cell is unstable, it can generate mutations at a faster rate. Thus, there is a chance that the mutation impairing apoptosis is generated before this cell variant has gone extinct. As soon as the apoptotic mechanism has been impaired, the unstable cell gains a selective advantage.

We can calculate which of these two pathways occurs faster, and this is the pathway that is more likely to lead to selection of instability (for mathematical details of this approach, see Ref. 25). We introduce the following notation (Fig. 5). The number of cells in a tissue is denoted by N. The rate at which a genetically stable cell mutates (to be either unstable or apoptosis impaired) is given by μ. The rate at which an unstable cell mutates toward a loss of apoptotic function is denoted by μ̄. Thus, μ̄ > μ. The relative reproductive rate of an unstable cell that has intact apoptotic responses is given by σ < 1 (we assume that the wild-type reproductive rate is 1), which reflects the fact that unstable cells with intact apoptosis have a selective disadvantage. The rate at which an advantageous mutator is generated via the first pathway (first apoptosis−, then repair−) is given by μ²N. The rate at which an advantageous mutator is generated via the second pathway (first repair−, then apoptosis−) is given by Nμσμ̄/(1 − σ). Therefore, if μ̄ > μ(1 − σ)/σ, then repair and stability are lost first. In the opposite case, apoptosis is lost first. In biological terms, if the relative fitness of the unstable and apoptosis-competent cell is sufficiently low (because mutants are killed efficiently), then generation of an advantageous mutator is likely to proceed by first losing apoptosis and then acquiring genetic instability. Knowledge of parameter values will be required to determine which of the two pathways is more likely. The result might vary between different tissues.

The results derived in this paper have implications for the use of chemotherapy (26, 27, 28). Chemotherapy essentially increases the degree of DNA damage. Therefore, it can be used to reverse the relative fitness of stable and unstable cells such that unstable cells are excluded (Fig. 6). This can drive progressing cancer cells extinct and result in the persistence of stable cells. These may be either healthy cells or less aggressive and slowly progressing tumor cells. In this scenario, because chemotherapy acts by modulating the competition between stable and unstable cells, it is not a requirement that every last cell is killed by the drugs. Selection and competition will make sure that the unstable cancer cells are driven extinct. This argument, however, requires that there is an element of competition between unstable and stable cells. Whether and under which circumstances this is the case remain to be determined.

This is a different mechanism of action compared with the traditional view, which assumes that chemotherapy only acts by killing dividing cells. For chemotherapy to reverse the fitness of stable and unstable cells, two conditions are required. (a) There needs to be a sufficient difference in the repair rate between stable and unstable cells. The higher the replication rate of unstable cells relative to stable cells, the higher this difference in the repair rate required to achieve success. Therefore, contrary to traditional views, a faster rate of cell division of cancerous unstable cells renders successful treatment more difficult in this scenario. (b) The cost of generating lethal mutants in unstable cells must be higher than the cost of cell cycle arrest in stable cells. If this is not the case, it does not pay to retain repair mechanisms, and the fitness of unstable cells can never be reversed. In this case, treatment has a higher negative impact on stable than on unstable cells, and the mutators are resistant.

This paper has provided a theoretical framework for studying the competition and evolutionary dynamics between genetically stable and unstable cells. We identified conditions under which selection will favor genetic instability and circumstances under which genetically unstable cells can be driven extinct. The results derived from the models can contribute to our understanding of the occurrence of genetic instability in many human cancers and have implications for the principles underlying cancer initiation and the mechanisms that may underlie chemotherapy.

To have greater practical implications, theoretical predictions need to be tested by experiments. First, model assumptions need to be validated. It would be important to measure the cost of cell cycle arrest and the cost of creating lethal mutations as a function of the level of DNA damage. In the model, these costs are assumed to be constant, and this may have to be revised. Then, in vitro competition experiments would have to be performed. A stable cell line would compete against a specific unstable cell line (e.g., deficient in mismatch repair) under different levels of DNA damage. Such experiments should be performed with cells that have intact apoptotic responses and cells with impaired apoptosis. An experimental approach, coupled with the mathematical analysis, would provide further insights into the dynamics of mutator phenotypes in cancer and the importance of instability in treatment.

The simplest system that captures the dynamics is given by Eqs. 1 and 2, with the average fitness of cells, φ, defined as follows: φ = Sr_s [1 − u(1 − βε_s − α(1 − ε_s))] + Mr_m[1 − u(1 − βε_m − α(1 − ε_m))].

The simple model assumes that mutations are either nonviable or neutral. A more detailed model that includes the disadvantageous and advantageous mutations is given by the following system of quasispecies equations:

where we introduced the following shorthand notations: R_i is the effective intrinsic reproductive rate, R_i = r_i(1 − α) for 1 ≤ i ≤ n and R₀ = r₀, and u_s,m are the two effective mutation rates, u_s,m = u(1 − βε_s,m). The variable w denotes the noviable mutants produced by the cells. The equations are coupled through the function φ, the average fitness, which is given by the following equation.