The purpose of this study is to develop a comprehensive mathematical model of DNA double-strand break (DSB) repair in a cell cycle-dependent manner to analyze the improved radiosensitivity of prostate cancer (PCa) cells to ionizing radiation (IR) when radiation treatment is combined with androgen deprivation therapy (ADT). The effectiveness of the combination treatment, which is currently a standard treatment for PCa, depend on DSB repair capacity of the cells as these are the main lesions following IR. It is reported in the literature that the major DSB repair pathways, non-homologous end joining (NHEJ) and homologous recombination (HR), are both impaired after ADT, which then resulted in increased radiosensitivity and better IR treatment outcomes in PCa. In our previous work, we have developed quantitative models for NHEJ and HR individually to analyze the mechanism of the effect of ADT on IR treatment outcomes through impaired repair dynamics. The present work combines these two models in a cell cycle-dependent manner in order to develop a comprehensive model to analyze the repair dynamics after IR only and IR+ADT treatments. NHEJ is the major pathway, whereas HR is restricted to S- or G2-phases of the cell cycle after DNA replication has been completed and we have incorporated the cell cycle dependent contributions of the NHEJ and HR models in our comprehensive model. The literature data used in the development of these previous models were from both in vitro experiments as well as clinical data from PCa patients. In our comprehensive model, we have used the data from the literature to determine the distribution of the initial DSBs for cells in different cell cycle phases as the damage depends on the cell cycle phase at the time of radiation. We have incorporated the data on the percent contributions of NHEJ and HR repair in different cell cycle phases into the model and calculated the repair outcomes from NHEJ and HR models according to these ratios. Cell cycle arrest is implemented in relation to the amount of remaining DSBs in each cell cycle phase after repair. Using the number of unrepaired DSBs, we have calculated the proportion of cells that would progress to the next cell cycle phase as well as the proportion for which the cell death mechanism is triggered. The simulation results showed that ADT combined with IR has enhanced the treatment outcome. The cell survival rate was lower for the combination treatment case (55.7% compared to 62.1 % of IR only case). The cell proliferation was also significantly slower (85-95h doubling time compared to 35-45h of IR only case). The results agreed well with the experimental data that showed 30h and 95.5 h of doubling times for IR only and IR+ADT treatment respectively. The comprehensive model outcomes show that impaired NHEJ and HR dynamics as a result of ADT have the potential to enhance the IR treatment outcomes for PCa patients.

Citation Format: Mengdi Qian, Alexandru Almasan, Evren Gurkan-Cavusoglu. Cell cycle-dependent, comprehensive mathematical modeling of the role of DNA repair in response to radiotherapy for prostate cancer [abstract]. In: Proceedings of the Annual Meeting of the American Association for Cancer Research 2020; 2020 Apr 27-28 and Jun 22-24. Philadelphia (PA): AACR; Cancer Res 2020;80(16 Suppl):Abstract nr 5499.