Probability dynamics of a repopulating tumor in case of fractionated external radiotherapy

In this work two analytical methods are developed for computing the probability distribution of the number of surviving cells of a repopulating tumor during a fractionated external radio-treatment. Both methods are developed for the case of pure birth processes. They both allow the description of the tumor dynamics in case of cell radiosensitivity changing in time and for treatment schedules with variable dose per fraction and variable time intervals between fractions. The first method is based on a direct solution of the set of differential equations describing the tumor dynamics. The second method is based on the works of Hanin et al. [Hanin LG, Zaider M, Yakovlev AY. Distribution of the number of clonogens surviving fractionated radiotherapy: a long-standing problem revisited. Int J Radiat Biol 2001;77:205-13; Hanin LG. Iterated birth and death process as a model of radiation cell survival. Math Biosci 2001;169:89-107; Hanin LG. A stochastic model of tumor response to fractionated radiation: limit theorems and rate of convergence. Math Biosci 2004;191:1-17], where probability generating functions are used. In addition a Monte Carlo algorithm for simulating the probability distributions is developed for the same treatment conditions as for the analytical methods. The probability distributions predicted by the three methods are compared graphically for a certain set of values of the model parameters and an excellent agreement is found to exist between all three results, thus proving the correct implementation of the methods. However, numerical difficulties have been encountered with both analytical methods depending on the values of the model parameters. Therefore, the Poisson approximation is also studied and it is compared to the exact methods for several different combinations of the model parameter values. It is concluded that the Poisson approximation works sufficiently well only for slowly repopulating tumors and a low cell survival probability and that it cannot always be used for the description of the tumor dynamics. The two analytical methods and the Monte Carlo method are compared in terms of computational time. The Monte Carlo method is found to be much more time consuming than the analytical methods.