A model for evaluation of in situ hybridization spot-count distributions in tissue sections

The interpretation of in situ hybridization (ISH) spot-count distributions, obtained from evaluation of ISH signals in tissue sections, is complicated by the unknown impact of nuclear truncation and of the localization of ISH spots within the nuclei. In this study, a mathematical model was developed to investigate the effects of nuclear truncation and of the distribution of ISH spots within the nucleus on the ISH spot-count distribution in tissue sections. In this model, it was assumed that nuclei are spherical and of constant diameter and that ISH spots have negligible size and are distributed randomly within the nucleus ("volume model") or along the nuclear membrane ("surface model"). A minimal nuclear profile diameter was introduced in order to study the effect of rejecting small nuclear fragments for spot-count evaluation. Given the section thickness, the nuclear size, the minimal nuclear profile diameter, and the true number of ISH spots per nucleus and their spatial distribution within the nucleus, the model predicts the proportion of nuclei observable in the section with a specific number of ISH spots. A program that performs the model calculations was developed for PC and is available upon request. For section thickness greater than 50% of the nuclear diameter, the main effect of increasing section thickness on spot-count distributions was the increase of the proportion of nuclei with the true chromosome copy number of spots. For lower section thickness, the total distribution shifted towards lower spot frequencies. The influence of the minimal profile diameter was most notable for values close to the nuclear diameter. The effect of the localization of ISH spots within the nucleus was shown to be prominent, especially for sections with thickness smaller than the nuclear diameter. Good correspondence between model-predicted distributions and measured distributions was obtained using the volume model and taking into account only large nuclear profiles.