Validation of logistic regression models in small samples: application to calvarial lesions diagnosis

We have used the leave-one-out (LOO) method and the area under the receiver operating characteristic (ROC) curve to validate logistic models with a sample of 167 patients with calvarial lesions. Seven logistic regression models were developed from 12 clinical and radiological variables to predict the most common diagnoses separately. The LOO method was used to test the validity of the equations. The discriminant power of every model was assessed by means of the area under the ROC curve (Az). The model with the greatest discrimination ability for the whole data set was the osteoma equation (Az = 0.951). The discriminatory ability of the statistical models decreased significantly with the LOO procedure, having the malignancy model the highest value (Az = 0.931). The LOO method can obtain a high benefit from small samples in order to validate prediction rules. In studies with small samples, resampling techniques such as the LOO should be routinely used in predictive modeling. This method may improve the forecast of infrequent diseases, such as calvarial lesions.