Discrimination and reproducibility of an information maximizing multivariable model

Predictor variables for multivariate rules are frequently selected by methods that maximize likelihood rather than information. We compared the discrimination and reproducibility of a prediction rule for pneumonia derived using extended dependency analysis (EDA), an information maximizing variable selection program, with that of a validated rule derived using logistic regression. Discrimination was measured by receiver-operating characteristic (ROC) analysis, and reproducibility by rederivation of the rule on 200 replicate samples of size 250 and 500, generated from a training cohort of 905 patients using Monte Carlo techniques. Four of the five predictor variables selected by EDA were identical to those selected by logistic regression. With each variable weighted by its conditional contribution to total information transmission, EDA discriminated pneumonia and nonpneumonia in the training cohort with an ROC area of 0.800 (vs 0.816 for logistic regression, p = 0.60), and in the validation cohort with an area of 0.822 (vs 0.821 for logistic regression, p = 0.98). EDA demonstrated reproducibility comparable to that of logistic regression according to most criteria for replicability. Replicate EDA models showed good discrimination in the training and testing cohorts, and met statistical criteria for validation (no significant difference in ROC areas at a one-tailed alpha level of 0.05) in 80.8% to 94.2% of cases. We conclude that extended dependency analysis selected the most important variables for predicting pneumonia, based on a validated logistic regression model. The information-theoretic model showed good discriminatory power, and demonstrated reproducibility according to clinically reasonable criteria. Information-theoretic variable selection by extended dependency analysis appears to be a reasonable basis for developing clinical prediction rules.