A discussion of calibration techniques for evaluating binary and categorical predictive models.

Area of Science:

• Veterinary epidemiology
• Statistical modeling
• Data mining

Background:

• Predictive models are crucial for simulating epidemiological processes in veterinary research.
• Commonly used techniques include logistic regression, decision trees, and support vector machines.
• Model evaluation typically relies on discrimination measures like sensitivity and specificity.

Purpose of the Study:

• To highlight the limitations of discrimination tests in assessing predictive model accuracy.
• To introduce and describe various calibration tests for evaluating predicted probabilities.
• To demonstrate the utility of calibration tests in veterinary epidemiological research.

Main Methods:

• Description of calibration tests: Hosmer-Lemeshow, Brier tests, calibration plots, unreliability test, and mean absolute calibration error.
• Comparison of calibration tests with traditional discrimination tests.
• Illustration using sample predictions from a binary logistic regression model.
• Provision of R statistical programming language code for implementing calibration tests.

Main Results:

• Discrimination tests assess a model's ability to distinguish outcomes but not probability accuracy.
• Calibration tests measure the accuracy of predicted probabilities against observed event rates.
• Calibration tests are particularly informative for models with narrow probability ranges or ~50% prevalence.

Conclusions:

• Thorough model evaluation in veterinary epidemiology necessitates using a suite of both discrimination and calibration tests.
• Calibration tests provide a more comprehensive assessment of predictive accuracy and bias than discrimination tests alone.
• Accurate probability assessment is vital for reliable decision-support and simulation modeling in veterinary research.