Latent Supervision: A Method for Improved Performance and Calibration of Machine Learning Classification Models in...

Area of Science:

• Ophthalmic artificial intelligence
• Medical diagnostics
• Machine learning

Background:

• Standard supervised learning models assume deterministic labels, overlooking clinical diagnostic uncertainty.
• Ophthalmic artificial intelligence classifiers require methods that can handle uncertainty for improved accuracy and calibration.

Purpose of the Study:

• To introduce latent supervision, a novel algorithm using latent class analysis.
• To incorporate multiple diagnostic tests or expert opinions for probabilistic labels (soft labels).
• To develop more accurate and calibrated ophthalmic artificial intelligence classifiers.

Main Methods:

• Compared latent supervision against standard supervised learning in two computer vision scenarios: trachoma screening and infectious keratitis diagnosis.
• Utilized probabilistic labels derived from multiple grader teams for trachoma screening.
• Employed labels from culture and smear results for infectious keratitis pathogen differentiation.

Main Results:

• Latent supervision achieved a higher area under the receiver operating characteristic curve (0.94) in trachoma screening compared to ensemble supervised models (0.93).
• In infectious keratitis, latent supervision demonstrated consistent performance (area under the receiver operating characteristic curve 0.86) across bacterial and fungal differentiation.
• Latent supervision models exhibited superior calibration in both scenarios.

Conclusions:

• Latent supervision offers a computationally inexpensive approach to train AI models using probabilistic labels that capture diagnostic uncertainty.
• The method enhances both classification performance and model calibration, crucial for clinical trust and implementation.
• Latent supervision shows promise for improving AI applications in medicine by addressing inherent diagnostic uncertainties.