Using Item Response Theory for Explainable Machine Learning in Predicting Mortality in the Intensive Care Unit:
Adrienne Kline1,2,3, Theresa Kline4, Zahra Shakeri Hossein Abad3,5
1Department of Biomedical Engineering, University of Calgary, Calgary, AB, Canada.
Journal of Medical Internet Research
|September 25, 2020
Summary
Item response theory (IRT) offers a novel approach to evaluate machine learning (ML) in healthcare by stratifying cases by difficulty. This method provides a more nuanced understanding of ML classifier performance beyond traditional metrics.
Area of Science:
- Machine learning applications in healthcare.
- Statistical modeling for clinical prediction.
Background:
- Current supervised machine learning (ML) evaluations in healthcare often rely on aggregate metrics (accuracy, sensitivity, etc.) that do not account for individual case difficulty.
- This overlooks the inherent variability in patient cases, limiting a comprehensive understanding of ML model performance.
Purpose of the Study:
- To demonstrate the utility of item response theory (IRT) for stratifying patient cases based on classification difficulty.
- To enable a case-based evaluation of ML classifiers, moving beyond single scalar performance measures to a distribution of performance across difficulty levels.
Main Methods:
- Utilized two large intensive care unit datasets (MIMIC-III, eICU) for mortality prediction.
- Employed a 2-parameter logistic model within IRT to score case difficulty.
- Evaluated multiple ML algorithms (logistic regression, LDA, k-NN, decision tree, naive Bayes, neural network) across varying sample balances and assessed performance using generalized linear mixed models.
Main Results:
- Significant effects of case difficulty strata, ML algorithm, and their interaction were observed on prediction accuracy (P<.001).
- All classifiers performed better on easier cases; neural networks showed the best overall performance.
- Specific algorithms were recommended for different difficulty strata, with logistic regression, LDA, decision tree, and neural networks suitable for challenging cases, while naive Bayes and k-NN were less effective.
Conclusions:
- IRT provides a viable, data-driven method to understand ML algorithm performance across varying case difficulties, independent of outcome measures.
- This approach enhances interpretability by identifying features indicative of health states and explains classifier behavior.
- Enables personalized medicine by allowing the selection of appropriate classifiers based on patient-specific case difficulty.
Related Concept Videos
01:24Kaplan-Meier Approach
444
The Kaplan-Meier estimator is a non-parametric method used to estimate the survival function from time-to-event data. In medical research, it is frequently employed to measure the proportion of patients surviving for a certain period after treatment. This estimator is fundamental in analyzing time-to-event data, making it indispensable in clinical trials, epidemiological studies, and reliability engineering. By estimating survival probabilities, researchers can evaluate treatment effectiveness,...
444
01:18Introduction To Survival Analysis
568
Survival analysis is a statistical method used to study time-to-event data, where the "event" might represent outcomes like death, disease relapse, system failure, or recovery. A unique feature of survival data is censoring, which occurs when the event of interest has not been observed for some individuals during the study period. This requires specialized techniques to handle incomplete data effectively.
The primary goal of survival analysis is to estimate survival time—the time...
The primary goal of survival analysis is to estimate survival time—the time...
568
01:20Actuarial Approach
217
The actuarial approach, a statistical method originally developed for life insurance risk assessment, is widely used to calculate survival rates in clinical and population studies. This method accounts for participants lost to follow-up or those who die from causes unrelated to the study, ensuring a more accurate representation of survival probabilities.
Consider the example of a high-risk surgical procedure with significant early-stage mortality. A two-year clinical study is conducted,...
Consider the example of a high-risk surgical procedure with significant early-stage mortality. A two-year clinical study is conducted,...
217
01:20Comparing the Survival Analysis of Two or More Groups
452
Survival analysis is a cornerstone of medical research, used to evaluate the time until an event of interest occurs, such as death, disease recurrence, or recovery. Unlike standard statistical methods, survival analysis is particularly adept at handling censored data—instances where the event has not occurred for some participants by the end of the study or remains unobserved. To address these unique challenges, specialized techniques like the Kaplan-Meier estimator, log-rank test, and...
452
01:15Methods of Documentation VI: Case Management Model
761
The case management model is a multidisciplinary approach that involves healthcare professionals from diverse disciplines, such as physicians, nurses, therapists, social workers, and pharmacists, working collaboratively to address the various needs of patients. Each healthcare professional brings unique expertise and perspectives, contributing to a more comprehensive understanding of the patient's condition and tailoring treatment plans accordingly.
For example, a patient with a chronic...
For example, a patient with a chronic...
761
01:18Steps in Outbreak Investigation
394
In the ever-evolving field of public health, statistical analysis serves as a cornerstone for understanding and managing disease outbreaks. By leveraging various statistical tools, health professionals can predict potential outbreaks, analyze ongoing situations, and devise effective responses to mitigate impact. For that to happen, there are a few possible stages of the analysis:
394