Semisupervised transfer learning for evaluation of model classification performance.

Area of Science:

• Machine Learning
• Statistical Modeling
• Biostatistics

Background:

• Transfer learning faces challenges adapting models to new data distributions.
• Evaluating model performance on unlabeled target populations is difficult.
• Existing methods lack robust transfer of performance metrics like ROC parameters.

Purpose of the Study:

• To evaluate a trained binary classifier's performance on unlabeled target data using ROC analysis.
• To propose an efficient method for transferring performance metrics in machine learning.
• To address the need for robust evaluation in transfer learning settings.

Main Methods:

• Proposed Semisupervised Transfer Learning of Accuracy Measures (STEAM), a three-step procedure.
• Employed double-index modeling for calibrated density ratio weights.
• Utilized robust imputation to leverage unlabeled data for improved efficiency.

Main Results:

• Established consistency and asymptotic normality of the proposed estimator.
• Corrected for overfitting bias using cross-validation in finite samples.
• Demonstrated reductions in bias and gains in efficiency compared to existing methods via simulations.

Conclusions:

• STEAM provides an efficient and robust method for evaluating classifier performance on unlabeled data.
• The method is applicable to real-world scenarios, such as evaluating phenotyping models in electronic health records.
• STEAM enhances the reliability of transfer learning by enabling accurate performance metric transfer.