Estimating Success in Predicting a Variable with Nominal Measurement Using Other Variables with Nominal Measurements

Area of Science:

• Statistics
• Machine Learning
• Predictive Modeling

Background:

• Estimating prediction accuracy is crucial in statistical modeling.
• Existing methods for nominal variables can be biased.

Purpose of the Study:

• To develop and evaluate estimators for prediction success probability (p) with nominal variables.
• To compare the bias and Mean Squared Error (MSE) of different estimators.

Main Methods:

• Proposed three estimators: pa (in-sample success rate), pb (holdout group), and pc (average of pa and pb).
• Conducted simulation studies to assess estimator performance.
• Analyzed bias and MSE of pa, pb, and pc.

Main Results:

• Estimator pa is consistently biased upwards.
• Estimator pb is consistently biased downwards.
• Estimator pc demonstrated improved performance in simulations.

Conclusions:

• The proposed combined estimator pc offers a more reliable estimation of prediction probability.
• pc mitigates the upward bias of pa and downward bias of pb.
• Simulation results support pc as a superior estimator for this prediction task.