Closed-Loop Uncertainty: The Evaluation and Calibration of Uncertainty for Human-Machine Teams under Data Drift

Area of Science:

• Human-Machine Systems
• Machine Learning
• Decision Science

Background:

• Accurate uncertainty measurement is crucial for human-machine team success.
• Current methods often aggregate evaluations, failing to capture iterative control processes.
• Entropy metrics may falter during cold start or data drift without immediate feedback.

Purpose of the Study:

• To present a stochastic framework for iteratively evaluating uncertainty models as a probability of machine correctness.
• To address the threshold selection problem, a novel subjective user task for experimentation.
• To explore incorporating machine correctness feedback into a baseline model using reinforcement learning.

Main Methods:

• Developed a stochastic framework for iterative uncertainty model evaluation.
• Introduced the threshold selection problem for human-machine experimentation.
• Implemented a reinforcement learning approach to refine a Naive Bayes uncertainty model with correctness feedback.

Main Results:

• The novel closed-loop uncertainty approach was tested iteratively.
• Experiments demonstrated consistent outperformance against the baseline model.
• An average improvement of approximately 45% in uncertainty evaluation was achieved.

Conclusions:

• Iterative feedback of machine correctness significantly enhances uncertainty modeling.
• The closed-loop uncertainty framework offers a robust method for improving human-machine team performance.
• This approach provides a more accurate probability of machine correctness, especially in dynamic environments.