HCCD-DS v2: a transparent synthetic benchmark for human-AI decision support under contextual uncertainty

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Machine Learning Benchmarking

Background:

• Current AI decision support benchmarks often neglect the crucial human-AI interaction process.
• This gap limits reproducible analysis of how human judgment impacts system performance.
• Evaluating AI and human collaboration requires datasets that capture interaction dynamics.

Purpose of the Study:

• Introduce HCCD-DS v2, a synthetic benchmark dataset for evaluating human-AI interaction in decision support.
• Model decision-time interactions between AI recommenders and users with varying expertise.
• Facilitate reproducible analysis of human intervention's effect on system outcomes.

Main Methods:

• Developed HCCD-DS v2, a transparent and configurable synthetic dataset.
• Simulated AI recommender and user interactions with continuous expertise levels.
• Incorporated contextual uncertainty, system confidence, explainability, and human override behaviors with rationales.
• Modeled distinct profiles for healthcare, cybersecurity, and Internet of Things (IoT) domains.

Main Results:

• Human intervention's impact on AI decision support is condition-dependent.
• Expert overrides were more likely to improve outcomes compared to novice overrides.
• Dataset supports analysis of trust calibration, explanation-aware policies, and deferral algorithms.

Conclusions:

• HCCD-DS v2 offers a unified, reusable resource for benchmarking human-AI interaction.
• The dataset enables controlled evaluation of AI systems that incorporate human judgment.
• Findings underscore the importance of user expertise in AI-assisted decision-making.