一个基准测试框架和数据集,用于学习在人-人工智能决策中延迟决策
Jean V Alves1, Diogo Leitão2, Sérgio Jesus2
1Feedzai, Coimbra, Portugal. jean.alves@feedzai.com.
Scientific data
|April 23, 2025
概括
学习推迟 (L2D) 算法增强了人与人工智能的协作. 一个新的框架,OpenL2D,为更好的L2D系统测试生成现实的合成专家,揭示基于专家多样性的性能变化.
科学领域:
- 人工智能的人工智能
- 人与计算机的交互
- 机器学习 机器学习
背景情况:
- 学习推迟 (L2D) 算法对于人类-人工智能在高风险领域的合作至关重要,例如欺诈检测.
- 目前的L2D基准经常使用简化的模拟专家,因为真实专家数据的成本很高.
- 这限制了在关键应用中对L2D系统的现实评估.
研究的目的:
- 引入OpenL2D,这是一个用于生成具有可调节参数的合成专家的新框架,用于L2D系统评估.
- 使用合成专家创建一个更现实的L2D算法基准数据集.
- 分析专家多样性对L2D算法性能的影响.
主要方法:
- 开发了OpenL2D以生成具有可控制决策过程和工作能力的合成专家.
- 将OpenL2D应用于公共欺诈检测数据集,以创建金融欺诈警报审查 (FiFAR) 数据集.
- 收集了50名欺诈分析师对FiFAR数据集中的3万个实例的预测.
- 评估了合成专家与真实专家的相似性,使用如一致性和专家间协议等指标.
主要成果:
- 由OpenL2D生成的合成专家表现出与真正的人类专家相比的一致性和专家间的一致性.
- 不同L2D算法的性能排名在与各种合成专家池进行评估时有显著差异.
- 该研究强调了专家特征对L2D算法有效性的关键影响.
结论:
- OpenL2D提供了一个可扩展和现实的方法来对L2D算法进行基准测试.
- 现实的专家建模对于在现实场景中准确评估L2D系统性能至关重要.
- 未来的L2D研究和开发应该考虑到人类专家行为的变化和多样性.
相关概念视频
01:21Heuristics
46
Heuristics are problem-solving strategies that use mental shortcuts to simplify decision-making. Unlike algorithms, which must be followed precisely to achieve a correct result, heuristics offer a general problem-solving framework. They save time and energy but can sometimes lead to less rational decisions.
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
People often rely on heuristics when faced with an overload of information, limited time, low importance of the decision, limited information, or when a heuristic readily comes to mind. For...
46
01:20Decision Making
69
Decision-making is a fundamental cognitive process that involves evaluating alternatives and selecting among them. This process can range from simple choices, such as deciding what to wear, to complex decisions, like choosing a major in college or a career path. The complexity of the decision often dictates the approach we use, which can be broadly categorized into two types: automatic and controlled decision-making.
Automatic decision-making is fast, intuitive, and relies on gut feelings...
Automatic decision-making is fast, intuitive, and relies on gut feelings...
69
01:08The Availability Heuristic
5.8K
A heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. Different types of heuristics are used in different types of situations, and the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):
5.8K