Individuals vs. BARD: Experimental Evaluation of an Online System for Structured, Collaborative Bayesian Reasoning

Area of Science:

• Cognitive Science
• Artificial Intelligence
• Decision Analysis

Background:

• US intelligence analysts face challenges in probabilistic and causal reasoning, often working without specialized tools.
• Existing methods can lead to reasoning errors, necessitating the development of structured, collaborative approaches.
• The US government's CREATE program aimed to foster innovation in analytic methods.

Purpose of the Study:

• To evaluate the integrated Bayesian Analysis and Reasoning in Delphi (BARD) system as a novel approach to intelligence analysis.
• To assess BARD's effectiveness in improving probabilistic and causal reasoning compared to traditional methods.
• To test BARD's potential as an alternative to current intelligence analysis practices.

Main Methods:

• Developed the BARD system, integrating causal Bayesian network (BN) models with the Delphi technique for small-group collaboration.
• Incorporated online training for novices, AI-assisted report generation, and checklist-inspired templates.
• Conducted an experiment with 256 participants, randomly assigning them to BARD teams or a control group using Google Suite and pen-and-paper methods.

Main Results:

• BARD teams significantly outperformed the control group across three probabilistic reasoning problems, demonstrating very large to huge effect sizes (Glass' Δ 1.4–2.2).
• These results exceeded the initial targets set by the CREATE program.
• Prior experiments indicated BARD's BN-building enhances individual reasoning (Δ 0.8) and Delphi collaboration improves BN structures (Δ 0.5–1.3).

Conclusions:

• The integrated BARD system offers substantial advantages for intelligence analysis over traditional methods and existing BN software for suitable problems.
• The combination of BN-building and collaborative techniques appears to be beneficial and cumulative.
• BARD shows significant potential for further development, testing on complex problems, and application beyond intelligence analysis.