AI assists adversarial collaboration in debate on minority salience

Area of Science:

• Cognitive Psychology
• Artificial Intelligence
• Scientific Methodology

Background:

• Scientific progress relies on testing hypotheses, but disputes often persist.
• Adversarial collaboration offers a method for resolving disagreements through joint experimental design.
• The potential of large language models (LLMs) in facilitating such collaborations is underexplored.

Purpose of the Study:

• To investigate the role of LLMs in AI-assisted adversarial collaboration for resolving scientific debates.
• To address a specific debate in PNAS concerning minority salience and its relation to community demographics.
• To formalize claims, structure disagreements, and generate experimental designs using AI.

Main Methods:

• Utilized LLMs to extract and organize competing propositions from existing research.
• Employed LLMs to generate initial experimental designs for testing hypotheses.
• Human collaborators refined AI-generated designs, leading to two preregistered experiments.

Main Results:

• Experiments confirmed that individuals overestimate the percentage of minority faces in visual displays.
• This overestimation was significantly greater when the minorities depicted were also minorities within the participants' communities.
• The study demonstrated convergence of confidence in key hypotheses among researchers after reviewing the results.

Conclusions:

• AI-assisted adversarial collaboration can effectively formalize scientific claims and structure disagreements.
• LLMs can lower barriers to collaboration and act as impartial observers, enhancing fairness perceptions.
• This approach offers a novel method for resolving scientific disputes and advancing research consensus.