Evaluating LLMs' divergent thinking capabilities for scientific idea generation with minimal context.

Area of Science:

• Artificial Intelligence
• Computational Science
• Scientific Discovery

Background:

• Large Language Models (LLMs) excel at scientific tasks like literature analysis and experimental design.
• Existing benchmarks often use rich contextual inputs, potentially overlooking specific creative capabilities.
• Evaluating divergent thinking in LLMs for scientific idea generation remains a challenge.

Purpose of the Study:

• Introduce LiveIdeaBench, a novel benchmark for assessing LLMs' scientific idea generation.
• Evaluate LLMs' divergent thinking using single-keyword prompts, inspired by creativity theory.
• Assess the relationship between general intelligence scores and scientific idea generation capabilities.

Main Methods:

• Developed LiveIdeaBench, a benchmark using 1180 keywords across 22 scientific domains.
• Employed a dynamic panel of over 40 state-of-the-art LLMs.
• Assessed generated ideas across five dimensions: originality, feasibility, fluency, flexibility, and clarity.

Main Results:

• LLMs' scientific idea generation capabilities are poorly predicted by standard general intelligence metrics.
• Models with lower general intelligence scores can exhibit comparable creative performance to higher-scoring models.
• Significant variation exists in creative idea generation across different LLMs.

Conclusions:

• Current evaluation methods may not fully capture LLMs' potential for scientific creativity.
• Specialized benchmarks like LiveIdeaBench are crucial for understanding and improving LLM scientific idea generation.
• Developing LLMs for scientific idea generation may necessitate distinct training strategies compared to general problem-solving.