Using large language models to suggest informative prior distributions in Bayesian regression analysis

Area of Science:

• Bayesian statistics
• Machine learning
• Statistical modeling

Background:

• Selecting prior distributions in Bayesian regression is complex and subjective.
• Existing methods for eliciting informative priors are resource-intensive and difficult to perform objectively.

Purpose of the Study:

• To investigate the potential of large language models (LLMs) in suggesting suitable prior distributions for Bayesian regression analysis.
• To evaluate the performance of different LLMs in generating knowledge-based and objective informative priors.

Main Methods:

• Developed an extensive prompt for LLMs to suggest, verify, and reflect on prior distributions.
• Evaluated three LLMs (Claude Opus, Gemini 2.5 pro, ChatGPT 4o-mini) on two real-world datasets (heart disease risk, concrete strength).
• Assessed prior distribution quality using Kullback-Leibler divergence against the maximum likelihood estimator's distribution.

Main Results:

• LLMs successfully suggested the correct direction of associations for variables in both datasets.
• Claude and Gemini generally outperformed ChatGPT in suggesting prior distributions.
• Moderate informative priors suggested by LLMs were often too confident, showing limited agreement with the data.
• Claude demonstrated an advantage by not defaulting to a mean of 0 for weakly informative priors, unlike ChatGPT and Gemini.

Conclusions:

• LLMs show significant potential for developing efficient and objective informative prior distributions in Bayesian regression.
• A key challenge lies in calibrating the width of LLM-suggested priors, as they exhibit tendencies towards overconfidence and underconfidence.
• Claude Opus exhibited a notable advantage in its approach to suggesting priors compared to Gemini and ChatGPT.