Evaluating the Performance of Large Language Models for One Atmosphere Using Automated Extracted Datasets

Area of Science:

• Atmospheric science
• Environmental science
• Artificial intelligence

Background:

• Large language models (LLMs) offer potential for advancing atmospheric research and air pollution control.
• Lack of systematic evaluation benchmarks hinders trust and real-world deployment of LLMs for critical environmental tasks.

Purpose of the Study:

• To develop and utilize a benchmark dataset (OneAtmos-Bench) for evaluating LLMs in air pollution control.
• To assess the performance of 11 LLMs on accuracy, instruction-following, and hallucination rates within the atmospheric domain.

Main Methods:

• A two-stage automated extraction pipeline was used to create the OneAtmos-Bench dataset.
• Eleven LLMs were evaluated using the OneAtmos-Bench dataset, focusing on accuracy, instruction-following (IF), and hallucination rate (HR).
• Expert validation confirmed the dataset's high accuracy (95.88%).

Main Results:

• LLMs demonstrated strong instruction-following capabilities.
• Accuracy and hallucination suppression remain challenges, particularly for specialized atmospheric tasks.
• Model scaling showed limited gains, indicating sparse domain adaptation; fine-tuning methods may increase hallucination rates.

Conclusions:

• General-purpose LLMs face reliability issues for trustworthy, low-hallucination air pollution control guidance.
• Environment-specific adaptation is necessary to overcome current LLM limitations in atmospheric science applications.
• Further research is needed to enhance LLM trustworthiness for critical environmental decision-making.