Achieving GPT-4o level performance in astronomy with a specialized 8B-parameter large language model.

Area of Science:

• Astronomy
• Astrophysics
• Cosmology

Background:

• General-purpose AI models often lack specialized knowledge for complex scientific domains.
• The need for accessible, high-performance AI tools in astronomical research is growing.

Purpose of the Study:

• To develop a domain-specialized AI assistant for astronomy, astrophysics, and cosmology.
• To evaluate the performance of this specialized AI against existing benchmarks and larger models.

Main Methods:

• Training AstroSage-Llama-3.1-8B on a comprehensive dataset of astronomy arXiv papers (2007-2024).
• Incorporating millions of synthetic question-answer pairs and astronomical literature into the training data.
• Evaluating performance on the AstroMLab-1 benchmark.

Main Results:

• AstroSage-Llama-3.1-8B achieved an 80.9% score on the AstroMLab-1 benchmark.
• Outperformed all other 8-billion parameter models, both proprietary and open-weight.
• Demonstrated performance comparable to GPT-4o, a significantly larger model.

Conclusions:

• Domain specialization in AI can yield superior performance compared to general-purpose models, even at smaller parameter counts.
• AstroSage-Llama-3.1-8B offers advanced, freely accessible AI capabilities for astronomical research and education.
• Focused AI training enhances proficiency in specialized scientific fields like astronomy.