Evaluating multimodal commercial and open-source large language models for dynamical astronomy: a benchmark study of...

Area of Science:

• Dynamical Astronomy
• Machine Learning Applications
• Computational Astrophysics

Background:

• Orbital resonances are key phenomena in celestial mechanics, influencing the long-term stability and evolution of planetary systems.
• Classifying these resonances typically requires specialized algorithms or manual analysis of complex orbital data.
• Modern large language models (LLMs) offer potential for image-based analysis and pattern recognition, but their application in dynamical astronomy is underexplored.

Purpose of the Study:

• To systematically evaluate the performance of various large language models (LLMs) in classifying mean-motion and secular resonances from images of resonant arguments.
• To establish standardized benchmarks and datasets for reproducible evaluation of LLMs in dynamical astronomy.
• To compare the capabilities of commercial, open-source, and small-scale LLMs on this classification task.

Main Methods:

• Development of four benchmark datasets (RB-TEST, RB-PILOT, RB-SMALL, RB-FULL) with varying complexity (clear, ambiguous, transient cases) and output types (binary, three-class).
• Standardized prompting strategies were employed for different model sizes (full prompts for large models, simplified for small models).
• Evaluation of flagship commercial LLMs, large open-source models, and small, locally runnable models on the prepared datasets.

Main Results:

• Commercial LLMs achieved high accuracy ([Formula: see text]–[Formula: see text]) on simpler cases and up to [Formula: see text] on a three-class dataset.
• Top open-source models demonstrated comparable performance, reaching [Formula: see text] on unambiguous cases and [Formula: see text] on complex ones.
• Open-source models approached commercial performance on the full binary benchmark ([Formula: see text]–[Formula: see text]), with most errors in transient or resonance-sticking regimes.

Conclusions:

• LLMs can perform resonance classification with accuracy comparable to traditional and machine-learning methods, requiring no specific training or fine-tuning.
• Even small, open-source LLMs achieve practically useful accuracy, making them viable tools for dynamical astronomy.
• The established benchmarks provide a reproducible standard for future LLM evaluations in this scientific domain.