APBench and benchmarking large language model performance in fundamental astrodynamics problems for space...

Area of Science:

• Space Engineering
• Astrodynamics
• Artificial Intelligence

Background:

• Large Language Models (LLMs) show promise for advancing STEM research, particularly in complex problem-solving.
• Enhancing LLM capabilities is crucial for their application in specialized fields like space engineering.

Purpose of the Study:

• To evaluate the problem-solving capabilities of LLMs in astrodynamics and space engineering.
• To establish a benchmark for assessing LLM performance in this domain.

Main Methods:

• Development of the Astrodynamics Problems Benchmark (APBench), a novel dataset of questions and answers.
• Evaluation of both open-source and closed foundational models using the APBench dataset.

Main Results:

• The APBench dataset covers diverse subfields within astrodynamics, astronautics, and space engineering.
• Performance validation of current foundational models in space engineering applications.

Conclusions:

• The study introduces the first benchmark for LLMs in astrodynamics.
• This work paves the way for future advancements in AI for space engineering and defining artificial intelligence for space.