Large language model based mutations in genetic improvement

Area of Science:

• Software Engineering
• Artificial Intelligence

Background:

• Large language models (LLMs) are increasingly used in software engineering.
• Combining LLMs with search-based software engineering (SBSE) for genetic improvement (GI) remains under-explored.

Purpose of the Study:

• To evaluate LLMs as mutation operators for GI in SBSE.
• To assess the effectiveness of LLM-generated code edits across different LLMs, prompts, and software projects.

Main Methods:

• Utilized three LLMs and three prompt types across five real-world software projects.
• Employed random sampling and local search for edit generation.
• Conducted a qualitative analysis of LLM-generated code edit failures.

Main Results:

• LLMs produced fewer unique edits than conventional statement GI edits.
• LLM-generated edits compiled and passed tests more frequently (OpenAI: 77%).
• OpenAI and Mistral LLMs showed similar performance in identifying runtime improvements; simpler prompts were more effective.

Conclusions:

• LLMs offer a viable alternative to traditional mutation operators in GI for SBSE.
• While LLMs generate fewer edits, they exhibit higher success rates in producing functional code.
• Common failure modes include formatting inconsistencies, invalid syntax, and refusal to generate solutions, necessitating further research into prompt engineering and model refinement.