Harmonizing organ-at-risk structure names using open-source large language models.

Area of Science:

• Radiation Oncology
• Artificial Intelligence
• Natural Language Processing

Background:

• Standardized nomenclature for radiotherapy structures is essential for automation, collaboration, and AI studies.
• Current implementation of nomenclature guidelines (AAPM-TG-263) faces challenges.
• Automated OAR renaming is needed for multi-institutional and multilingual datasets.

Purpose of the Study:

• To evaluate open-source large language models (LLMs) for automated organ-at-risk (OAR) renaming.
• To assess LLM performance on a diverse, multi-institutional, and multilingual dataset.
• To compare reasoning vs. non-reasoning LLMs and uncertainty estimation methods.

Main Methods:

• Four open-source LLMs (Llama 3.3, Llama 3.3 R1, DeepSeek V3, DeepSeek R1) were tested.
• A dataset of 34,177 OAR structures from 1684 patients was used.
• Few-shot prompting and Monte Carlo sampling for uncertainty were employed.

Main Results:

• High renaming accuracy was achieved, with DeepSeek R1 reaching 98.6% unique accuracy.
• Reasoning-enhanced LLMs outperformed non-reasoning models.
• Monte Carlo sampling demonstrated better error correlation and detection than prompt-based confidence.

Conclusions:

• Open-source LLMs can accurately harmonize OAR nomenclature across diverse datasets.
• LLMs facilitate TG-263 adoption and the creation of standardized datasets for research.
• Reasoning LLMs are promising for AI development in radiation oncology.