Deriving the OTA/AO fracture classification from routinely collected radiology reports using a large language model

Area of Science:

• Orthopaedic surgery
• Medical informatics
• Artificial intelligence in medicine

Background:

• Accurate fracture classification (e.g., OTA/AO) is crucial for research and quality assurance.
• Current documentation of OTA/AO classification in electronic medical records is limited, hindering registry creation and interdisciplinary communication.

Purpose of the Study:

• To investigate the efficacy of large language models (LLMs) in translating free-text fracture descriptions from radiology reports into standardized OTA/AO classification labels.

Main Methods:

• A Health Insurance Portability and Accountability Act-compliant LLM was used to classify 109 fracture descriptions from deidentified radiology reports.
• Ground-truth classifications were established by orthopaedic traumatologists.
• Multiple prompting strategies (zero-shot, zero-shot chain-of-thought, retrieval-augmented generation) were evaluated, alongside classification of "ideal" fracture descriptions.

Main Results:

• LLMs achieved near-perfect agreement at the bone and bone/location levels for radiology report descriptions but declined at the subgroup level.
• Retrieval-augmented generation with ideal descriptions showed moderate agreement, with errors attributed to imprecise or inaccurate information.
• Performance was similar across tested prompting strategies for free-text reports.

Conclusions:

• LLMs demonstrate potential for efficient, large-scale labeling of fracture descriptions in radiology reports for OTA/AO classification.
• Further research is needed to enhance LLM classification accuracy through advanced prompting techniques.