Characterising liver lesions from free-text computer tomography reports - A real-world multicentre analysis

This study evaluates the performance of a general-purpose (GPT-4) and a medically fine-tuned (Med-LM) large language model (LLM) in classifying liver lesions from unstructured Computed Tomography (CT) reports.

Consecutive CT reports (2014-2020) from five institutions were input into GPT-4 and Med-LM withsimple (sp)andoptimised (op) prompts. Lesion-andpatient-level performance were benchmarked against LI-RADS scores assigned by two radiologists, and report quality was analysed using a5-point Likert scale.

A total of 296 CT reports (mean age, 64.6 years ± 11.3 [SD]; 193 men; 654 lesions) were included. Lesion- and patient-level accuracies for LI-RADS scoring ranged from 40.8% (Med-LMsp) to 61.3% (Med-LMop) and from 27.7% (Med-LMsp) to 52.4% (Med-LMop), respectively. When dichotomized into malignant and benign lesions, lesion- and patient-level accuracies rose to 56.1% (GPT-4sp) - 82.3% (Med-LMop) and 71.3% (Med-LMsp) - 86.5% (Med-LMop). Med-LMop demonstrated the highest performance in all analyses and was statistically superior to other models (all p < 0.001). Non-classification rates ranged between 12.7% (Med-LMop) and 40.5% (GPT-4sp), particularly for benign lesions. Kappa values were weak to moderate between the two reviewers in different aspects of report quality (0.471-0.766), and Likert scores for lesion information differed significantly between correctly and incorrectly classified lesions (all p ≤ 0.04). Repeatability varied widely from 12.7% (Med-LMop) to 39.0% (GPT-4sp).

Med-LM outperforms GPT-4 in classifying liver lesions from unstructured CT reports with both models better at detecting malignancy than full LI-RADS classification. However, high misclassification rates and inconsistent repeatability hinder their clinical use.