Bridging the performance gap: systematic optimization of local LLMs for Japanese medical PHI extraction

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Cloud-based Large Language Models (LLMs) offer advanced medical text processing but face privacy challenges due to Protected Health Information (PHI) transmission restrictions.
• Healthcare institutions often require data sovereignty, limiting the use of external AI services despite potential contractual agreements like Business Associate Agreements (BAAs).
• Local LLMs provide data privacy but have historically lagged in performance compared to cloud-based counterparts.

Purpose of the Study:

• To introduce a novel five-phase optimization framework to enhance the performance of local LLMs for medical text processing.
• To benchmark local LLMs against cloud leaders using synthetic Japanese radiology reports.
• To investigate performance patterns and identify potential optimization thresholds for local LLMs.

Main Methods:

• Development and application of a five-phase optimization framework for local LLMs.
• Benchmarking 14 local LLMs against cloud-based models using 160 synthetic Japanese radiology reports.
• Performance analysis, including statistical evaluation of score improvements and processing times.

Main Results:

• A significant performance gain (+6.92 points, p < 0.001) was observed in local LLMs with baseline scores below 87-88, suggesting a potential optimization threshold.
• The optimized Mistral-Small-3.2 model achieved 91.54 points, reaching 97.8% of GPT-4.1's performance.
• The optimized model demonstrated perfect rule adherence and a clinically acceptable processing time of 24.6 seconds per report in batch workflows.

Conclusions:

• Systematically optimized local LLMs can achieve performance comparable to leading cloud-based models.
• The proposed optimization framework guides healthcare institutions in strategically deploying AI, balancing performance, efficiency, and patient privacy.
• This approach enables trustworthy AI adoption by ensuring data sovereignty and adherence to privacy regulations.