Comprehensive testing of large language models for extraction of structured data in pathology

Area of Science:

• Medical Informatics
• Natural Language Processing
• Computational Pathology

Background:

• Pathology departments produce vast amounts of unstructured text data in diagnostic reports.
• Manual structuring of this data for AI and analytics is labor-intensive and costly.
• Existing solutions often rely on proprietary language models, posing privacy and cost challenges.

Purpose of the Study:

• To evaluate the performance of open-source language models in extracting structured data from pathology reports.
• To compare open-source models against proprietary models like GPT-4.
• To investigate the impact of prompt engineering and model quantization on performance for practical deployment.

Main Methods:

• A bilingual dataset of 579 annotated German and English pathology reports was created.
• Six language models (GPT-4, Llama2, Llama3, Qwen2.5) were tested for extracting 11 key parameters.
• Prompt engineering strategies and quantization techniques were analyzed for deployment feasibility.

Main Results:

• Open-source language models achieved high precision in data extraction, comparable to GPT-4.
• Model performance varied significantly based on the specific model, prompt strategy, and quantization method.
• These factors are critical for optimizing practical deployment in pathology settings.

Conclusions:

• Open-source language models offer a viable, high-performance alternative to proprietary solutions for structuring pathology data.
• This research provides a cost-effective and privacy-conscious approach for healthcare institutions.
• The findings and the public dataset offer valuable insights for future research and development in computational pathology.