Streamlining Ophthalmic Documentation With Anonymized, Fine-Tuned Language Models: Feasibility Study

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Clinician administrative burden, especially in medical documentation, contributes to burnout and patient safety risks.
• Generative artificial intelligence (AI) presents a potential solution for improving documentation and mitigating these challenges.

Purpose of the Study:

• To evaluate the feasibility of using a fine-tuned OpenAI Curie model for automated medical report summary (epicrisis) generation in ophthalmology.
• To assess AI model performance through human and automated evaluations for accuracy, usefulness, and regulatory compliance.
• To determine the potential of AI in reducing clinician workload.

Main Methods:

• A dataset of approximately 60,000 anonymized medical letters was created adhering to General Data Protection Regulation (GDPR) guidelines.
• The OpenAI Curie model was fine-tuned on this dataset to generate epicrises from medical histories, diagnoses, and findings.
• Performance was evaluated using human assessments and automated evaluations from two large language models (LLMs).

Main Results:

• Nearly 50% of AI-generated epicrises were rated as helpful or excellent in a clinical context.
• Human evaluation showed significantly high formal correctness (mean 3.59/4.0) and a significant reduction in correction time compared to manual writing (54.25s vs 109.52s).
• AI-generated reports were significantly shorter, and automated LLM assessments showed consistency with human ratings, supporting the proof of concept.

Conclusions:

• Fine-tuned commercial LLMs are technically and practically feasible for integration into clinical practice, demonstrating time-saving potential.
• AI-generated epicrises showed formal and clinical correctness in many instances, indicating significant workload reduction potential.
• The study successfully demonstrated an anonymization process for handling patient data with AI and outlined a pipeline for integrating LLMs into EU clinical practice, emphasizing safety and efficiency.