Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types

Medical documentation, including reply letters from specialty to primary care, imposes a substantial burden on clinicians. Recent advances in large language models (LLMs) may help alleviate this burden. This study aims to develop an instruction-tuned LLM to generate reply letters by integrating five types of electronic medical record documents: admission notes, progress notes, procedure notes, operative notes, and discharge summaries. A total of 8,527 hospitalization episodes from the cardiology and thoracic surgery departments of a tertiary hospital were used for training, and 105 episodes were allocated to the test set. Mistral-Nemo-Instruct-2407 was fine-tuned using the instruction-tuning dataset generated with GPT-4.1. Model performance was evaluated using a dual framework combining expert assessment and a GPT-based evaluation. Experts rated human-written letters higher in accuracy (4.77 vs. 4.21, p < 0.001) and appropriateness (4.90 vs. 4.63, p < 0.001), whereas GPT-based evaluations rated AI-generated letters higher across all dimensions. A hybrid scoring approach integrating expert and GPT-based evaluations improved score consistency among evaluators; in an exploratory comparison using a common weighting, the largest proportion of cases was classified as comparable between the two letter types across all quality dimensions. Expert ratings for workflow suitability (M = 4.30) indicated positive expectations for practical implementation. These findings suggest that LLM-based reply letter generation may serve as a supportive tool for drafting clinical correspondence, although its safety and impact on documentation burden require prospective evaluation in real-world clinical workflows.