Configuring large language models to deliver patient-facing explanations of pathology reports

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Patient Communication

Background:

• Increasing need for accessible patient-facing explanations of complex pathology reports.
• Evaluating the quality of patient-facing medical information generated by artificial intelligence.
• Assessing the impact of large language model (LLM) configuration on response quality.

Purpose of the Study:

• To compare the quality of LLM-generated explanations of pathology reports with pathologist-written explanations.
• To evaluate how different LLM configurations influence the performance of patient-facing medical information.
• To determine if LLMs can provide accurate, clear, and empathetic responses to patient questions about pathology reports.

Main Methods:

• Utilized 5 LLM configurations with varying model architectures, prompting strategies, and retrieval-augmented generation.
• Anonymized patient questions from an online pathology education platform were used for response generation.
• Responses were assessed using a structured rubric (accuracy, relevance, clarity, empathy, safety) and compared via pairwise arena testing.

Main Results:

• LLM responses showed performance comparable to pathologist explanations across rubric domains.
• One LLM configuration met noninferiority criteria, demonstrating high-quality output.
• Configuration parameters, particularly model size and retrieval augmentation, significantly influenced response preference, with a larger model and retrieval from a curated knowledge base performing best.

Conclusions:

• Well-configured LLM systems can produce patient-facing pathology report explanations of quality similar to those by pathologists.
• System configuration (prompting, model size, retrieval integration) is crucial for developing effective LLM-based tools.
• LLMs hold potential to improve access to understandable pathology information for patients.