Enhancing Oncology-Specific Question Answering With Large Language Models Through Fine-Tuned Embeddings With...

Area of Science:

• Medical Informatics
• Natural Language Processing
• Oncology Data Extraction

Background:

• Advancements in retrieval-augmented generation (RAG) and large language models (LLMs) have transformed real-world evidence extraction from electronic health records (EHRs).
• Extracting precise clinical information from unstructured oncology EHRs remains a challenge, impacting research and patient care.

Purpose of the Study:

• To enhance RAG effectiveness for oncology EHRs by developing a specialized retriever encoder.
• To improve the precision and relevance of retrieved clinical notes for oncology-specific queries.

Main Methods:

• Pretraining a retriever encoder on over six million oncology notes from 209,135 patients.
• Fine-tuning the model as a sentence transformer using 12,371 LLM-synthesized query-passage pairs.
• Evaluating retrieval performance against six embedding models using NDCG, Precision, and Recall metrics on 50 oncology questions.

Main Results:

• The developed model outperformed the runner-up by 9% in NDCG, 7% in Precision, and 6% in Recall (top 10 results).
• Exceptional retrieval performance was observed across all metrics for key oncology categories such as diagnosis, disease status, and tumor characteristics.
• The model demonstrated superior ability in retrieving pertinent clinical notes from oncology EHRs.

Conclusions:

• Pretrained contextual embeddings and sentence transformers are effective for retrieving relevant oncology EHR notes.
• LLM-synthesized query-passage pairs offer a viable data augmentation strategy for specialized domains.
• This fine-tuning approach shows promise for improving data extraction in healthcare settings with limited annotated data.