Using Open-Source Large Language Models to Identify Access to Germline Genetic Testing in Veterans With Breast Cancer...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Genomics and Precision Medicine

Background:

• Access to germline genetic testing is crucial for guiding cancer treatment and management.
• The use of unstructured clinical notes presents a challenge for extracting this vital information.
• Large language models (LLMs) offer a potential solution for automated data extraction.

Purpose of the Study:

• To evaluate the performance of open-source, locally deployable LLMs in identifying germline genetic testing access among Veterans with breast cancer.
• To compare the efficacy of different LLM models (Llama 3 70B, Llama 3 8B, Llama 2 70B) in this task.

Main Methods:

• A cohort of 1,201 Veterans with breast cancer was identified within the VA system.
• Clinical notes from a subset of 200 patients were reviewed by experts to establish ground truth.
• LLM performance was assessed using accuracy, precision, recall, and F1 scores, with expert consensus as the benchmark.

Main Results:

• Llama 3 70B achieved a high F1 score (0.912), significantly outperforming Llama 3 8B (0.811) and Llama 2 70B (0.644).
• The performance of Llama 3 70B was comparable to that of individual expert reviewers.
• LLM-identified access distribution across the cohort matched expert-identified distribution.

Conclusions:

• Open-source, locally deployable LLMs can effectively and efficiently identify germline genetic testing access from clinical notes.
• LLMs have the potential to enhance the quality and efficiency of cancer care.
• The use of LLMs can aid in safeguarding sensitive patient data within healthcare systems.