Related Experiment Video
Updated: Jun 12, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Large language models can effectively extract stroke and reperfusion audit data from medical free-text discharge
Rudy Goh1, Benjamin Cook2, Brandon Stretton1
1Lyell McEwin Hospital, Elizabeth Vale, SA 5112, Australia; SA Health, Adelaide, SA 5000, Australia; University of Adelaide, Adelaide, SA 5005, Australia; Royal Adelaide Hospital, Adelaide, SA 5000, Australia.
Introduction:
Audits are an integral part of effective modern healthcare. The collection of data for audits can be resource intensive. Large language models (LLM) may be able to assist. This pilot study aimed to assess the feasibility of using a LLM to extract stroke audit data from free-text medical documentation.
Method:
Discharge summaries from a one-month retrospective cohort of stroke admissions at a tertiary hospital were collected. A locally-deployed LLM, LLaMA3, was then used to extract a variety of routine stroke audit data from free-text discharge summaries. These data were compared to the previously collected human audit data in the statewide registry. Manual case note review was undertaken in cases of discordance.
Results:
Overall, there was a total of 144 data points that were extracted (9 data points for each of the 16 patients). The LLM was correct in 135/144 (93.8%) of individual datapoints. This performance included binary categorical, multiple-option categorical, datetime, and free-text extraction fields.
Conclusions:
LLM may be able to assist with the efficient collection of stroke audit data. Such approaches may be pursued in other specialties. Future studies should seek to examine the most effective way to deploy such approaches in conjunction with human auditors and researchers.
