Related Experiment Video
Updated: May 16, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Large Language Models Accurately Identify People Who Inject Drugs From Infectious Diseases Discharge Summaries in an
David Goodman-Meza1,2, Marianne Martinello2,3, Jeffrey Masters2,4
1St Vincent's Hospital, Sydney, Australia.
Introduction:
People who inject drugs (PWID) face a high risk for serious infections, yet International Classification of Diseases (ICD) codes fail to identify this population. Large language models (LLM) offer a promising alternative by extracting information from unstructured clinical text. This study evaluated the diagnostic performance of off-the-shelf LLMs in identifying PWID and related attributes from hospital discharge summaries.
Methods:
In this cross-sectional study, discharge summaries from the Infectious Diseases service at St Vincent's Public Hospital, Sydney, between 2018 and 2022 were reviewed. A single reviewer manually annotated each de-identified summary for PWID status, drugs reported, injection recency and opioid agonist therapy. Eight LLMs (Gemma3, Llama 3.3, Mistral, Phi4, hippomistral, llama3-med [8B and 70B] and OpenBioLLM) were compared using prevalence-weighted average-F1 scores. Diagnostic metrics with bootstrapped 95% confidence intervals were calculated for each annotated category.
Results:
Of 859 first admissions, manual review identified 149 (17.1%) PWID. ICD codes showed low sensitivity (≤ 0.32) but high specificity (≥ 0.97) for identifying PWID. The best-performing model (Llama 3.3) achieved a prevalence-weighted average-F1 of 0.845 (0.733, 0.927). For injecting drug use, sensitivity was 0.819 (95% CI 0.753, 0.879) and specificity 0.999 (0.996, 1.00). Identification of heroin, methamphetamine, cannabis and methadone was near perfect (F1 > 0.973), while illicit prescription opioid and benzodiazepine use were identified less accurately (F1 = 0.400 and 0.606).
Discussion And Conclusions:
LLMs accurately identify PWID from discharge summaries, outperforming ICD codes. Challenges remain for certain substances, underscoring the need for task-specific tuning, external validation and integration with structured data to enhance surveillance and interventions.
