Assessment of a Zero-Shot Large Language Model in Measuring Documented Goals-of-Care Discussions.

Area of Science:

• Clinical Informatics
• Natural Language Processing (NLP)
• Palliative Care Research

Background:

• Goals-of-care (GOC) discussions are crucial in palliative care but identifying their documentation is challenging.
• Existing NLP models require expensive, task-specific training data.
• Large language models (LLMs) present a potential solution for data-efficient outcome measurement.

Purpose of the Study:

• To evaluate a zero-shot LLM's performance in identifying documented GOC discussions.
• To compare LLM performance against a traditional, task-specific NLP model.

Main Methods:

• Compared Llama 3.3 (zero-shot LLM) with a task-specific BERT model trained on 4,642 notes.
• Evaluated models on clinical trial records of hospitalized patients with life-limiting illnesses (2018-2023).
• Assessed performance using AUC, AUPRC, and F1 score at note and patient levels.

Main Results:

• GOC documentation was sparse (<1% of text, 7.3-9.9% of notes).
• Llama 3.3 (zero-shot) and BERT (task-specific) showed comparable performance in identifying GOC documentation.
• Key metrics: Llama 3.3 (AUC 0.979, AUPRC 0.873, F1 0.83) vs. BERT (AUC 0.981, AUPRC 0.874, F1 0.83).

Conclusions:

• Zero-shot LLMs can identify GOC documentation comparably to trained NLP models.
• LLMs offer a promising, data-efficient approach for measuring clinical research outcomes.
• This validates LLMs for novel outcome measurement in palliative care research.