Prompt Engineering for Eastern Cooperative Oncology Group Status Extraction: Comparing Large Language Model...

Area of Science:

• Natural Language Processing (NLP) in Oncology
• Clinical Data Extraction
• Artificial Intelligence in Healthcare

Background:

• Eastern Cooperative Oncology Group (ECOG) performance status is vital for cancer patient management.
• ECOG status is frequently documented in unstructured clinical notes, posing extraction challenges.
• Current methods for extracting ECOG status from clinical text are often limited.

Purpose of the Study:

• To compare various approaches for extracting ECOG performance status from unstructured oncology clinical notes.
• To evaluate the effectiveness of advanced prompting techniques for large language models (LLMs) in this task.
• To assess the generalizability of these methods across different cancer types.

Main Methods:

• Evaluated four ECOG extraction methods: rule-based NLP, simple LLM prompting, Chain-of-Thought (CoT), and Double Filtering (DFT).
• Utilized unstructured clinical notes from non-small cell lung cancer, multiple myeloma, and ovarian cancer patients (2017-2021).
• Assessed performance using binary and three-class outcomes, and the QUEST questionnaire for human evaluation.

Main Results:

• Both CoT and DFT achieved 94% accuracy, surpassing rule-based (91%) and simple prompting (86%).
• DFT demonstrated the highest specificity (0.91) and PPV (0.93); CoT achieved the highest sensitivity (0.98).
• DFT and CoT showed superior output quality, reasoning, bias reduction, and user satisfaction, with DFT receiving the highest rating.

Conclusions:

• Advanced LLM prompting techniques (DFT, CoT) significantly enhance ECOG status extraction accuracy and reliability.
• These methods can standardize ECOG documentation, facilitate patient cohort identification, and support personalized treatment planning.
• Implementation requires consideration of computational costs and the necessity of human oversight.