Deep sight: enhancing periprocedural adverse event recording in endoscopy by structuring text documentation with...

Area of Science:

• Medical informatics
• Natural Language Processing
• Endoscopy

Background:

• Accurate assessment of adverse events in endoscopic procedures is crucial for patient safety and quality control.
• Current free-text documentation hinders scalable analysis of intervention outcomes.
• Large language models (LLMs) present a novel solution for structured adverse event extraction.

Purpose of the Study:

• To evaluate the efficacy of LLMs in automatically extracting structured adverse event data from endoscopy reports.
• To compare the performance of GPT-4 and Llama-2 models in identifying adverse events like bleeding, perforation, and aspiration.
• To assess the impact of prompt engineering on LLM performance for medical documentation.

Main Methods:

• Analysis of 672 endoscopy reports using OpenAI's GPT-4 and Llama-2 models.
• Development of an automated LLM pipeline for adverse event extraction into JavaScript Object Notation.
• Dataset split into a proof-of-concept set (n=171) for prompt engineering and an external test set (n=501).

Main Results:

• GPT-4 demonstrated high accuracy (e.g., 97% sensitivity, 92% specificity in PoC-S).
• Llama-2 models showed comparable, privacy-compliant performance (e.g., 94% sensitivity, 92% specificity in PoC-S).
• Prompt engineering significantly influenced model sensitivity and specificity.

Conclusions:

• LLMs provide an efficient and scalable method for extracting structured adverse event data from endoscopy reports.
• Automated extraction reduces manual effort and facilitates immediate quality reporting.
• Privacy-conscious Llama-2 models offer a practical alternative for real-world implementation.