Large language models can accurately populate Vascular Quality Initiative procedural databases using narrative...

Area of Science:

• Vascular surgery data management
• Artificial intelligence in healthcare
• Health informatics

Background:

• Vascular Quality Initiative (VQI) participation offers valuable resources but is often hindered by time and personnel constraints for data entry.
• Large language models (LLMs) demonstrate potential in natural language processing and text generation, offering a solution to data entry challenges.

Purpose of the Study:

• To evaluate the accuracy of LLMs in populating VQI procedural databases using operative reports.
• To assess the feasibility of using generative AI to streamline data entry for vascular surgery quality initiatives.

Main Methods:

• A retrospective study analyzed 150 operative reports for carotid endarterectomy (CEA), endovascular aneurysm repair (EVAR), and infrainguinal lower extremity bypass (LEB) procedures.
• A HIPAA-compliant LLM (Versa, based on ChatGPT) was used to automatically extract and populate VQI data from reports.
• The accuracy of two models, gpt-35-turbo and gpt-4, was compared against existing VQI data, with a metric defined as 'unavailable' if discussed in less than 20% of reports.

Main Results:

• The gpt-35-turbo model achieved median accuracy rates of 84.0% for CEA, 92.2% for EVAR, and 84.3% for LEB.
• Excluding routinely unavailable metrics, accuracy increased to 95.5% for CEA, 94.8% for EVAR, and 93.2% for LEB.
• Gpt-4 did not significantly improve performance over gpt-35-turbo, and processing costs were minimal ($0.12 for gpt-35-turbo vs. $3.39 for gpt-4 for 150 reports).

Conclusions:

• LLMs can accurately populate VQI databases with structured and unstructured data at a low cost.
• Increased workflow efficiency through LLMs may enhance a center's ability to participate in the VQI.
• Further research is warranted to explore other VQI databases and improve LLM accuracy for surgical data management.