Related Experiment Video
Updated: Jul 4, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Validation of a Zero-shot Learning Natural Language Processing Tool to Facilitate Data Abstraction for Urologic
Basil Kaufmann1, Dallin Busby2, Chandan Krushna Das2
1Milton and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai, New York, NY, USA; Department of Urology, University Hospital Zurich, University of Zurich, Zurich, Switzerland.
A new zero-shot learning natural language processing (NLP) tool significantly speeds up data abstraction from urologic electronic health records. This AI tool demonstrates high accuracy, offering a generalizable solution for researchers needing to extract information from unstructured text.
Area of Science:
- Urologic research
- Medical informatics
- Artificial intelligence in healthcare
Background:
- Urologic research relies on data extraction from unstructured electronic health records (EHRs).
- Existing natural language processing (NLP) tools often require task-specific training, limiting their generalizability.
- Efficient and accurate data abstraction is crucial for advancing urologic research.
Purpose of the Study:
- To develop and validate a zero-shot learning NLP tool for urologic data abstraction.
- To assess the tool's performance against human abstractors in terms of speed and accuracy.
- To provide a generalizable AI solution for extracting information from unstructured EHR text.
Main Methods:
- Development of a zero-shot learning NLP tool utilizing OpenAI's GPT-3.5 model.
- Comparison of the NLP tool with three physicians for abstracting 14 variables from 199 radical prostatectomy pathology reports.
- Evaluation of performance on both vectorized and scanned (OCR) report formats to assess optical character recognition impact.
Main Results:
- The NLP tool abstracted data significantly faster than human abstractors (12-15 seconds vs. 93 seconds per report, p < 0.001).
- The tool achieved high accuracy (94.2% for vectorized, 88.7% for scanned reports), demonstrating noninferiority to human abstractors.
- Accuracy for scanned reports was slightly lower but still noninferior to two of three human abstractors.
Conclusions:
- The developed zero-shot learning NLP tool provides a highly generalizable and accurate method for urologic data abstraction.
- This AI tool can significantly accelerate research by automating data extraction from unstructured EHR text.
- An open-access version is available, enabling immediate use by the urologic research community.
