A critical assessment of using ChatGPT for extracting structured data from clinical notes

Area of Science:

• Natural Language Processing (NLP)
• Artificial Intelligence in Medicine
• Computational Pathology

Background:

• Traditional NLP methods for clinical note analysis require extensive task-specific annotations and model training.
• Unstructured clinical data presents a significant barrier to large-scale research and efficient clinical decision-making.

Purpose of the Study:

• To evaluate the efficacy of ChatGPT, a large language model (LLM), in extracting structured information from free-text medical notes.
• To assess ChatGPT's performance against expert-curated data in pathology reports without task-specific training.

Main Methods:

• A novel LLM-based workflow using prompt engineering and OpenAI's API was developed.
• ChatGPT-3.5 was evaluated on datasets of lung cancer (over 1000 reports) and pediatric osteosarcoma (191 reports) pathology reports.
• Outputs were compared against expert-curated structured data to determine accuracy and identify misclassification patterns.

Main Results:

• ChatGPT-3.5 achieved 89% accuracy in pathological classification for lung cancer, surpassing traditional NLP methods.
• For pediatric osteosarcoma, ChatGPT-3.5 demonstrated high accuracy: 98.6% for grade and 100% for margin status.
• Misclassifications were primarily linked to specialized terminology and TNM staging interpretation; performance remained stable over time.

Conclusions:

• ChatGPT offers a feasible and efficient method for processing large volumes of clinical notes into structured data.
• This LLM-driven approach significantly reduces the need for manual annotation and model training, accelerating data utilization.
• The findings highlight the potential of LLMs to transform unstructured healthcare data, supporting medical research and clinical decision support systems.