CeRTS: certainty retrieval token search in large language model clinical information extraction.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Clinical Informatics

Background:

• Large language models (LLMs) require robust uncertainty estimation for clinical applications, particularly in extracting information from electronic health records.
• Existing token-level uncertainty estimators are limited as they only consider probabilities within a single output sequence.

Purpose of the Study:

• To develop and evaluate Certainty Retrieval Token Search (CeRTS), a novel uncertainty estimator for structured information extraction using LLMs.
• To leverage JSON output structure constraints to consider all likely sequences and their probabilities for a more accurate measure of model confidence.

Main Methods:

• CeRTS was evaluated against a gold-standard uncertainty estimator on eight open-source LLMs.
• Clinical features were extracted from lung cancer discharge summaries.
• Performance was quantified using calibration (Brier score) and discrimination (AUROC).

Main Results:

• CeRTS demonstrated superior discriminatory power compared to the previous estimator across all evaluated LLMs.
• CeRTS achieved better calibration in most cases, indicating improved reliability.
• The strongest agreement between model confidence and accuracy was observed with the Qwen-2.5 model.

Conclusions:

• CeRTS provides well-calibrated confidence scores for LLM-based information extraction from clinical text, offering researchers a quantitative reliability measure.
• While generally robust, CeRTS faced challenges with the DeepSeek-R1 model, potentially due to its Chain-of-Thought reasoning.
• The CeRTS method is applicable to any domain necessitating reliable uncertainty estimation beyond clinical data.