Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study

To determine whether large language models can compute the Charlson Comorbidity Index (CCI) from unstructured, German-language diagnosis lists with reliability approaching that of expert human consensus.

We retrospectively analyzed the complete free-text diagnosis lists of 200 patients from a tertiary orthopedic center, stratified by American Society of Anesthesiologists (ASA) physical status I-IV. Two board-certified orthopedic surgeons and 1 chiropractor independently assigned CCI values; a consensus reading served as the reference standard. Twelve open-source models, hosted on-premise, and 5 commercial models were each evaluated with and without guideline-based prompting. Agreement was quantified by accuracy, Cohen kappa, and the coefficient of determination, and discrepancies were classified using an adapted error taxonomy.

Human interrater reliability was excellent (intraclass correlation 0.97). Across models, accuracy against consensus ranged from 0.34 to 0.79 and kappa from 0.27 to 0.75. The best open-source model (GPT-OSS-120B with guidelines) matched the best commercial model (coefficient of determination 0.93). Agreement declined with increasing ASA class, falling to 0.34-0.56 in ASA IV. Most discrepancies were knowledge-related (41.7%-58.3%); computation errors were rare, and 22.2%-32.0% arose from diagnosis lists too incomplete for any classification.

Performance approached expert level in low-complexity records but degraded as documentation complexity increased, indicating that record quality rather than model reasoning is the dominant constraint. Locally hosted open-source models performed comparably to proprietary systems, allowing comorbidity extraction without transferring data off-premise.

Large language models can approximate expert CCI scoring directly from unstructured German-language clinical text, supporting scalable risk adjustment where documentation is adequate.