Automated Tumor International Classification of Diseases Coding of Real-World Pathology Reports Using Self-Hosted...

Area of Science:

• Medical Informatics
• Computational Pathology

Background:

• Manual coding of pathology reports using International Classification of Diseases for Oncology (ICD-O)-3 is inefficient and prone to errors.
• There is a need for automated solutions to improve the speed and accuracy of cancer coding.

Purpose of the Study:

• To evaluate the performance of state-of-the-art large language models (LLMs) in extracting ICD-O-3 topography and morphology codes.
• To assess the potential of LLMs for clinical implementation in pathology coding.

Main Methods:

• Analysis of 21,364 pathology reports from 10,823 patients.
• Evaluation of five LLMs (Llama-3.3-70B-Instruct, DeepSeek-R1-Distill-Llama, Qwen3-235B-A22B, Gemma-3-12B-it) on a secure hospital IT infrastructure.
• Development of three prompts for topography and morphology extraction, with performance measured by exact and three-position code matches.

Main Results:

• Qwen3-235B-A22B achieved the highest exact topography code prediction (F1: 71.6%).
• DeepSeek-R1-Distill-Llama-70B led in morphology code prediction (exact F1: 34.7%).
• Models struggled with rare conditions, indicating poor generalization.

Conclusions:

• LLMs show potential as assistive tools for expert-guided pathology coding.
• Current LLM performance is insufficient for fully automated, unsupervised clinical use.
• LLMs demonstrated dependence on context and lower accuracy for morphology compared to topography.