Using Large Language Models for Efficient Cancer Registry Coding in the Real Hospital Setting: A Feasibility Study.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Cancer Registry Management

Background:

• Manual cancer case reporting is labor-intensive and time-consuming.
• Existing rule-based and custom-supervised models have limitations in real-world workflows.
• The need for efficient and accurate cancer registry coding is critical.

Purpose of the Study:

• To evaluate the feasibility of using publicly available large language models (LLMs) for cancer registry coding.
• To develop and assess an agentic retrieval-augmented generation (RAG) system for this purpose.
• To explore the impact of prompt engineering on LLM performance in cancer coding.

Main Methods:

• Developed an agentic RAG system using publicly available LLMs for lung cancer case coding.
• Evaluated the system on a dataset of patient medical reports.
• Employed prompt engineering techniques, including chain of thought (CoT) reasoning and coding item grouping.

Main Results:

• Direct application of off-the-shelf LLMs is feasible for cancer registry coding.
• Prompt engineering significantly enhanced LLM capabilities, improving the macro-averaged F-score by 0.187.
• The developed system achieved a macro-averaged F-score of 0.637, outperforming baseline methods.

Conclusions:

• LLMs, particularly with prompt engineering, show significant potential for improving cancer registry coding.
• The proposed RAG system offers a promising reference tool for cancer registrars.
• This approach can enhance the efficiency and accuracy of cancer case reporting workflows.