A Study of Classification Methods for Structural Changes in Japanese Medical Institutions Using Generative AI

Area of Science:

• Health Informatics
• Artificial Intelligence
• Medical Administration

Background:

• Longitudinal analysis of Japanese medical institutions is hindered by frequent structural changes and institution code updates.
• Accurate classification of these changes is crucial for reliable health services research and policy-making.

Purpose of the Study:

• To evaluate the effectiveness of generative AI models in classifying the reasons behind structural changes in Japanese medical institutions.
• To assess the impact of Retrieval-Augmented Generation (RAG) and Chain-of-Thought (COT) prompting on classification accuracy.

Main Methods:

• Utilized Ministry of Health data from 2020-2024, focusing on cases involving institutional code changes.
• Employed generative AI models, including gpt-4o mini, and enhanced them with Google Search API-based RAG and COT prompting.
• Assessed classification accuracy for different change categories: Merged, Relocate, New, Organizational Change, and Closed.

Main Results:

• The baseline gpt-4o mini model achieved an accuracy of 0.307.
• Implementing Google Search API-based RAG significantly improved accuracy to 0.573.
• Chain-of-Thought prompting further boosted accuracy to 0.601, with 'Merged' and 'Relocate' categories classified more effectively than 'New,' 'Organizational Change,' or 'Closed.'

Conclusions:

• Integrating generative AI with external web information via RAG shows promise for classifying medical institution structural changes.
• While current accuracy is insufficient for full automation, the approach offers a foundation for future improvements.
• Further model refinement is necessary to accurately classify all types of institutional changes, particularly 'New,' 'Organizational Change,' and 'Closed' events.