Comparing the Accuracy of Deidentification in Japanese Text Using Large Language Models

The purpose of this study is to evaluate the accuracy of personal information extraction using large language models (LLMs) to develop an deidentification tool for in-house use. We used three LLMs (BERT, GPT3.5 and GPT4o-mini) to target the extraction of personal names, facility names, and place names. As a pilot study, we analyzed 20 Japanese newspaper articles containing these three types of information and assessed the extraction accuracy. The results showed that BERT achieved the highest accuracy, with F1-scores of 0.94 overall, 0.99 for personal names, 0.89 for facility names, and 0.93 for place names. Therefore, BERT is the most useful for the automatic extraction of personal information from clinical data.