Comparing the Accuracy of Deidentification in Japanese Text Using Large Language Models

Area of Science:

• Natural Language Processing
• Data Deidentification
• Machine Learning

Background:

• Accurate extraction of personal information is crucial for data privacy.
• Large Language Models (LLMs) show potential for automated information extraction.
• Developing effective deidentification tools is essential for handling sensitive data.

Purpose of the Study:

• To evaluate the accuracy of different LLMs in extracting personal, facility, and place names.
• To assess the feasibility of using LLMs for developing an in-house deidentification tool.
• To compare the performance of BERT, GPT3.5, and GPT4o-mini for named entity recognition.

Main Methods:

• Three LLMs (BERT, GPT3.5, GPT4o-mini) were employed for information extraction.
• A pilot study analyzed 20 Japanese newspaper articles.
• Extraction accuracy was measured using F1-scores for personal names, facility names, and place names.

Main Results:

• BERT achieved the highest overall F1-score of 0.94.
• BERT demonstrated superior performance in extracting personal names (0.99), facility names (0.89), and place names (0.93).
• GPT3.5 and GPT4o-mini showed lower accuracy compared to BERT.

Conclusions:

• BERT is the most effective LLM for the automatic extraction of personal information among the models tested.
• The findings support the use of BERT for developing robust deidentification tools for clinical data.
• Further research can explore BERT's application in diverse datasets for enhanced data privacy.