Performance Evaluation of 18 Generative AI Models (ChatGPT, Gemini, Claude, and Perplexity) in 2024 Japanese...

Area of Science:

• Artificial Intelligence in Healthcare
• Pharmacy Education Technology
• Large Language Models

Background:

• Generative artificial intelligence (AI) is increasingly applied in healthcare.
• Previous studies assessed AI on medical exams, but pharmacy licensing exams are less explored.
• Pharmacists require diverse knowledge, necessitating AI evaluation in this field.

Purpose of the Study:

• To evaluate the performance of 18 new online chat-based large language models (OC-LLMs) on the 107th Japanese National License Examination for Pharmacists (JNLEP).
• To compare the accuracy of 2024 OC-LLMs against earlier models and identify areas for improvement.

Main Methods:

• The 107th JNLEP (345 questions) served as the benchmark.
• OC-LLMs were prompted with original Japanese questions; image uploads were used where permitted.
• Accuracy was calculated by subject area and question type, with Fleiss' κ measuring consistency.

Main Results:

• Four leading models (ChatGPT o1, Gemini 2.0 Flash, Claude 3.5 Sonnet, Perplexity Pro) achieved >80% accuracy, exceeding the passing threshold.
• Significant accuracy improvements were observed in text-only and diagram-based questions compared to earlier models.
• Accuracy for chemistry-related and chemical structure questions remained lower, with moderate consistency among top models (κ=0.334).

Conclusions:

• Online chat-based large language models (OC-LLMs) demonstrate significantly improved capabilities for pharmacy licensing examination content.
• Despite high accuracy (>80%), the remaining error rate necessitates continued human oversight in clinical practice.
• The 107th JNLEP provides a crucial benchmark for ongoing generative AI performance evaluations in pharmacy.