Global variations in artificial intelligence-generated information on juvenile idiopathic arthritis

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Rheumatology

Background:

• Large Language Models (LLMs) are increasingly used for medical information retrieval.
• Assessing the accuracy and regional variations of LLM-generated content is crucial for clinical practice.
• Oligoarticular juvenile idiopathic arthritis (oJIA) requires accurate and context-specific treatment information.

Purpose of the Study:

• To evaluate similarities and variations in LLM responses regarding oJIA across different global regions.
• To analyze the readability, terminology distinctiveness, and adherence to clinical recommendations of LLM-generated content.

Main Methods:

• Utilized ten PICO questions based on 2021 ACR oJIA treatment guidelines.
• Prompted ChatGPT 4o in English from five countries: Canada, India, Italy, Kenya, and Türkiye.
• Assessed readability (FRES), term distinctiveness (TF-IDF), and co-occurrence networks (CONs).
• Expert review using a Likert-like scale evaluated adherence to recommendations.

Main Results:

• Responses were difficult to read (median FRES 30).
• Adherence rates varied by expert (52%-84%) but were similar across countries.
• Inter-rater agreement on adherence was weak (Kappa < 0.40).
• Terminology distinctiveness varied by country (0.60-0.85).
• Regional differences in focus were observed (e.g., Italy on corticosteroids, Kenya on outcomes).

Conclusions:

• LLM-generated medical information requires critical appraisal in clinical settings.
• Regional variations in LLM content necessitate careful consideration.
• Standardized methods for evaluating AI-generated medical information are needed.