Assessing the Capability of Large Language Models for Navigation of the Australian Health Care System: Comparative...

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Human-Computer Interaction

Background:

• Navigating healthcare systems in Australia, particularly in rural areas, presents significant challenges.
• Generative search tools, utilizing large language models (LLMs), offer potential for improved health information retrieval.
• Concerns exist regarding the accuracy and reliability of generative AI compared to traditional search engines in healthcare.

Purpose of the Study:

• To compare the effectiveness of generative artificial intelligence (AI) search (Microsoft Copilot) against conventional search engines (Google Web Search) for health information navigation.
• To assess the accuracy and user experience of AI-powered health information retrieval tools.

Main Methods:

• A web-based survey involving 97 adults in Queensland, Australia.
• Participants used either Microsoft Copilot or Google Web Search for scenario-based health care navigation questions.
• Accuracy was measured by correctness ratings and numerical scores; user experience was evaluated using the Technology Rating Questionnaire (TRQ).

Main Results:

• Microsoft Copilot demonstrated superior performance in 2 specific health care navigation tasks compared to Google Web Search.
• No significant difference in performance was observed for the remaining 6 tasks.
• Users rated Google Web Search higher for willingness to adopt and perceived quality of life impact, but lower for learning effort. Both tools received similar ratings for perceived value, confidence, and privacy concerns.

Conclusions:

• Generative AI tools can achieve accuracy comparable to traditional search engines for health care navigation.
• Despite comparable accuracy, generative AI did not enhance the overall user experience in this study.
• Further research is necessary to evaluate AI advancements and user adaptation to these technologies.