Semantic Clinical Artificial Intelligence (SCAI) Usability Testing.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Human-Computer Interaction

Background:

• Traditional information retrieval methods, such as internet searches, can be time-consuming and may yield variable results.
• Large language models (LLMs) are emerging as powerful tools for synthesizing and presenting complex information.
• Evaluating the usability of specialized LLMs in clinical contexts is crucial for their adoption.

Purpose of the Study:

• To assess the usability and performance of Semantic Clinical Artificial Intelligence (SCAI), an LLM knowledge resource.
• To compare the efficiency and user-friendliness of SCAI against traditional internet (Web) searches for information retrieval.
• To determine SCAI's value as an innovative tool for delivering end-user information.

Main Methods:

• A pretest-intervention-posttest mixed-methods usability study design was employed.
• Cognitive walkthrough approach via Zoom was used for usability testing sessions.
• Participants completed information retrieval tasks using SCAI and the Web, followed by feedback sessions.

Main Results:

• SCAI demonstrated potential for more comprehensive and user-friendly information delivery compared to Web searches.
• Usability testing focused on efficiency and the ability to achieve accurate, reliable outcomes.
• User feedback was collected to rate satisfaction and overall usability of SCAI.

Conclusions:

• SCAI shows promise as an efficient and user-friendly tool for accessing clinical information.
• The study highlights the potential of LLMs to enhance information packaging and presentation in healthcare.
• Further evaluation is warranted to fully establish SCAI's role in clinical decision-making support.