The Quality of Information Produced by ChatGPT About Conditions Managed by Interventional Radiologists

The emergence of search engines powered by artificial intelligence and large language models (LLMs), such as ChatGPT, provides easy access to seemingly accurate health information. However, the accuracy of the information produced is uncertain. The purpose of this research is to assess the quality of information produced by ChatGPT about the treatment of health conditions commonly managed by Interventional Radiologists (IRs).

ChatGPT was asked "what is the best treatment" in relation to six conditions commonly managed by IRs. The output statements were assessed using the DISCERN instrument and compared against the current evidence base for the management of those conditions.

Six conditions were assessed. The mean overall score for the ChatGPT output statements was 1.3 compared to 3.8 for the reference articles. This poor performance by ChatGPT is largely attributable to the lack of transparency regarding sources. Although ChatGPT performed well in some areas such as presenting information in an unbiased manner, it showed significant weaknesses regarding source materials, the risks and benefits of each treatment, and the treatment's mechanism of action.

LLMs signify a considerable shift in how patients obtain and consume medical information. Understanding the strengths and weaknesses of ChatGPT's outputs regarding conditions commonly treated by IRs will enable tailored messaging and constructive discussions with patients in consultation with their IR.