Related Experiment Video
Updated: Jul 3, 2025
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Patients with floaters: Answers from virtual assistants and large language models
Gloria Wu1, Weichen Zhao2, Adrial Wong2
1Department of Ophthalmology, University of California San Francisco School of Medicine, San Francisco, California, USA.
Objectives:
"Floaters," a common complaint among patients of all ages, was used as a query term because it affects 30% of all people searching for eye care. The American Academy of Ophthalmology website's "floaters" section was used as a source for questions and answers (www.aao.org). Floaters is a visual obstruction that moves with the movement of the eye. They can be associated with retinal detachment, which can lead to vision loss. With the advent of large language model (LLM) chatbots ChatGPT, Bard versus virtual assistants (VA), Google Assistant, and Alexa, we analyzed their responses to "floaters."
Methods:
Using AAO.org, "Public & Patients," and its related subsection, "EyeHealth A-Z": Floaters and Flashes link, we asked four questions: (1) What are floaters? (2) What are flashes? (3) Flashes and Migraines? (4) Floaters and Flashes Treatment? to ChatGPT, Bard, Google Assistant, and Alexa. The American Academy of Ophthalmology (AAO) keywords were identified if they were highlighted. The "Flesch-Kincaid Grade Level" formula approved by the U.S. Department of Education, was used to evaluate the reading comprehension level for the responses.
Results:
Of the chatbots and virtual assistants, Google Assistant is the only one that uses the term "ophthalmologist." There is no mention of the urgency or emergency nature of floaters. AAO.org shows a lower reading level vs the LLMs and VA (p = .11). The reading comprehension levels of ChatGPT, Bard, Google Assistant, and Alexa are higher (12.3, 9.7, 13.1, 8.1 grade) vs the AAO.org (7.3 grade). There is a higher word count for LLMs vs VA (p < .0286).
Conclusion:
Currently, ChatGPT, Bard, Google Assistant, and Alexa are similar. Factual information is present but all miss the urgency of the diagnosis of a retinal detachment. Translational relevance: Both the LLM and virtual assistants are free and our patients will use them to obtain "floaters" information. There may be errors of omission with ChatGPT and a lack of urgency to seek a physician's care.
