Exploring the Intersection of Artificial Intelligence and Neurosurgery: Let us be Cautious With ChatGPT

ChatGPT is a novel natural language processing artificial intelligence (AI) module where users enter any question or command and receive a single text response within seconds. As AI becomes more accessible, patients may begin to use it as a resource for medical information and advice. This is the first study to assess the neurosurgical information that is provided by ChatGPT.

ChatGPT was accessed in January 2023, and prompts were created requesting treatment information for 40 common neurosurgical conditions. Quantitative characteristics were collected, and four independent reviewers evaluated the responses using the DISCERN tool. Prompts were compared against the American Association of Neurological Surgeons (AANS) "For Patients" webpages.

ChatGPT returned text organized in paragraph and bullet-point lists. ChatGPT responses were shorter (mean 270.1 ± 41.9 words; AANS webpage 1634.5 ± 891.3 words) but more difficult to read (mean Flesch-Kincaid score 32.4 ± 6.7; AANS webpage 37.1 ± 7.0). ChatGPT output was found to be of "fair" quality (mean DISCERN score 44.2 ± 4.1) and significantly inferior to the "good" overall quality of the AANS patient website (57.7 ± 4.4). ChatGPT was poor in providing references/resources and describing treatment risks. ChatGPT provided 177 references, of which 68.9% were inaccurate and 33.9% were completely falsified.

ChatGPT is an adaptive resource for neurosurgical information but has shortcomings that limit the quality of its responses, including poor readability, lack of references, and failure to fully describe treatment options. Hence, patients and providers should remain wary of the provided content. As ChatGPT or other AI search algorithms continue to improve, they may become a reliable alternative for medical information.