Potential of ChatGPT in facilitating research in radiation oncology?

To evaluate the potential of the artificial intelligence (AI) chatbot ChatGPT in supporting young clinical scientists with scientific tasks in radio oncological research.

Seven scientific tasks were to be completed in 3 h by 8 radiation oncologists with different scientific experience working at a university hospital: creation of a scientific synopsis, creation of a research question and corresponding clinical trial hypotheses, writing of the first paragraph of a manuscript introduction, clinical trial sample size calculation, and clinical data analyses (multivariate analysis, boxplot and survival curve). No participant had prior experience with an AI chatbot. All participants were instructed in ChatGPT v3.5 and its use was provided for all tasks. Answers were scored independently by two blinded experts. The subjective value of ChatGPT was rated by each participant. Data were analyzed with regression-, t-test and Spearman correlation (p < 0.05).

Participants completed tasks 1-3 with an average score of 50% and 4-7 with 56%. Scientific experience, number of original publications and of first/last authorships showed a positive correlation with overall scoring (p = 0.01-0.04). Participants with little to moderate scientific experience scored ChatGPT to be more helpful in solving tasks 4-7 compared to more experienced participants (p = 0.04), with simultaneously presenting lower scorings (p = 0.03).

ChatGPT did not compensate for differences in scientific experience of young clinical scientists, with less experienced researchers believing false AI-generated scientific results.