Can ChatGPT identify predatory biomedical and dental journals? A cross-sectional content analysis

To assess whether ChatGPT can help to identify predatory biomedical and dental journals, analyze the content of its responses and compare the frequency of positive and negative indicators provided by ChatGPT concerning predatory and legitimate journals.

Four-hundred predatory and legitimate biomedical and dental journals were selected from four sources: Beall's list, unsolicited emails, the Web of Science (WOS) journal list and the Directory of Open Access Journals (DOAJ). ChatGPT was asked to determine journal legitimacy. Journals were classified into legitimate or predatory. Pearson's Chi-squared test and logistic regression were conducted. Two machine learning algorithms determined the most influential criteria on the correct classification of journals.

The data were categorized under 10 criteria with the most frequently coded criteria being the transparency of processes and policies. ChatGPT correctly classified predatory and legitimate journals in 92.5 % and 71 % of the sample, respectively. The accuracy of ChatGPT responses was 0.82. ChatGPT also demonstrated a high level of sensitivity (0.93). Additionally, the model exhibited a specificity of 0.71, accurately identifying true negatives. A highly significant association between ChatGPT verdicts and the classification based on known sources was observed (P <0.001). ChatGPT was 30.2 times more likely to correctly classify a predatory journal (95 % confidence interval: 16.9-57.43, p-value: <0.001).

ChatGPT can accurately distinguish predatory and legitimate journals with a high level of accuracy. While some false positive (29 %) and false negative (7.5 %) results were observed, it may be reasonable to harness ChatGPT to assist with the identification of predatory journals.

ChatGPT may effectively distinguish between predatory and legitimate journals, with accuracy rates of 92.5 % and 71 %, respectively. The potential utility of large-scale language models in exposing predatory publications is worthy of further consideration.