Evaluating the Ability of Open-Source Artificial Intelligence to Predict Accepting-Journal Impact Factor and...

Area of Science:

• Bibliometrics and scientometrics
• Artificial intelligence in scientific publishing
• Medical informatics

Background:

• Selecting appropriate target journals is crucial for timely dissemination of research findings.
• Machine learning (ML) algorithms are increasingly utilized in recommender systems for academic journal submissions.

Purpose of the Study:

• To evaluate the performance of open-source artificial intelligence (AI) in predicting journal impact factor (IF) and Eigenfactor score (ES) tertiles.
• To assess AI model accuracy using academic article abstracts as input.

Main Methods:

• Collected PubMed-indexed articles (2016-2021) in ophthalmology, radiology, and neurology, including titles, abstracts, authors, and MeSH terms.
• Utilized Bidirectional Encoder Representations from Transformers (BERT) for abstract preprocessing and analysis, alongside logistic regression and XGBoost models.
• Trained and tested models on a 3:1 ratio to predict publication in the top, middle, or bottom tertile of journals based on IF and ES.

Main Results:

• The BERT model achieved the highest accuracy in predicting impact factor tertiles (75.0%) and Eigenfactor score tertiles (73.6%).
• XGBoost and logistic regression models showed lower, yet significant, predictive accuracies for both metrics.
• The study included 10,813 articles from 382 unique journals.

Conclusions:

• Open-source AI, particularly BERT, demonstrates strong capability in predicting the impact factor and Eigenfactor score of target journals.
• These AI-driven recommender systems hold potential for optimizing journal selection in academic publishing.
• Further research is needed to evaluate the impact of these systems on publication success rates and time-to-publication.