Expert assignment system based on natural language processing for Marie Sklodowska-Curie actions

Area of Science:

• Research Evaluation
• Artificial Intelligence in Science

Background:

• Traditional expert assignment relies on keyword matching, often requiring manual correction by Vice Chairs (VCs).
• Existing systems like the Single Evaluation Platform (SEP) have limitations in semantic relevance and accuracy.

Purpose of the Study:

• To develop and evaluate a novel expert assignment system using Natural Language Processing (NLP) and Large Language Models (LLMs).
• To enhance semantic relevance in expert-proposal matching beyond simple keyword correlation.

Main Methods:

• Integrated dynamic retrieval of expert publications via ORCID with GALACTICA, a scientific LLM.
• Computed fine-grained semantic similarity between proposal abstracts and expert publications.
• Evaluated three similarity aggregation strategies: Sum, Product, and Maximum.

Main Results:

• The Maximum similarity aggregation strategy achieved an AUC of 0.82, outperforming the traditional SEP system (AUC = 0.75).
• The LLM-based system significantly outperformed Sum (AUC = 0.69) and Product (AUC = 0.57) strategies.
• The Maximum approach closely replicated VC-reviewed assignments, indicating effective capture of human decision-making.

Conclusions:

• LLM-based semantic matching offers a more accurate and scalable alternative to traditional Information Retrieval (IR) systems for expert assignment.
• The Maximum similarity strategy effectively identifies the most relevant expert matches, mirroring human judgment.
• This approach provides nuanced, fine-grained ratings for better expert differentiation, surpassing discrete affinity scores.