Scalable scientific interest profiling using large language models

Area of Science:

• Biomedical informatics
• Artificial intelligence in research

Background:

• Scientific research profiles are crucial for talent discovery and collaboration.
• Existing profiles are often outdated, necessitating automated and scalable solutions.
• Large Language Models (LLMs) offer a potential solution for dynamic profile generation.

Purpose of the Study:

• To design and evaluate LLM-based methods for generating scientific interest profiles.
• To compare machine-generated profiles with researchers' self-summarized interests.
• To assess the performance of profiles generated from PubMed abstracts versus Medical Subject Headings (MeSH) terms.

Main Methods:

• Two LLM-based methods were developed: one summarizing researcher abstracts and another using MeSH terms.
• GPT-4o-mini was used to generate summaries for 595 researchers from Columbia University Irving Medical Center.
• Automated metrics (ROUGE-L, BLEU, METEOR, BERTScore, KL Divergence) and manual evaluations were employed for comparison.

Main Results:

• Automated metrics showed low lexical overlap but moderate semantic similarity (BERTScore F1: ~0.55) between machine-generated and human-written profiles.
• Manually paraphrased summaries achieved higher similarity (F1: 0.851).
• MeSH-based profiles demonstrated superior readability (93.44% favorable ratings) and were preferred in 67.86% of manual reviews, despite differences in keyword usage and factual accuracy compared to human-written profiles.

Conclusions:

• LLMs show promise for scalable automation of scientific interest profiling.
• MeSH-based LLM-generated profiles offer better readability than abstract-based ones.
• While LLMs can generate semantically similar profiles, human-written summaries tend to introduce more novel concepts.