Leveraging AI Large Language Models for Writing Clinical Trial Proposals in Dermatology: Instrument Validation Study.

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Research Operations
• Natural Language Processing

Background:

• Large language models (LLMs) are increasingly adopted in clinical trial design.
• Their application in research proposal development remains underutilized.
• This study evaluates LLMs' efficacy in research proposal creation and assessment.

Purpose of the Study:

• To compare the performance of open-access large language models (LLMs) against human experts in research proposal writing and review.
• To assess the accuracy and comprehensiveness of LLM-generated research proposals.

Main Methods:

• Ten large language models (LLMs) were tasked with generating research proposals.
• Six physicians and the LLMs themselves evaluated eleven blinded research proposals.
• Assessments focused on proposal accuracy and comprehensiveness.

Main Results:

• Human scorers identified ChatGPT-o1 as most accurate and Llama 3.1 as least accurate.
• LLM scorers found ChatGPT-o1 and DeepSeek R1 most accurate.
• ChatGPT-o1 and Llama 3.1 were rated most and least comprehensive, respectively, by both human and LLM scorers.
• LLMs generally overestimated proposal quality, scoring them 1.9 points higher than humans on average.

Conclusions:

• Paid versions of ChatGPT offer the highest quality and versatility among current LLMs for research proposal tasks.
• LLMs can function as valuable assistants, improving efficiency and productivity in research development.
• Expert human input remains indispensable, as LLMs cannot fully replace the nuanced judgment of experienced researchers.