Paediatric rare diseases: Can large language models assist off-label prescribing?

To evaluate the effectiveness and reliability of large language models (LLMs) in retrieving and synthesizing biomedical information to support off-label drug prescribing in paediatric rare diseases, and to compare their performance with human-authored references in terms of scientific rationale, adverse events and drug interactions.

The study reviewed 20 cases of off-label prescriptions in rare paediatric diseases using 4 LLMs (i.e., GPT-4o, Sophos-2, Claude-3, Scopus AI). The queries addressed focused on scientific rationale, adverse events and drug interactions. The performance measures encompassed sensitivity, precision, accuracy, F1-score, response quality and reference quality. A Global Performance Score integrated all measures.

After evaluating 2758 references and 480 responses, a significant discrepancy was found among 4 LLMs concerning Global Performance Score (P = .001). Posthoc analysis showed that Scopus AI vs. GPT-4o comparison was significant, with GPT-4o showing higher values. Median LLM reference quality often surpassed human performance, yet variability limits conclusions regarding superiority.

LLMs are capable of retrieving and synthesizing biomedical information, but performance varies depending on query type and search mode. These tools speed up retrieving relevant information to assess off-label prescribing appropriateness. Despite the promise of artificial intelligence, human oversight remains critical to ensure data accuracy and reliability.