Is ChatGPT Ready for Public Use in Organ-Specific Drug Toxicity Research?

Area of Science:

• Artificial Intelligence in Medicine
• Pharmacovigilance
• Computational Toxicology

Background:

• Large language models (LLMs) like ChatGPT are increasingly used, raising concerns about their reliability in public health.
• Accurate drug toxicity assessment is critical for patient safety and regulatory oversight.

Purpose of the Study:

• To evaluate the accuracy of GPT-4 in assessing drug toxicity for liver, heart, and kidney functions.
• To compare the performance of general versus expert prompts for GPT-4 drug toxicity assessments.

Main Methods:

• GPT-4 was prompted using two distinct approaches: a 'General prompt' and an 'Expert prompt'.
• Assessments were benchmarked against expert evaluations derived from US Food and Drug Administration (FDA) drug-labeling documents.
• Drug toxicity for liver, heart, and kidney endpoints was evaluated.

Main Results:

• The 'Expert prompt' demonstrated higher accuracy (64-75%) compared to the 'General prompt' (48-72%).
• Overall performance for GPT-4 in drug toxicity assessment was moderate, suggesting limitations for direct public health application.
• GPT-4's accuracy varied across different organ systems.

Conclusions:

• GPT-4 shows potential but requires cautious application in public health due to moderate accuracy.
• Enhancements, such as Retrieval Augmented Generation (RAG), may be necessary to improve the reliability of LLMs for drug toxicity assessments.
• Further research is needed to optimize LLM frameworks for sensitive public health tasks.