Agreement Between Artificial Intelligence Platforms and the UpToDate Lexidrug Database in Identifying...

This study aimed to compare the agreement and consistency of classifications generated by ChatGPT-4.5 and Meta AI (Llama 4 Scout) with those of the UpToDate Lexidrug reference standard for identifying potential drug interactions involving antihypertensive medications in older adults.

A cross-sectional study was conducted using electronic medical record data from January to December 2024. Sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV) were calculated to assess agreement between classifications generated by ChatGPT-4.5 or Meta AI and those of UpToDate Lexidrug as the reference standard. In addition, Cohen kappa values were used to assess inter-rater agreement regarding the severity of the identified drug interactions.

ChatGPT-4.5 showed greater agreement with the reference standard (UpToDate Lexidrug), with an accuracy of 0.804, sensitivity of 0.862, specificity of 0.779, PPV of 0.628, and NPV of 0.929, compared with Meta AI, which had an accuracy of 0.706, sensitivity of 0.750, specificity of 0.686, PPV of 0.509, and NPV of 0.863. Cohen kappa analysis showed fair agreement between both platforms and the reference standard (0.367 and 0.359, respectively).

ChatGPT-4.5 demonstrated greater alignment with the UpToDate Lexidrug classification compared to Meta AI across all evaluated metrics. However, both artificial intelligence platforms showed only fair agreement in severity classification, indicating substantial differences between the classifications generated by the platforms and those of the reference database.