Development and Preliminary Evaluation of the Patient-Artificial Intelligence Rating Instrument

Patients increasingly use artificial intelligence (AI) for health-related conversations, but evaluation measures largely reflect clinician judgment.

To develop and preliminarily evaluate a patient instrument assessing the quality of patient-AI health conversations.

Cross-sectional survey conducted June 5-14, 2026, following patient-partnered instrument development and cognitive interviews.

An anonymous online survey.

A convenience sample of adults with prior use of large language models for personal health-related queries, recruited from an academic medical center-affiliated Insight Panel and patient peers.

Participants' own prior health-related conversations with AI tools of their choosing.

The Patient-AI Rating instrument comprised 12 communication-quality and patient engagement items rated from 1 to 5 and a separately scored patient-perceived-harm item. Internal consistency was assessed using Cronbach α, with sensitivity analyses using first ratings only for those using the tool to rate multiple conversations and participant-mean profiles. We examined item-total correlations, inter-item structure, and between- and within-participant variation.

Fourteen participants contributed 46 complete ratings; among 13 providing demographics, median age was 63 years (IQR, 53-70 years), and 8 (62%) reported female gender. Participants provided a median of 2 ratings (range, 1-17). Cronbach α was 0.88 (95% CI, 0.83-0.93) across all ratings, 0.85 using first ratings only, and 0.92 using participant-mean profiles. Corrected item-total correlations ranged from 0.42 to 0.72; mean inter-item correlation was 0.40. No item deletion improved Cronbach α. The mean quality score was 4.43 (SD, 0.45). Among 9 participants contributing multiple ratings, the intraclass correlation coefficient was 0.45, suggesting inter-conversation variability. No perceived harm was reported in 44 of 46 ratings (96%); the harm item correlated minimally with the instrument total (r = 0.09).

In this small cross-sectional study of engaged patients, the Patient-AI Rating instrument showed promising internal consistency and variation across rated conversations. These findings support further evaluation of patient-reported communication quality alongside clinician-based measures. Validation in larger, more diverse populations, including assessment of construct and criterion validity, is needed before broader use.

Question: Can a patient instrument measure the quality of patients' health-related conversations with artificial intelligence (AI)?Findings: In this cross-sectional study, 14 participants provided 46 ratings of their health-related AI conversations using a 12-item scale and a perceived-harm item (the Patient AI Rating (PAIR) instrument). The patient metric showed good internal consistency, and scores varied both by raters and across conversations rated by the same participant.Meaning: The PAIR instrument offers a preliminary approach to incorporate patient perspective and experience into AI evaluation - a measure largely missing from current AI assessment; larger studies are needed to establish validity and generalizability.