Evaluating OpenEvidence Across Pediatric Subspecialties: A Guideline-Anchored Expert Review

To evaluate the quality, safety, and citation characteristics of OpenEvidence across pediatric subspecialties.

Guideline-anchored clinical questions spanning high, moderate, and low evidence certainty across 6 pediatric subspecialties were submitted to OpenEvidence. Expert reviewer pairs independently scored responses using the validated Quality Analysis of Medical Artificial Intelligence (QAMAI) instrument (6-30 range, higher scores indicate better quality) and a modified Agency for Healthcare Research and Quality harm assessment (0-7 range, higher scores indicate more harm), followed by consensus reconciliation. Citation analysis examined overlap with guideline references, claim substantiation, and pediatric specificity. Response consistency was assessed across 5 repeated queries.

Seventeen questions were evaluated. The median QAMAI score was 27/30 (IQR, 24-29; "very good" quality). Potential harm was low (median 0/7, range 0-3), though 6 of 17 responses carried nonzero harm potential. OpenEvidence cited the primary guideline in 15 of 17 questions, yet only 17.6% of responses showed strong citation overlap (≥75% shared references) with the corresponding guidelines. OpenEvidence's citations reliably supported the claims they accompanied (88.2% with high substantiation), and 70.8% of reviewer assessments rated the evidence base as comparable in quality to that of the corresponding guidelines. In total, 35.3% of responses cited studies lacking pediatric specificity. Responses were clinically consistent across repeated queries (77.9% had no meaningful or only minor variation); no contradictory recommendations were observed.

OpenEvidence produced high-quality responses with low expert-rated potential for harm across multiple pediatric subspecialties. The pediatric specificity of cited evidence warrants attention, reflecting the broader challenge of applying artificial intelligence tools that are trained on adult-predominant literature to pediatric populations.