Clinician Evaluation of AI-Generated Responses to Girls' Puberty-Related Questions

Adolescents often look online for answers to sensitive questions about puberty and sexual health. As chat-based AI tools become more accessible, they may influence how young people interpret symptoms and decide whether to consult a doctor. We examined whether GPT-4o responses to common puberty-related questions from girls aged 8-17 years were judged by clinicians to be acceptable and appropriate for the user's age.

Ten common puberty-related queries from a Polish search context were identified using Google Autocomplete in May 2024 and converted into standardized first-person prompts beginning with "I am X years old and…". GPT-4o generated one response to each prompt in separate new chat sessions without manual editing. Eighteen clinicians (11 pediatric and adolescent gynecologists and 7 pediatricians) rated each response for content quality, adequacy of recommendations, empathy, and age appropriateness on 5-point scales. The main outcome was the proportion of ratings considered acceptable (scores of 4 or 5).

Overall, 79.2% of ratings (570/720; 95% CI: 76.0-82.1) were in the acceptable range, exceeding the predefined threshold. Across domains, acceptability ranged from 72.2% to 84.4%. Internal consistency was high for empathy and age appropriateness. Agreement between individual raters was low but improved to a moderate level when scores were averaged. Greater concern about adolescents placing too much trust in AI was strongly associated with the expectation that its use could reduce contact with physicians (Spearman's ρ = 0.86; p < .001; n = 18).

Most GPT-4o responses to common puberty-related questions were judged acceptable by clinicians. At the same time, concerns persisted that young users might rely too heavily on AI and delay seeking medical advice. Any use of such tools by adolescents should therefore include clear advice on when in-person medical assessment is needed.