Social Desirability Bias in Health Screening Questionnaire Responses Generated by OpenAI Models: Quantitative Experimental Study

Social desirability bias, the tendency to respond in ways that present the respondent favorably, is well documented in human self-report and has recently been observed in large language models (LLMs) completing personality inventories. Whether it also affects LLM responses to validated health screening instruments, where LLMs are increasingly proposed as simulated patients or synthetic respondents, has not been examined.

This formative study aimed to determine whether 4 OpenAI models alter their responses to health screening questionnaires when the instrument and its evaluative purpose can be inferred from the prompt and to quantify the size and direction of any shift.

GPT-4o, GPT-4o mini, o1-mini, and o1 were prompted through the OpenAI application programming interface in April 2025 to complete the Generalized Anxiety Disorder-7 (GAD-7), Patient Health Questionnaire-9 (PHQ-9), Alcohol Use Disorders Identification Test (AUDIT), and FANTASTIC lifestyle assessment while adopting the persona of a random member of the public. Each instrument was administered under a naive condition, wherein every item was presented alone in a separate call so that the instrument could not be recognized, and an informed condition, in which all items were presented together, with 100 completed trials per model, instrument, and condition. Naive and informed scores were compared within each model using independent-samples 2-tailed t tests with Bonferroni correction for 16 comparisons, with effect sizes expressed as Cohen d and in units of published general-population SDs. Published human norms served only as descriptive reference points.

In 14 of 16 model-instrument combinations, scores shifted significantly toward the socially desirable pole when the instrument was recognizable (all P<.001): fewer anxiety and depressive symptoms, lower reported alcohol use, and healthier lifestyle behaviors. Mean shifts were 3.2 to 8.2 points on the GAD-7, 2.4 to 11.9 on the PHQ-9, 2.8 to 4.5 on the AUDIT for the o1 models, and 8.2 to 13.5 on the FANTASTIC lifestyle assessment (Cohen d=1.0-8.6). GPT-4o's AUDIT shift (0.36 points; P=.04) did not survive correction, and GPT-4o mini shifted in the opposite direction on the AUDIT (-2.69 points; P<.001). The largest shifts on the mental health instruments occurred in GPT-4o mini, whereas the largest AUDIT shifts occurred in the 2 reasoning models.

When a screening instrument was recognizable, the 4 OpenAI models tested consistently produced more socially desirable responses than when the same items were presented in isolation. The design cannot separate evaluative inference from other consequences of presenting items together, and the findings are confined to one vendor's models. Therefore, they constitute formative evidence that social desirability bias may extend to health screening contexts-not evidence of a general property of LLMs. Studies using LLMs as simulated patients or synthetic respondents should report prompt framing and audit for framing-dependent shifts before relying on generated data.