Four OpenAI models were used to quantify SDB by probing stigmatized conditions and lifestyle factors considered either virtuous or unhealthy. Specifically, Generalized Anxiety Disorder-7 (GAD-7) was used to screen for anxiety [12], Patient Health Questionnaire-9 (PHQ-9) for depression [13] (mental health), Alcohol Use Disorders Identification Test (AUDIT) for hazardous alcohol consumption [14], and FANTASTIC lifestyle assessment for lifestyle indicators often linked to poor health [15] (lifestyle risk). These tools were chosen because they address issues sensitive to SDB, and each consists of a limited number of items, making them practical for inclusion in this study.

The 4 OpenAI models included in this study were GPT-4o (a versatile, high-intelligence flagship model), GPT-4o mini (a fast, affordable small model for focused tasks), o1-mini (a smaller, faster, and more affordable reasoning model), and o1 (a larger reasoning model). This range of models enabled the quantification of response biases across model size and across the 2 model families available from this vendor at the time (the GPT-4o family and the o1 reasoning family). All data were collected through the OpenAI application programming interface in April 2025 using the model aliases gpt-4o, gpt-4o mini, o1-mini, and o1, which at that time resolved to the snapshots gpt-4o-2024-08-06, gpt-4o mini-2024-07-18, o1-mini-2024-09-12, and o1-2024-12-17, respectively. Default sampling parameters were used throughout; no temperature or other sampling parameters were set (the o1 models did not accept a temperature parameter, and the GPT-4o models therefore ran at their default temperature of 1.0). The findings therefore describe these specific model versions and are not claimed to generalize to models from other vendors.

Two distinct prompt formats were used to present the screening instruments to the models. Two distinct prompt formats were used to present the screening instruments to the models. First, questionnaire items were presented independently of one another (naive prompt), obscuring the purpose of the evaluation and measuring baseline or naive responses. Second, all items within a questionnaire were presented together (informed prompt). This format provided the model with a higher density of semantic cues, allowing it to infer the evaluative nature of the prompt and therefore measure an “informed” response. All prompts are presented in Multimedia Appendix 1.

There were 100 trials for each combination of prompt format and model. Because some questions related to mental ill-health, suicidal ideation, and self-harm, some models occasionally refused to provide a numerical response as requested in the prompt; these responses were excluded, and new ones were generated until 100 successful trials were achieved. To ensure that the LLM had no access to previous queries, a separate application programming interface call was initiated for each item.

Published general-population means and SDs for each instrument were used as descriptive reference points only and were obtained from large validation studies [16-19]:

No inferential comparison between model responses and these human norms was made. The norms derive from demographically specific samples, whereas the prompts specified only a “random member of the public,” and 100 trials from a single model do not constitute a sample from a comparable population; a statistical test of the difference would therefore not be interpretable. The human means are shown in Figure 1 [16-19] to orient the reader to the absolute level of the scores, and the pooled (gender-combined) human SDs are used solely as a scaling constant so that shifts can be compared across instruments with different ranges (Figure 2). The primary analysis was the within-model contrast between conditions. For each model and questionnaire, total scores from the “naive” and “informed” prompts were compared using an independent-samples 2-tailed t test. Because 16 such comparisons were made, a Bonferroni-corrected significance threshold of P<.003 (α=0.05/16) was applied. Effect sizes are reported as Cohen d (difference in means divided by the pooled within-condition SD) and, for cross-instrument comparison, as the difference in means divided by the published human SD for that instrument. Analyses were conducted in Microsoft Excel; the summary statistics, t tests, and effect sizes were independently reverified from the deposited dataset in Python (version 3.11; Python Software Foundation) and SciPy (version 1.16) during revision.

Because this study did not involve human participants, no institutional ethics approval was required.

Absolute scores are shown in Figure 1 and Table 1. In the naive condition, all 4 models returned mental health scores in the moderate to severe range (GAD-7: 13.0-15.8 of a possible 21; PHQ-9: 13.5-17.5 of 27), well above the published general-population means for these instruments [16,17]. AUDIT scores in the naive condition differed markedly by model family: the GPT-4o family returned scores of 2.5 to 3.0 (below the conventional hazardous drinking threshold of 8 and close to the published population mean [18]), whereas the o1 family returned scores of 11.7 to 13.1. FANTASTIC lifestyle assessment total scores in the naive condition ranged from 59.6 to 64.4 of 100, below the published mean of 73.9 [19]. These comparisons with published norms are descriptive; as set out in the Methods section, no inferential test against the norms was performed.

In 14 of the 16 model-questionnaire combinations, scores were significantly more socially desirable (Bonferroni-corrected P<.003) when the model was able to infer the evaluative nature of the prompt (Table 1). The 2 exceptions were both on AUDIT: GPT-4o showed a small shift in the socially desirable direction that did not survive correction (difference=0.36; P=.04; Cohen d=0.30), and GPT-4o mini shifted in the opposite direction, reporting more alcohol use in the informed condition (difference=–2.69; Cohen d=–1.59). Effect sizes for the 14 significant shifts were large (Cohen d=1.0-8.6), reflecting both the size of the mean differences and the low trial-to-trial variance of model responses.

To compare the size of the shift across instruments with different score ranges, differences are expressed in units of the published human SD for each instrument (Figure 2). The largest shifts on the mental health instruments were produced by GPT-4o mini (2.4 human SDs on the GAD-7 and 3.4 on the PHQ-9), with o1 producing the smallest (SD 0.9 and SD 0.7). The AUDIT pattern reversed: the 2 reasoning models produced the largest shifts (o1: SD 1.3; o1-mini: SD 0.8), GPT-4o produced a negligible shift (SD 0.1), and GPT-4o mini shifted in the opposite direction (SD –0.8). On FANTASTIC lifestyle assessment, all 4 models shifted by SD 0.9 to SD 1.4. Model size alone did not order the effects; the pattern was better described by the model family, with the o1 reasoning model showing smaller shifts on the mental health instruments and larger shifts on AUDIT than the other models.

The aim of this study was to investigate the presence and extent of SDB in 4 OpenAI language models when responding to health screening questionnaires. The findings indicate a consistent pattern: in 14 of 16 model-questionnaire combinations, responses shifted toward the socially desirable pole when the items were presented together, and the instrument could be recognized. Consistent with prior observations of SDB in personality psychometrics [4], the models reported fewer symptoms of depression and anxiety, less alcohol misuse, and more virtuous lifestyle behaviors under these conditions. The 2 exceptions were both on AUDIT, where GPT-4o showed a negligible shift and GPT-4o mini shifted in the opposite direction. The pattern parallels classic SDB in human respondents, wherein individuals alter their self-reports to align with perceived social expectations, although, as discussed below, the present design cannot establish that the same inferential mechanism is responsible. As anticipated in the Introduction section, the significance of this finding is application dependent: the same humanlike skew that may enhance fidelity in educational simulation directly threatens validity when LLMs are used as objective screeners or as proxies for data generation.

The direction and size of the shift varied by instrument and model family rather than simply by model size. The o1 reasoning model showed the smallest shifts on the mental health instruments and returned comparatively high absolute anxiety and depression scores in the informed condition, whereas it showed the largest shifts on AUDIT, moving from hazardous-range scores in the naive condition toward lower scores when the instrument was recognizable. The GPT-4o models showed the reverse: large shifts on the mental health instruments and small or reversed shifts on AUDIT, where their naive scores were already low. All 4 models are posttrained with reinforcement learning from human feedback, and the o1 models additionally use reinforcement learning to train chain-of-thought reasoning; the training data and reward models for all of them are proprietary. It is therefore not possible to attribute these family-level differences to specific training choices, and no mechanistic account is offered here. The observation that the direction of any framing effect depends on the instrument and model family is nonetheless practically important because it means that the effect cannot be corrected with a single global adjustment. Framing-dependent shifts in the socially desirable direction have also been reported for GPT-4 in survey-style settings [20], which suggests that the phenomenon is not confined to the model versions tested here, although it remains to be replicated across vendors.

The role of prompt length and context availability must also be considered as a potential contributor to the observed phenomenon. While the prompt sizes in this study were well within the context windows of the models tested, the difference in token density between the naive (single item) and informed (full questionnaire) conditions is nontrivial. The “Informed” prompts provided a richer semantic field, which may allow the model to attend to interitem correlations and identify the specific clinical instrument being administered. The resulting shift is most readily interpreted as the model inferring an evaluative context and adjusting its responses accordingly. However, this interpretation cannot be fully disentangled from other effects of the longer prompt, and at least three nonmutually exclusive mechanisms could contribute to the same effect: (1) recognition of the evaluative context, activating safety-alignment behaviors (the social desirability account); (2) the longer context anchoring the model’s output toward a more formal clinical or psychiatric register, independent of any evaluative inference; and (3) the aggregation of items with differing valences balancing the overall sentiment of the prompt and shifting responses toward the scale midpoint. A pattern in the present data bears on the third account. In the naive condition, mean per-item responses on the GAD-7 (1.9-2.3 on a 0-3 scale) and PHQ-9 (1.5-1.9) lay at or slightly above the scale midpoint for all models, so part of the naive signal on these instruments could reflect a context-free tendency to select a middle response rather than a considered report of symptoms. However, the AUDIT naive responses do not fit this description: the GPT-4o models responded near the floor of the scale and the o1 models well above the midpoint, and the shift between conditions was in opposite directions for the 2 families. Midpoint responding therefore cannot account for the pattern as a whole. Prior work also weighs against the latter 2 accounts as complete explanations: Salecha et al [4] found that explicitly signaling an evaluative context produced the socially desirable skew even for a single item, an effect comparable in magnitude to presenting multiple items together, which decouples the effect from prompt length. They also found that reverse-coding items roughly halved the bias, whereas positive-coding items left it essentially unchanged, indicating the shift is not merely an acquiescence or aggregate-sentiment artifact. These controls were established in a personality inventory rather than the clinical instruments used here, so they support the evaluative-inference account as the most parsimonious mechanism without confirming it for the present prompts. The present design, which contrasts single-item prompts with full-questionnaire prompts, cannot definitively adjudicate between these accounts, a point returned to as a key limitation below.

The use of clinical screening tools provides a novel perspective on SDB in LLMs and its potential implications for health care applications. Just as human participants often underreport undesirable behaviors (eg, problematic alcohol use and mental health challenges), the current findings suggest that the models tested also “underreport” negativity when the instrument, and by implication the possibility of scrutiny, is recognizable. This carries implications for researchers considering LLMs as proxies for human participants in clinical or survey research or in medical education and simulation.

If unaccounted for, this bias could produce artificially positive or virtuous depictions of health when simulating or analyzing responses in LLM-driven studies. This parallels earlier concerns about LLM biases and emergent behaviors [21], urging careful consideration of how models interpret and adapt to prompts, contexts, and perceived demands. In medical education, the same bias could plausibly sanitize simulated patient scripts, such that trainees encounter compliant, neatly articulated cases rather than the messy reality of malingering, nonadherence, and sociocultural friction, potentially limiting opportunities to practice boundary setting or to recognize red-flag behaviors. These downstream educational and clinical consequences are extrapolations from the present screening questionnaire findings and were not themselves measured in this study. Possible mitigations include prompting explicitly for dispreferred behaviors, fine-tuning on clinical corpora that retain such behaviors, triangulating LLM-generated data with probability-based human samples, and embedding bias audits at each stage of deployment; the effectiveness of these measures for the present effect has not been tested.

It is critical to interpret these findings based on the intended application of the LLM. In medical education simulations, where the goal is to portray realistic patients, LLM-driven SDB may be a feature rather than a flaw, as it reflects the complexities of real-world patient interactions. However, in clinical decision support or synthetic data generation, this mimicry becomes a liability. If an LLM is tasked with screening for hazardous alcohol use, “humanlike” underreporting undermines the tool’s sensitivity. Researchers and clinicians must therefore distinguish between simulation fidelity (mimicking humans) and measurement accuracy (reporting ground truth) when selecting and prompting these models.

Several limitations merit discussion. First, this is a formative study of 4 models from a single vendor. The models tested have since been superseded, and the findings should not be generalized to models from other developers or to later OpenAI models without replication; the title and conclusions have been framed accordingly. Cross-vendor replication using the same pipeline is the most important next step and is discussed in the Future Work section. Second, although multiple screening tools known to be sensitive to SDB were used, future studies could broaden the scope by testing additional measures or examining cultural factors that might influence the salience of social desirability. Third, the comparison with human normative data is descriptive only. The published norms are drawn from demographically specific samples (eg, the GAD-7 norm was 53.6% female, with a mean age of 48.4 years), whereas the prompts asked the model only to “imagine you are a random member of the public” without specifying demographic characteristics. LLMs are known to adopt implicit default demographic profiles under such underspecified personas [22], so the absolute level of the scores partly reflects an unknown default persona; therefore, no statistical comparison with the human norms was made. The principal finding rests instead on the within-model contrast between the naive and informed conditions: because any default demographic profile is held constant across both conditions for a given model, demographic defaulting cannot account for the systematic shift observed when the instrument became recognizable. Demographically anchored personas matched to a specific norm sample would be needed to make any absolute comparison interpretable. Finally, while the methods aimed to replicate a real-world “evaluation context,” the actual dynamics of user interactions with LLMs in clinical or research environments may involve more nuanced cues and pressures.

This study did not measure the effects of item order and model temperature, although previous studies have assessed these variables and established that they have negligible effects [4]. Similarly, acquiescence bias (ie, being more likely to agree with questions) has been shown to be present to some degree in LLM responses [23,24]. However, reverse-coding items has previously been shown to reduce bias by only around 50% [4]. It should also be noted that trial-to-trial variance in model responses was small (total score SDs of approximately 1-5 points), so the 100 trials per condition are better regarded as repeated samples from a near-deterministic response distribution than as 100 independent respondents. The t tests establish that the 2 conditions differ reliably for a given model, but the very large standardized effect sizes partly reflect this low variance and should not be interpreted as they would be for a human sample.

Two extensions follow directly from the limitations above. The pipeline used here can be applied without modification to models from other developers; replication with at least 1 non-OpenAI model family, and with current-generation models, is required before any general claim about LLMs can be made. Second, administering the instruments under demographically anchored personas matched to published norm samples would allow absolute scores, and not only the between-condition shift, to be interpreted. Beyond these, the “politeness skew” of LLMs should be probed in domains where its impact is subtler but equally consequential. There is a need for further systematic audits of diversity and representation to establish, for example, whether synthetic patient cases underrepresent minoritized sexual orientations, neurodivergent presentations, or intersectional identities [21]. Similarly, evaluations of LLMs’ emotional intelligence should move beyond sentiment classification to measure how convincingly models can simulate ambivalent, conflicted, or socially disapproved emotions (eg, resentment toward medical staff or ambivalence about sobriety). Classic technical benchmarks (factual recall, reasoning chains, and robustness to adversarial prompts) should also be reevaluated under conditions explicitly designed to elicit norm-violating content; preliminary evidence suggests accuracy degrades when truthful answers carry reputational risk [25].

Finally, the “naive” condition involved presenting scale items in isolation. While this successfully blinded the model to the evaluative context, it disrupts the psychometric structure of the tools. It is possible that the absence of surrounding items altered response patterns through mechanisms other than SDB, such as the loss of central tendency often seen in full-scale completion. More fundamentally, because the naive and informed conditions differ simultaneously in evaluative transparency, prompt length, and aggregate item sentiment, the present design cannot definitively establish that the observed shift is mediated by the model’s inference of an evaluative context rather than by context-length anchoring or sentiment equalization. The interpretation in terms of SDB should therefore be regarded as the most parsimonious account consistent with prior work rather than as a directly demonstrated mechanism. Future work could disentangle these explanations by holding prompt length and sentiment constant while varying only evaluative framing, for example, by embedding scale items within length-matched “filler” items of neutral valence or by explicitly signaling vs concealing the evaluative purpose at fixed context length. A complementary and more direct approach would be to construct application-level test harnesses that task models with the specific job of concern (eg, generating synthetic patient cohorts or simulating survey respondents for biomedical research) and then measure SDB in those generated data against ground truth distributions. Demonstrating the bias within the end-use task itself, rather than inferring it from screening questionnaire responses, would more directly establish the practical risk to LLM-driven data generation.

In 4 OpenAI models, presenting a validated health screening instrument in a form that allowed it to be recognized shifted responses toward the socially desirable pole in 14 of 16 model-instrument combinations, with the direction and size of the shift depending on the instrument and the model family. These results provide formative evidence for the hypothesis that SDB, previously documented in LLM responses to personality inventories, extends to clinical screening instruments; they do not establish it as a general property of LLMs, and the mechanism remains to be isolated. Two practical safeguards follow regardless of mechanism: prompt-level bias audits before an LLM is used as a screening or data generation tool in health settings and explicit reporting of prompt framing in studies that use LLM-generated health data, as framing alone systematically shifted responses in this study.

SDB in LLMs has been documented in personality and survey settings; these results extend that observation to validated health screening instruments for one vendor’s models, highlighting its potential impact on research, clinical practice, and medical education. Whether the propensity persists, intensifies, or resolves as models evolve and whether it is shared across developers are empirical questions that the pipeline described here is designed to answer. Understanding (and ultimately mitigating) this bias is a prerequisite for the valid use of LLMs as simulated patients or synthetic respondents in health care, psychology, and related fields.