Major depressive disorder (MDD) is a leading cause of disability worldwide, accounting for approximately one-third of years lived with disability from mental disorders, and its incidence has increased by approximately 60% between 1990 and 2019 [1,2]. The clinical course and treatment response in MDD are highly variable, and many individuals experience incomplete remission or treatment resistance [3,4]. Capturing this variability in real-world settings is important to gain a more nuanced phenotypic understanding of depression to inform more personalized care. Although clinical trials routinely use validated symptom rating scales, outcome data in routine psychiatric care are inconsistently measured [5], with less than 20% of mental health providers incorporating measurement-based care (MBC) into their practice and as few as 5% using it consistently at every session [6]. Although MBC has been shown to improve outcomes and is recommended in treatment guidelines [7,8], real-world uptake has been slow due to practical and organizational obstacles [9]. Implementation programs such as the National Network of Depression Centers Mood Outcomes Program have only recently demonstrated that adoption is feasible at scale when supported by coordinated infrastructure and system-level investment [10].

In this context, the Clinical Global Impression (CGI) scale [11] represents a promising candidate for scalable, measurement-based assessment in real-world settings. The CGI is a brief, Likert-type scale rated by clinicians during patient encounters and includes both the CGI-Severity (CGI-S) scale, which rates cross-sectional illness severity from 1 (not at all ill) to 7 (among the most extremely ill patients), and the CGI-Improvement (CGI-I) scale, which assesses longitudinal change from 1 (very much improved since initiation of treatment) to 7 (very much worse since initiation of treatment). The CGI-S scale, the focus of the present work and hereafter referred to simply as CGI, has been validated in a wide range of psychiatric settings (eg, inpatient [12], outpatient [9,13], and clinical trial contexts [14]) and psychiatric conditions including mood [15-18], anxiety [19,20], and psychotic disorders [21-23]. In inpatient care, CGI scores have shown strong convergent validity with established measures such as the Health of the Nation Outcome Scales (HoNOS), Mental Health Questionnaire–14 (MHQ-14), and Depression Anxiety Stress Scales–21 (DASS-21) [12], underscoring their value as a brief and pragmatic outcome measure. In depression specifically, CGI ratings have demonstrated strong correlations with standard symptom scales (r≈0.6‐0.8), including the Beck Depression Inventory (BDI), Hamilton Depression Rating Scale (HAM-D), and Montgomery-Åsberg Depression Rating Scale (MADRS), and have shown comparable or greater sensitivity to treatment-related change across outpatient and clinical trial populations [13,24-26].

Interrater reliability for CGI has generally been reported to be in the moderate-to-excellent range across psychiatric disorders, with Cohen κ or interrater coefficient values of approximately 0.7-0.8 [23,27-29]. However, critiques have highlighted its relative lack of specificity and ambiguous response anchors, which may contribute to variability in ratings [30-32]. To address these concerns, Kadouri et al [27] developed an improved CGI (iCGI) for depression, incorporating standardized case vignettes to guide rater calibration. This approach achieved excellent interrater reliability when averaging rater scores (intraclass correlation coefficient [ICC]≈0.9) and demonstrated greater sensitivity for detecting clinical change than with the HAM-D [27].

Despite extensive validation in research and its potential for broad use across conditions, CGI scores are not routinely used or documented in clinical settings, where they could serve as a readily available outcome measure for large-scale observational studies. Instead, in typical clinical practice, details regarding symptom severity and improvement are recorded in the narrative text of clinical notes rather than quantified in a standardized way.

Advances in natural language processing (NLP) have introduced the possibility of extracting outcome measures directly from unstructured text. Previous approaches demonstrated that mood states and longitudinal treatment response could be accurately classified from outpatient psychiatric notes using rule-based NLP models [33]. More broadly, NLP methods have been shown to consistently improve the performance of predictive models created solely from structured data [34-38]. More recently, large language models (LLMs) have enabled the analysis of clinical text with a deeper understanding of linguistic context and meaning, moving beyond the keyword- or rule-based features used in previous NLP approaches [39,40], with robust performance in both psychiatric feature extraction and classification tasks [41-44]. However, recent work using LLMs (GPT-4o; OpenAI) to estimate depression severity scores from primary care notes demonstrated only modest performance, likely reflecting the limited psychiatric clinical documentation in primary care settings [45].

Recent applications within psychiatry, where notes typically contain richer descriptions of symptoms and clinical impressions, have provided encouraging evidence that LLMs can generate structured clinical outcomes from narrative text. Wiest et al [46] applied a Llama-2 model to psychiatric admission notes to classify the presence of suicidal ideation, plans, or attempts, achieving more than 85% accuracy and strong agreement with expert consensus ratings. McCoy and Perlis [47] extended this work by applying reasoning-style LLM prompts to hospital discharge summaries to estimate suicide risk, yielding model-derived hazard ratios that were predictive of subsequent suicide or accidental death, while also generating step-by-step, interpretable textual explanations for the predictions. In a related study, GPT-4 was used to translate psychiatric emergency department notes into quantitative symptom ratings across domains such as mood and cognition and demonstrated that these scores could predict hospitalization and length of stay [48].

Building on this foundation, we developed and tested a pipeline for using LLMs to generate CGI scores from the clinical notes of patients with depression. We first created a psychiatrist-annotated consensus reference and then compared different zero-shot and few-shot LLM prompting approaches to assess which showed the best alignment with clinician assessment. Through this approach, we sought to evaluate the feasibility of using LLMs to generate standardized outcome scores from unstructured psychiatric documentation, thereby bridging the gap between clinical assessment and scalable measurement for clinical use and research.

In this study, we sought to assess whether LLMs could produce reliable CGI scores, a clinician-rated measure of current patient symptom burden and impairment, from psychiatric clinical notes and how these scores compare with psychiatrist ratings. Using a prompt engineering strategy without fine-tuning, GPT-4o produced CGI scores that closely matched clinician ratings. Agreement between the model-generated and clinician-assigned CGI scores was comparable to CGI interrater reliability values previously reported among human raters (ICC or κ≈0.7-0.8) [23,27-29] and was especially high when compared with the average of all human raters (κ=0.84-0.85). Comparing the model against the average may inflate the estimate of κ, but it provides a useful indication of the upper bound of the model’s performance. Notably, the GPT-4o model performed markedly better than a leading open-source model, and few-shot prompting did not demonstrate improved agreement compared with zero-shot prompting. Although a formal statistical test comparing the zero-shot and few-shot models was not performed, the point estimates were nearly identical, with substantial overlap in their 95% CIs. These findings suggest that a clear, task-specific prompt anchored to the CGI rubric may be sufficient to achieve strong performance with GPT-4o.

Two key design choices were made to optimize model performance. First, we explicitly aligned the prompt with an established guide for CGI scoring. This design choice was intentional to standardize the rating task between the model and human raters; however, performance may not generalize to settings in which the rubric is omitted or modified. Further work is needed to evaluate whether similar performance is maintained across alternative rubrics or psychiatric conditions other than depression. We also included language instructing the model to prioritize recent information in the clinical notes, to ensure that the generated scores reflected the patient’s current status rather than past clinical history. The latter may be especially important given the frequency with which clinical notes are copy-forwarded, with studies estimating that approximately half of note content is duplicated from the author’s previous note [55,56]. The phenomenon of copy-forward within the clinical notes could potentially bias the model, without appropriate remediation, toward prioritizing prior information and lead it to fail to detect more recent clinical changes. Second, for few-shot prompting, we selected examples from notes with complete human rater agreement to maximize gold-standard fidelity. The lack of incremental benefit from few-shot over zero-shot prompting, however, may reflect a ceiling effect of our rubric-based prompt. Because human interrater reliability in our sample was κ=0.77‐0.78, model-human agreement would be expected to plateau near this level, even though agreement with a consensus gold-standard reference could be higher.

To assess model performance across potential subgroups, we further carried out stratified analyses by age, sex, race, treatment location, note length, and copy-forwarded text burden. Agreement remained high across most subgroups, but it did differ significantly by note length, with notes at or above the median length having higher agreement with average human ratings than those below the median length. This suggests that the amount of available clinical context may influence model performance, with shorter notes providing fewer details from which the model can infer depression severity. Given this finding, future work with LLM-derived CGI scores may need to be paired with minimum note length or information-sufficiency thresholds and may require human review when notes are brief or clinically sparse. This interpretation is consistent with our qualitative review of discrepant cases, in which some errors appeared to arise when brief symptom descriptions or isolated mental status findings were weighed disproportionately relative to the overall clinical impression. In contrast, agreement did not significantly differ by copy-forward burden, despite the high prevalence of copy-forwarded text in the sample. Although copy-forwarded text could theoretically bias the model toward prior clinical information rather than the patient’s current status, this finding suggests that prompting the model to prioritize recent information may have helped mitigate this concern. In exploratory error analyses, we also examined whether patient- or note-level characteristics were associated with the direction of model disagreement and found that no characteristics were significantly associated with LLM overestimation or underestimation, although these analyses were limited by the modest number of discordant cases. Together, these findings suggest generally consistent model performance across the measured subgroups while underscoring the need for larger studies to evaluate potential sources of systematic error.

Our study has several important limitations. First, we demonstrated only the reliability of an LLM compared with psychiatrists in assigning CGI scores based on information available in the text of clinical notes. This approach does not consider other information available in the EHR, such as diagnostic subtypes, clinical comorbidities, and treatment history, which may contribute to the global clinical impression of a patient. Moreover, it is unclear to what extent CGI scores based on clinical notes would compare with CGI scores assigned by psychiatrists based on their direct clinical evaluation of the patients (ie, concurrent validity) or with other measures of depression-related constructs such as the PHQ-9 or MADRS (ie, convergent validity). Unfortunately, we did not have sufficient data to examine the correlations between model-based CGI scores and other concurrent measures of depression. Importantly, LLM-generated CGI scores from clinical notes are not intended to replace clinical judgment or standardized symptom scales such as the PHQ-9 or MADRS, which serve complementary roles in clinical assessment and MBC. The relationship between CGI ratings and these scales has been well established in the prior literature, reflecting related but distinct constructs, and the present study does not aim to re-evaluate the validity of the CGI or to compare these instruments. Future work will be needed to evaluate concurrent validity by comparing text-derived CGI scores with clinician-assigned CGI ratings obtained at the time of care; however, such analyses were beyond the scope of the present study. Second, the study was limited to encounters with a primary diagnosis of MDD within a single hospital system and to notes authored by psychiatrists. Consequently, the findings may not be generalizable to other diagnoses, settings, or types of clinical providers, where varying documentation styles are practiced. Third, the 3 psychiatrist raters were study coauthors and were not blinded to the overall study objective, so potential expectancy effects cannot be fully excluded despite independent annotation procedures. This is a common limitation of annotation-based studies and should be addressed in future work using external raters independent of the study team. Finally, we observed the best performance with a closed-source LLM, which raises concerns about reproducibility, version stability over time, and cost feasibility. Performance may vary across model versions due to model updates or drift, and access to proprietary models may be constrained in some research or clinical settings. Due to these limitations, more work is needed to establish the validity of using LLMs to assign CGI scores based on clinical notes, especially in comparison with the concurrent assessments of psychiatrists based on direct clinical evaluation, and to determine the generalizability of the approach to other psychiatric diagnoses and clinical settings. Our study is an important first step, and the findings demonstrate that this future work is warranted.

These findings extend prior work showing that NLP methods, especially LLMs, can be used to support patient phenotyping and outcome measurement such as depression response or remission [33], suicide risk [46,47], and dimensional symptom ratings [48]. Our study demonstrates that LLMs can generate CGI scores, a clinician-rated global measure that is validated and widely used in clinical research but rarely captured in EHRs, based on clinical notes with agreement comparable to psychiatrist ratings. Because CGI ratings have shown good interrater reliability and strong correlations with other standardized measures [9,12,13,57], the ability to generate CGI scores from narrative clinical notes offers, if further validated, the potential of a pragmatic approach to generate outcome measures suitable for real-world studies using EHR data, as well as open up the possibility of deriving such measures in real time from ambient AI technology to facilitate MBC.