This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2410) Diagnostic Tools in Mental Health (242) Digital Mental Health Interventions, e-Mental Health and Cyberpsychology (1514) Mental Health Issues in Adolescence (442) Innovations in Mental Health Systems (390) Methods and Feasibility Studies (379) Users' and Patients' Needs for Mental Health Services (403)

Published on in Vol 7 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/45161, first published .
Digital Application of Clinical Staging to Support Stratification in Youth Mental Health Services: Validity and Reliability Study

Digital Application of Clinical Staging to Support Stratification in Youth Mental Health Services: Validity and Reliability Study

Digital Application of Clinical Staging to Support Stratification in Youth Mental Health Services: Validity and Reliability Study

Authors of this article:

Min K Chong1 Author Orcid Image ;   Ian B Hickie1 Author Orcid Image ;   Shane P Cross2 Author Orcid Image ;   Sarah McKenna1 Author Orcid Image ;   Mathew Varidel1 Author Orcid Image ;   William Capon1 Author Orcid Image ;   Tracey A Davenport3 Author Orcid Image ;   Haley M LaMonica1 Author Orcid Image ;   Vilas Sawrikar4 Author Orcid Image ;   Adam Guastella1, 5 Author Orcid Image ;   Sharon L Naismith1, 6 Author Orcid Image ;   Elizabeth M Scott1, 7 Author Orcid Image ;   Frank Iorfino1 Author Orcid Image
Article Authors Cited by (3) Tweetations (17) Metrics

    Original Paper

    1Brain and Mind Centre, University of Sydney, Camperdown, Australia

    2Orygen, Parkville, Australia

    3Design and Strategy Division, Australian Digital Health Agency, Sydney, Australia

    4School of Health and Social Sciences, University of Edinburgh, Edinburgh, United Kingdom

    5Children's Hospital Westmead Clinical School, Faculty of Medicine and Health, The University of Sydney, Sydney, Australia

    6Healthy Brain Ageing Program, University of Sydney, Sydney, Australia

    7St Vincent's and Mater Clinical School, The University of Notre Dame, Sydney, Australia

    Corresponding Author:

    Min K Chong, BSc (Hons)

    Brain and Mind Centre

    University of Sydney

    94 Mallett Street

    Camperdown, 2050

    Australia

    Phone: 61 291142199

    Email: min.chong@sydney.edu.au


    Background: As the demand for youth mental health care continues to rise, managing wait times and reducing treatment delays are key challenges to delivering timely and quality care. Clinical staging is a heuristic model for youth mental health that can stratify care allocation according to individuals’ risk of illness progression. The application of staging has been traditionally limited to trained clinicians yet leveraging digital technologies to apply clinical staging could increase the scalability and usability of this model in services.

    Objective: The aim of this study was to validate a digital algorithm to accurately differentiate young people at lower and higher risk of developing mental disorders.

    Methods: We conducted a study with a cohort comprising 131 young people, aged between 16 and 25 years, who presented to youth mental health services in Australia between November 2018 and March 2021. Expert psychiatrists independently assigned clinical stages (either stage 1a or stage 1b+), which were then compared to the digital algorithm’s allocation based on a multidimensional self-report questionnaire.

    Results: Of the 131 participants, the mean age was 20.3 (SD 2.4) years, and 72% (94/131) of them were female. Ninety-one percent of clinical stage ratings were concordant between the digital algorithm and the experts’ ratings, with a substantial interrater agreement (κ=0.67; P<.001). The algorithm demonstrated an accuracy of 91% (95% CI 86%-95%; P=.03), a sensitivity of 80%, a specificity of 93%, and an F1-score of 73%. Of the concordant ratings, 16 young people were allocated to stage 1a, while 103 were assigned to stage 1b+. Among the 12 discordant cases, the digital algorithm allocated a lower stage (stage 1a) to 8 participants compared to the experts. These individuals had significantly milder symptoms of depression (P<.001) and anxiety (P<.001) compared to those with concordant stage 1b+ ratings.

    Conclusions: This novel digital algorithm is sufficiently robust to be used as an adjunctive decision support tool to stratify care and assist with demand management in youth mental health services. This work could transform care pathways and expedite care allocation for those in the early stages of common anxiety and depressive disorders. Between 11% and 27% of young people seeking care may benefit from low-intensity, self-directed, or brief interventions. Findings from this study suggest the possibility of redirecting clinical capacity to focus on individuals in stage 1b+ for further assessment and intervention.

    JMIR Form Res 2023;7:e45161

    doi:10.2196/45161

    Keywords

    clinical staging; digital health solution; online diagnosis; service transformation; staged care; stratified care; youth mental health 


    Overview

    Mental disorders present a substantial global challenge [1-3]. Poorly managed mental health too often leads to premature death [4] and has profound psychological, physical, and socioeconomic impacts. As 75% of mental disorders first emerge before the age of 25 years [5], early intervention and secondary prevention in young people are critical to halting the chronic impacts of mental illnesses [6,7].

    Reforms in youth mental health services, digital platforms, and new models of care have been introduced in response to this critical need [8]. These models of care include stepped care, which offers low-intensity treatments for individuals’ specific needs [9], and integrated care, a multidisciplinary model of care that addresses diverse presentations of mental health in young people, including physical, mental, and functional concerns [10]. These innovations recognize that delayed access to treatment leads to increases in disengagement, discourages future help-seeking behavior, and results in poorer outcomes at higher costs [11-14]. However, despite such collective effort, demand for youth mental health services continues to surge, which impacts the delivery of timely and effective care [15-17]. Staged care is another model of care that aims to match the intervention intensity to the individual’s needs from the start of treatment [18]. A proposed solution for demand management in services is to use this stratification process (ie, staged care) to accurately and efficiently assess the complexity of a young person’s mental health condition and recommend appropriate care pathways [19].

    Clinical Staging Model

    Clinical staging is a tool of risk stratification that guides decision-making for care pathways [20,21]. It stratifies help-seeking individuals based on illness severity, progression, and prognosis, ensuring that the intensity and timeliness of care match the urgency and complexity of their needs [18]. Current diagnostic systems, including the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition [22] and the International Classification of Diseases, Tenth Revision, Clinical Modification [23], are siloed and discrete, which limit their use in youth mental health for predicting an individual’s illness trajectory. On the contrary, the clinical staging model places an individual on a transdiagnostic continuum of their illness based on factors such as symptom duration, syndrome severity, and comorbidities. Hence, the clinical stage can indicate one’s clinical severity and risk of illness progression [21,24], facilitating clinicians to take preemptive action and allocate appropriate care from the start of treatment. This is especially useful for young people in the early stages of illness, as they often present with an admixture of subthreshold symptoms and comorbidities that do not fit into a specific diagnosis [3].

    Furthermore, clinical staging assessments use a multidimensional framework that considers mental and physical health, social and occupational functioning, and substance use. Stratification systems that rely on single or state-based dimensions (ie, distress score or severity of symptoms) lack clinical usability as they fail to account for increased vulnerability associated with previous episodes of illness, comorbidity, and complex syndromes [25,26]. Studies show that while clinical predictors (eg, negative symptoms or psychotic characteristics) overlap across different mood disorders [27], nonclinical predictors (eg, family history or substance misuse) could strongly determine the emergence of specific mental illnesses in young people [26]. Therefore, this transdiagnostic model is a more appropriate risk stratification tool for early intervention in mental health services [18].

    The clinical staging model provides a framework for differentiating individuals with persistent, full-threshold mental disorders (stage 2+), from those with nonspecific (stage 1a) and attenuated symptoms (stage 1b) [28,29]. A detailed set of criteria for clinical stages has been published elsewhere [20,30]; hence, only a short summary has been provided here. It is especially crucial to understand the distinction between stage 1a and stage 1b, as this can help predict the rate of illness progression [21]. While individuals in both stages 1a and 1b may present with subthreshold symptoms, they differ in clinical severity. Stage 1a individuals are characterized by nonspecific and mild symptoms of depression and anxiety disorders, whereas stage 1b individuals present a more complex symptom profile, including mild to severe symptoms of mood disorders, often with comorbidities. Therefore, while low-intensity services (eg, self-management and monitoring interventions) would be suitable for stage 1a, moderate-intensity services (eg, cognitive behavioral therapy) may be appropriate for stage 1b. Risk stratification using the clinical staging framework may help to prevent illness progression by aligning treatment intensity to an individual’s level of needs from the start of treatment.

    This approach of the clinical staging model (ie, staged care) contrasts with the stepped care model, which initially assigns low-intensity care to all individuals and gradually elevates the level of care based on treatment response. Previous studies have demonstrated that staged care leads to faster and improved treatment outcomes compared to stepped care [31-33]. Additionally, it can facilitate early intervention and secondary prevention by detecting individuals who are susceptible to rapid illness progression. Therefore, incorporating clinical staging assessment at service intake can provide an opportunity to expedite care provision for those with complex needs and minimize their risk of illness progression.

    Digitalization of Clinical Staging

    One of the main challenges of using clinical staging in services is the intense time and labor required for the assessment process. Conducting comprehensive multidimensional assessments can be time-consuming, and the associated costs and labor involved with educating clinicians pose major barriers to the widespread adoption of clinical staging. Relying solely on clinicians to perform these assessments would not be feasible for implementing clinical staging in mental health services at scale.

    In recent years, digital technologies have been developed to improve access, efficiency, and quality in the health system. Technology-enabled assessment and care coordination have the potential to enhance access to standardized screening assessments and improve efficiency in the early detection of individuals with high and urgent needs [34-36]. Additionally, these technologies have the advantage of reaching a wider population, including services in rural and low-resource settings [37-39]. In the field of mental health, technology has been reported as a preferred method for self-disclosure and has been well-received among young people [40].

    Hence, as a potential solution to the challenges of clinical staging implementation, a digital algorithm was developed to automatically assess clinical stage based on a multidimensional self-report questionnaire [41]. By leveraging this computational power, the algorithm offers a time- and cost-efficient solution for services to differentiate young people who require low-intensity interventions (stage 1a) from those who require moderate- to high-intensity interventions (stage 1b+) before their first consultation. This differentiation is an important early clinical decision for services to make with regard to service pathways and resource allocation. Validation of this digital algorithm may facilitate the implementation of clinical staging at scale to improve service efficiencies and expedite care pathways.

    Therefore, the aim of this study was to assess the validity of the digital algorithm to accurately allocate young people to stage 1a or stage 1b+, compared to expert raters. Demographic and clinical characteristics of individuals in each clinical stage were compared to further evaluate the algorithm.


    Participants

    A total of 131 participants aged between 16 and 25 years were randomly selected from help-seeking young people who presented at primary mental health care services (headspace [10]). All young people were deemed eligible if they presented for the first time to a participating headspace service between November 2018 and March 2021 and used the Innowell Platform to provide assessment information. The participating headspace services were in Central and Eastern Sydney, Queensland, and South Australia. The cohort was a subset of the total presentation at these services, representing the clinical characteristics of the help-seeking young people.

    Innowell Platform

    Innowell Platform is a digital technology that supports individuals’ mental health and well-being by facilitating personalized and measurement-based care [41]. It is a web-based platform that provides multidimensional assessments and immediately generates outcome reports that can be used to identify individuals’ psychological, functional, and physical needs [42]. Individuals can review their clinical assessment scores with their health professionals on their personalized dashboard, and their symptom progression can be tracked by repeating the questionnaires over time. The platform offers other resources, such as links to fact sheets, apps, or web-based tools, according to individuals’ clinical needs.

    Procedures

    Registration on the Innowell Platform was part of the intake process at participating mental health clinics. All participants completed a self-report questionnaire on the Innowell Platform before their initial consultation with a clinician. The questionnaire assessed individuals’ background information, including demographics, current education and employment participation, mental health concerns (ie, psychological distress, depressed mood, anxiety, mania-like experiences, psychosis-like experiences, and posttraumatic experiences), self-harm and suicidal behaviors, tobacco, alcohol and other substance use, physical activity, sleep-wake cycles, mental health history, eating behaviors, and social connectedness. After the web-based assessment, the digital algorithm embedded in the platform automatically allocated individuals to stage 1a or stage 1b+ based on the questionnaire results. Details of the algorithm have been published previously [41]. Only 2 stages were allocated (stage 1a or 1b+), as individuals in clinical stage 1b or above should be directed to further clinical assessment to examine the level of care they require.

    Measures

    For the purposes of this study, the following measures were specifically selected and included for analysis: demographics, mental health, suicidality, functioning, alcohol and substance use, eating behavior, and circadian disturbance.

    Demographics

    Participant age, gender, highest level of education, and current education, employment, and training status (used to determine “not in education, employment, or training” [NEET] status) were collected.

    Mental Health

    Current psychological distress was assessed using the Kessler Psychological Distress Scale (K10) [43], a well-validated and widely used measure of general psychological symptoms and distress in adult and adolescent populations in both clinical and community settings. Manic-like experiences over the last 12 months were assessed using a screener derived from the Altman Self-Rating Scale [44]. To assess subclinical psychotic symptoms, the Prodromal Questionnaire (PQ-16) was used [45]. The Overall Anxiety Severity and Impairment Scale (OASIS) [46] and Quick Inventory of Depressive Symptomatology (QIDS) [47] were used to assess anxiety and depressed mood, respectively. Participants were also asked, “Have you ever experienced a major mental health or behavioral problem that has affected your everyday life?” to determine any mental health problems or hospitalization history.

    Suicidality

    The Suicide Ideation Attributes Scale (SIDAS) was used to assess suicidal ideation over the past month. The scale is comprised of 5 items that assess the frequency of suicidal thoughts and attempts, related distress, and impact on daily activities on a 10-point Likert scale. A score above 21 indicated a high risk of suicidal behavior. The scale is a valid web-based measure with strong internal reliability (Cronbach α=.91) [48]. A self-harm history question was adapted from the Brief Non-Suicidal Self-Injury Assessment Tool [49], and a suicide attempt history question was extracted from the Columbia-suicide severity rating scale [50].

    Functioning

    Everyday functioning was assessed using the Work and Social Adjustment Scale [51]. A total of 5 items were scored using an 8-point Likert scale. Participants were asked to rate their level of agreement with statements such as “Because of my mental health, my ability to work is impaired.” Scores 0 and 8 represented “not at all” and “very severely,” respectively.

    Alcohol and Substance Use

    Participants’ alcohol use, onset age, and related impairments were assessed using a combination of 3 measures. The frequency and impact of alcohol use were assessed with questions extracted from the Alcohol Use Disorders Identification Test [52] and the Alcohol, Smoking, and Substance Involvement Screening Test [53], respectively. One question on age onset was added based on past literatures [54-56]. Additionally, frequency of cannabis use and its associated impacts on health, social, legal, or financial domains were assessed using questions extracted from the Alcohol, Smoking, and Substance Involvement Screening Test [53].

    Eating Behavior

    Questions were adapted from the investigator-based interview, the Eating Disorders Examination (adapted version) [57]. The questions assessed behaviors such as binge eating, purging, strict dieting, and body image importance. Body image importance was rated on a 6-point Likert scale. A score of 0 indicated “not at all important,” and score 6 indicated “most important.” Individuals were rated as having an abnormal eating behavior if they experienced binge eating, purging, and diet restriction over the past 3 months and gave their body image importance a rating above 3.

    Circadian Disturbance

    Individuals’ sleep and wake time, sleep duration, and quality of sleep were assessed using questions from the Pittsburgh Sleep Quality Index [58] and the Munich Chronotype Questionnaire [59]. An additional question on restorative sleep was included based on its significance in the past literature [60].

    Clinical Staging Assessment

    To evaluate the validity of the digital algorithm, expert psychiatrists (IBH and EMS), who were involved with the development of the clinical stage model [20], allocated clinical stages using the results of the web-based questionnaire. An expert rating represents the collaborative clinical stage allocation by the 2 psychiatrists, which represents the current gold standard and best practice application of clinical staging [20]. The expert rating was conducted independently to the digital algorithm, and all participants were allocated to either stage 1a or 1b+.

    Statistical Analyses

    All statistical analyses were performed using the RStudio software (version 4.2.1; R Foundation) [61]. Demographic, functioning, and clinical characteristics between stages 1a and 1b+ were compared using the nonparametric Mann-Whitney U test for continuous variables and the chi-square test for categorical variables. To further evaluate discrepancies between the expert and the digital algorithm assigned clinical stages, pairwise comparisons were conducted with concordant and discordant ratings. Due to the number of comparisons made within the dataset, a Bonferroni correction was used. The α value of .05 was adjusted to P<.001. Cases with missing measures were excluded from the analyses.

    The reliability of the digital algorithm was assessed using the Cohen κ statistics [62] and the confusion matrix [63]. A κ coefficient between 0.01-0.20 was interpreted to be in slight agreement; κ=0.21-0.40 in fair agreement; κ=0.41-0.60 in moderate agreement; κ=0.61-0.80 in substantial agreement; and κ=0.81-0.99 in almost perfect agreement [64]. Using the confusion matrix, accuracy, positive predictive value (precision), sensitivity (recall), and the F1-score were analyzed. Accuracy represents the percentage of correct ratings in the total sample, and the F1-score presents a balanced mean using precision and recall. A higher F1-score indicates a greater rate of both precision and recall.

    Ethics Approval

    The Northern Sydney Local Health District Human Research Ethics Committees approved this study (HREC/17/HAWKE/480), and all participants gave web-based informed consent (through an opt-out process).


    Sample Characteristics

    A total of 131 young people were included in the analyses (Table 1). The mean age of participants was 20.3 (SD 2.4) years, 72% (94/131) of them were female, and 36% (47/131) of them completed tertiary education. Most participants reported that they did not have a disability (91%, 115/126).

    Table 1. Differences in demographic and clinical characteristics among participants with concordant and discordant stage ratings.
    CharacteristicsTotal participant (n=131)Stage 1a (agree)a (n=16) Stage 1b+ (agree)b (n=103)Stage 1b+ (disagree)c (n=8)Comparison d





    		Stage 1a (agree) versus stage 1b+ (disagree)Stage 1b+ (agree) versus stage 1b+ (disagree)
    Age (years), mean (SD)20.3 (2.4)20.6 (2.5)20.1 (2.5)21.5 (1.9)— e—
    Female, n (%)94 (71.8)11 (68.8)75 (72.8)4 (50)——
    Education, n (%)

    		Secondary84 (64.1)4 (25)75 (72.8)4 (50)——

    		Tertiary47 (35.9)12 (75)28 (27.2)4 (50)——
    Mental health history, n (%)f101 (78.3)8 (50)86 (84.3)3 (42.9)——
    With disability, n (%)f11 (8.7)0 (0)10 (10.1)1 (12.5)——
    Functioning, mean (SD)20 (9.1)8.9 (5.1)21.9 (8.6)17.4 (3)<.001
    Clinical presentation

    		Psychological distress, mean (SD)32.8 (8.6)22.2 (6.4)35.1 (7.6)25.8 (4.8)——

    		Depression, mean (SD)f14.8 (5.7)6.8 (3.5)16.5 (4.9)9.9 (3.5)—<.001

    		Anxiety, mean (SD)f10.1 (5.0)5.1 (3.4)11.1 (4.7)5.6 (1.1)—<.001

    		Manic-like experiences, n (%)f37 (40.7)1 (12.5)36 (48.6)0 (0)——

    		Psychotic-like experiences, n (%)f58 (45.7)0 (0)58 (57.4)0 (0)——

    		Circadian disturbance, n (%)f86 (67.2)5 (31.3)73 (72.3)5 (71.4)——

    		Abnormal eating behavior, n (%)f9 (7.2)0 (0)9 (9.2)0 (0)——
    Self-harm and suicidal thoughts and behaviors, n (%)

    		Self-harm historyf72 (55.4)3 (18.8)66 (64.1)2 (28.6)——



    		Suicidal ideation, mean (SD)9.8 (12.6)1.7 (4)12.1 (13.2)1.4 (2.7)——



    		Suicide attempt historyf49 (37.7)0 (0)49 (47.6)0 (0)——
    Alcohol and other substance misuse, mean (SD)

    		Alcohol usef4 (3.1)2.9 (2.8)4.1 (3.1)6.1 (2.1)——

    		Cannabis usef2.1 (3.4)0.5 (0.9)2.3 (3.6)2.9 (3.8)——

    a Stage 1a by the algorithm and experts.

    b Stage 1b+ by the algorithm and experts.

    c Rated stage 1b+ by experts and stage 1a by the algorithm.

    dP<.001 for statistical significance

    eNot available.

    fThere were some missing data from the sample for following demographic characteristics. Mental health history, n=129; with disability, n=126; depression, n=126; anxiety, n=129; manic-like experiences, n=91; psychotic-like experiences, n=127; circadian disturbance, n=128; abnormal eating behavior, n=125; self-harm history, n=130; suicide attempt history, n=130; alcohol use, n=124; cannabis use, n=126. Percentages have been calculated only with available data to represent the proportion of young people with corresponding characteristics. Corresponding measures; Functioning, Work and Social Adjustment Scale; Psychological distress, Kessler-10; Depression, Quick Inventory of Depressive Symptomatology; Anxiety, Overall Anxiety Severity and Impairment Scale; Manic-like experiences, Altman Self-Rating Mania Scale; Psychotic-like experiences, Prodromal Questionnaire; Circadian disturbances, Pittsburgh Sleep Quality Index and Munich Chronotype Questionnaire; Abnormal eating behavior, Eating Disorder Examination (adapted version); Self-harm history, Brief Non-Suicidal Self-Injury Assessment Tool; Suicidal ideation, Suicide Ideation Attributes Scale; Suicide attempt history, Columbia-Suicide Severity Rating Scale; Alcohol use, Alcohol Use Disorders Identification Test; Cannabis use, Alcohol, Smoking and Substance Involvement Screening Test.

    Overall, the sample reported moderate to severe mental health concerns. The sample displayed severe psychological distress (K10 total mean 32.8, SD 8.6; range 10-50), moderately depressed mood (QIDS total mean 14.8, SD 5.7; range 0-27), and moderate anxiety symptoms (OASIS total mean 10.1, SD 5.0; range 0-20). Manic-like experiences (37/91, 41%) and psychotic-like experiences (58/127, 46%) were commonly reported. Over half of the participants had a history of self-harm (72/130, 55%), 38% (49/130) with previous suicide attempts and one-fifth exhibited high suicidality (26/131, 20%). A large proportion reported sleep disturbances (86/125, 69%).

    Interrater Reliability

    Out of the 131 participants, 119 (91%) clinical stages allocated by the digital algorithm were concordant with those assigned by experts (Table 2). Among the 12 discordant ratings, the algorithm assigned 8 participants to a less severe clinical stage (ie, stage 1a) compared to the experts. Cohen κ of 0.67 indicated a substantial agreement between the algorithm and the expert ratings, and the digital algorithm achieved an accuracy of 91% (95% CI 85%-95%; P=.03) and an F1-score of 73%. The sensitivity and specificity of the algorithm were 80% and 93%, respectively.

    Table 2. Confusion matrix comparing clinical stage assessments between digital algorithm and expert rating.
    Expert ratingDigital algorithm

    		Stage 1a (n=24), n (%)Stage 1b+ (n=107), n (%)
    Stage 1a (n=20)16 (12)4 (3)
    Stage 1b+ (n=111)8 (6)103 (79)

    Demographic, Functional, and Clinical Characteristics Differences Between Concordant and Discordant Ratings

    To further develop the algorithm, the demographic, functional, and clinical differences between concordant and discordant cases were evaluated. The first set of analyses compared participants with concordant stage 1a with those who were allocated to stage 1a by algorithm but to stage 1b+ by the experts (addressed as discordant stage 1b+ hereafter). The analyses showed a significant functional impairment in the discordant stage 1b+ group compared to concordant stage 1a group (U=7.5; z=–3.3; P<.001; Table 1).

    Then, the second set of analyses compared participants with concordant stage 1b+ and the discordant stage 1b+ group. The results showed that the discordant stage 1b+ group had less symptoms of depression (U=77; z=–3.2; P<.001) and anxiety (U=86; z=–3.2; P<.001; Table 1).

    Due to the small sample size, no analysis was conducted for participants with discordant stage 1a rating (allocated to stage 1a by the experts and stage 1b+ by the algorithm).

    Demographic, Functional, and Clinical Characteristics of Stage 1a and Stage 1b+

    To evaluate the characteristics that differentiated stage 1a and stage 1b+ for each examiner, participants were grouped by stages per examiner, and pairwise comparison analyses of their characteristics were performed. For both examiners, participants in stage 1b+ had greater psychological distress (algorithm: U=305.5, z=–5.7, P<.001; expert: U=349, z=–4.7, P<.001), depressed mood (algorithm: U=192, z=–6.2, P<.001; expert: U=252, z=–5.3, P<.001), anxiety symptoms (algorithm: U=343.5, z=–5.3, P<.001; expert: U=527.5, z=–3.5, P<.001), and functional impairment (algorithm: U=447.5, z=–4.8, P<.001; expert: U=379.5, z=–4.5, P<.001). Further, psychotic-like experiences were more common in stage 1b+ (algorithm: χ21=20.2, P<.001; expert: χ21=17.8, P<.001). However, episodes of self-harm (χ21=11.2, P<.001) and mental illness history (χ21=13.2, P<.001) were only significantly different between the algorithm-rated stage 1a and stage 1b+ participants. Additionally, the level of education (χ21=13.8, P<.001) was only different between the expert-rated stage 1a and stage 1b+ participants (Multimedia Appendix 1).


    Principal Findings

    This study demonstrates that a digital algorithm [41] can differentiate individuals in very early stages of mental illness (stage 1a) from those with increased risk of illness progression or more developed syndromes (stage 1b+) based on a web-based multidimensional self-report assessment (accuracy 91%; κ=0.67). Validation of this tool provides support for its further evaluation and use in services for stratification, which may help youth mental health services to reduce unnecessary delays for assessment and treatment, as well as enhance the quality of care.

    Evaluation of the Digital Algorithm

    Our results show that the algorithm was more conservative when assessing clinical stage, indicating that it had a greater tendency to assign lower and less severe clinical stage (ie, stage 1a) compared to the experts (Table 2). Participants who were rated higher by the experts (stage 1b+) than the algorithm (stage 1a) displayed greater functional impairment, but lower levels of depression and anxiety symptoms. This indicates that while the algorithm can detect symptom severity (eg, depression and anxiety symptoms), it may not be as sensitive to other multidimensional factors (eg, functioning) that influence the risk of illness progression [21]. However, the conservativeness of the digital algorithm aligns with the recommended practice of allocating lower stages based on uncertainty [20]. While, in practice, all digitally allocated clinical stages should be reviewed by clinicians, the presented digital algorithm has demonstrated the capability to differentiate young people in the early stages of illness from those at later stages.

    Clinical Usability of the Clinical Staging Algorithm in Mental Health Services

    When used in mental health services, the digital algorithm validated here could promote standardization and implementation of clinical staging at scale. Traditionally, clinical staging assessments require an intensive assessment by a clinician, which takes significant time and resources. For this reason, the heuristic has limited use, particularly when considering the large demand for services. The digitalization of clinical staging proposes a potential solution to this problem by condensing a large volume of biopsychosocial measures in a self-reported assessment into a clinical stage, which can then be translated into actionable treatment strategies. The major usability of this digital algorithm focuses on the differentiation of stage 1a from stage 1b+. Young people in stage 1b and stage 2+ may be similar across many clinical characteristics, making it difficult to differentiate without further assessment. However, most stage 1a cases could readily be distinguished from the population. The large degree of concordance here supports this hypothesis and illustrates how the digital algorithm could be used to direct young people to the appropriate level of care. Young people in stage 1a could be directed to web-based, self-directed, or brief clinician-supported resources (eg, web-based CBT or psychoeducation [65-67]), while using digital technologies to track their symptoms and monitor any changes (Figure 1 [68,69]). Concurrently, access to early intervention and further assessments can be expedited for individuals with attenuated or more developed syndromes (stage 1b+) so that a decision can be made about the type and intensity of care required for these individuals, who are more likely to have complex presentations.

    ‎
    Figure 1. Care pathway transformation using the digital algorithm.

    The proportion of stage 1a represented in this cohort was smaller than previous reports on clinical stage presentation at youth mental health services with larger sample sizes [70-72]. The lowest estimate is 15% (the current sample), but other independent samples suggest that the proportion could be as high as 30% [21,70,71]. This means that based on the digital algorithm’s accuracy (91%) and F1-score (73%), between 11% and 27% of total presentations to youth mental health services can accurately be allocated to stage 1a. This could prove to be a beneficial tool for services to manage demand and stratify young people to the appropriate level of care. Directing young people at stage 1a to low-intensity interventions may reduce unnecessary delays in accessing evidence-based treatments that are most likely to be effective for their stage of illness. This may then facilitate access to early interventions for those young people with an increased risk of mental illness progression.

    Limitations

    There are several limitations to the current study. First, this study only assessed the interrater reliability between the digital algorithm and 2 expert psychiatrists’ clinical stages. A team-based assessment of clinical stage should be the focus of future studies to provide stronger validation of the digital algorithm [73]. Second, the sample may not be representative of the general population presenting to youth mental health services. For example, the age difference between stage 1a and stage 1b+, which has previously been reported [18], was not reflected in this study. Future studies would benefit from validating the algorithm in other representative samples. However, the impact of representativeness on the analyses reported here is minimal since the algorithm is rule-based and expert assessors were drawing upon their own experience. Lastly, the digital algorithm was validated by assessing concordance with experts who used the same questionnaire and assessment results. The chosen methodology aims to focus on comparing algorithm performance against the information available to clinicians who are assessing the needs of young people before a face-to-face assessment. Therefore, it was important for the algorithm to align with clinical judgment based on the same questionnaire. Future validation should focus on comparing performance to a separate clinical interview, which would provide support for its wider clinical usability in services.

    Conclusions

    This study validates a digital algorithm for clinical staging. We present a digital health solution for managing demand in current youth mental health services by applying clinical staging to allocate care according to an individual’s risk of illness progression. This work provides preliminary evidence for the use of the digital algorithm as a stratification tool for efficient treatment allocation. There are many avenues for future research to further the development and evaluation of this algorithm, which includes assessing the longitudinal outcomes of young people stratified into each group and identifying its impact on treatment outcomes and waitlist management in youth mental health settings.

    Acknowledgments

    The authors would like to thank all the young people who participated in this study and all the staff in the Youth Mental Health Team at the University of Sydney’s Brain and Mind Centre, past and present, who contributed to this work.

    The funding sources of this study have had no input into the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. This work was supported by the Medical Research Future Fund Applied Artificial Intelligence in Health Care grant (MRFAI000097). MKC and WC were supported by the Australian Government RTP Scholarship. IBH is supported by a NHMRC L3 Investigator Grant (GNT2016346). FI was supported by the National Health and Medical Research Council (NHMRC) Emerging Leadership Fellowship (GNT2018157).

    Data Availability

    The data presented in this manuscript are available upon request.

    Conflicts of Interest

    TAD is now Director (Research and Evaluation) of the Design and Strategy Division, Australian Digital Health Agency.

    EMS is Principal Research Fellow at the Brain and Mind Centre, The University of Sydney. She is Discipline Leader of Adult Mental Health, School of Medicine, University of Notre Dame, and a Consultant Psychiatrist. She was the Medical Director, Young Adult Mental Health Unit, St Vincent’s Hospital Darlinghurst until January 2021. She has received honoraria for educational seminars related to the clinical management of depressive disorders supported by Servier, Janssen and Eli-Lilly pharmaceuticals. She has participated in a national advisory board for the antidepressant compound Pristiq, manufactured by Pfizer. She was the National Coordinator of an antidepressant trial sponsored by Servier.

    IBH is the Co-Director, Health and Policy at the Brain and Mind Centre (BMC) University of Sydney. The BMC operates an early-intervention youth services at Camperdown under contract to headspace. He is the Chief Scientific Advisor to, and a 3.2% equity shareholder in, InnoWell Pty Ltd. InnoWell was formed as a joint venture by the University of Sydney (32% equity) and PwC (Australia; 32% equity) to deliver the Aus $30 million (equivalent to US $19.6 million) Australian Government funded Project Synergy (2017-20; a 3-year program for the transformation of mental health services) and to lead transformation of mental health services internationally through the use of innovative technologies.

    Multimedia Appendix 1

    Differences in demographic, clinical, and functional characteristics between clinical stages assessed by experts and the digital algorithm.

    DOCX File , 32 KB
    1. GBD 2019 Mental Disorders Collaborators. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Psychiatry. 2022;9(2):137-150. [FREE Full text] [CrossRef] [Medline]
    2. GBD 2019 Diseases and Injuries Collaborators. Global burden of 369 diseases and injuries in 204 countries and territories, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. 2020;396(10258):1204-1222. [FREE Full text] [CrossRef] [Medline]
    3. Malla A, Shah J, Iyer S, Boksa P, Joober R, Andersson N, et al. Youth mental health should be a top priority for health care in Canada. Can J Psychiatry. 2018;63(4):216-222. [FREE Full text] [CrossRef] [Medline]
    4. Barak-Corren Y, Castro VM, Nock MK, Mandl KD, Madsen EM, Seiger A, et al. Validation of an electronic health record-based suicide risk prediction modeling approach across multiple health care systems. JAMA Netw Open. 2020;3(3):e201262. [FREE Full text] [CrossRef] [Medline]
    5. National study of mental health and wellbeing. Australian Bureau of Statistics. 2022. URL: https:/​/www.​abs.gov.au/​statistics/​health/​mental-health/​national-study-mental-health-and-wellbeing/​latest-release [accessed 2023-08-03]
    6. Cross SPM, Hermens DF, Hickie IB. Treatment patterns and short-term outcomes in an early intervention youth mental health service. Early Interv Psychiatry. 2016;10(1):88-97. [CrossRef] [Medline]
    7. Correll CU, Galling B, Pawar A, Krivko A, Bonetto C, Ruggeri M, et al. Comparison of early intervention services vs treatment as usual for early-phase psychosis: a systematic review, meta-analysis, and meta-regression. JAMA Psychiatry. 2018;75(6):555-565. [FREE Full text] [CrossRef] [Medline]
    8. McGorry PD, Mei C, Chanen A, Hodges C, Alvarez-Jimenez M, Killackey E. Designing and scaling up integrated youth mental health care. World Psychiatry. 2022;21(1):61-76. [FREE Full text] [CrossRef] [Medline]
    9. Bower P, Gilbody S. Stepped care in psychological therapies: access, effectiveness and efficiency. Narrative literature review. Br J Psychiatry. 2005;186:11-17. [CrossRef] [Medline]
    10. McGorry PD, Tanti C, Stokes R, Hickie IB, Carnell K, Littlefield LK, et al. headspace: Australia's National Youth Mental Health Foundation--where young minds come first. Med J Aust. 2007;187(S7):S68-S70. [CrossRef] [Medline]
    11. Stunden C, Zasada J, VanHeerwaarden N, Hollenberg E, Abi-Jaoudé A, Chaim G, et al. Help-seeking behaviors of transition-aged youth for mental health concerns: qualitative study. J Med Internet Res. 2020;22(10):e18514. [FREE Full text] [CrossRef] [Medline]
    12. DiMino J, Blau G. The relationship between wait time after triage and show rate for intake in a nonurgent student population. J Coll Stud Psychother. 2012;26(3):241-247. [CrossRef]
    13. Drake RJ, Husain N, Marshall M, Lewis SW, Tomenson B, Chaudhry IB, et al. Effect of delaying treatment of first-episode psychosis on symptoms and social outcomes: a longitudinal analysis and modelling study. Lancet Psychiatry. 2020;7(7):602-610. [FREE Full text] [CrossRef] [Medline]
    14. Patel R, Shetty H, Jackson R, Broadbent M, Stewart R, Boydell J, et al. Delays before diagnosis and initiation of treatment in patients presenting to mental health services with bipolar disorder. PLoS One. 2015;10(5):e0126530. [FREE Full text] [CrossRef] [Medline]
    15. Gao CX, McDonald LP, Hamilton MP, Simons K, Menssink JM, Filia K, et al. Inequalities in access to mental health treatment by Australian Youths during the COVID-19 pandemic. Psychiatr Serv. 2023;74(6):581-588. [CrossRef] [Medline]
    16. Newlove-Delgado TMF, Williams T, Mandalia D, Davis J, McManus S, Savic M, et al. Mental Health of Children and Young People in England. Leeds. NHS Digital; 2022. URL: https:/​/digital.​nhs.uk/​data-and-information/​publications/​statistical/​mental-health-of-children-and-young-people-in-england/​2022-follow-up-to-the-2017-survey#:~:text=​Key%20findings,between%202020%2C%202021%20and%202022 [accessed 2023-08-03]
    17. Patel K, Robertson E, Kwong ASF, Griffith GJ, Willan K, Green MJ, et al. Psychological distress before and during the COVID-19 pandemic among adults in the United Kingdom based on coordinated analyses of 11 longitudinal studies. JAMA Netw Open. 2022;5(4):e227629. [FREE Full text] [CrossRef] [Medline]
    18. Sawrikar V, Stewart E, LaMonica HM, Iorfino F, Davenport TA, Cross S, et al. Using staged care to provide "Right Care First Time" to people with common affective disorders. Psychiatr Serv. 2021;72(6):691-703. [CrossRef] [Medline]
    19. Iorfino F, Piper SE, Prodan A, LaMonica HM, Davenport TA, Lee GY, et al. Using digital technologies to facilitate care coordination between youth mental health services: a guide for implementation. Front Health Serv. 2021;1:745456. [FREE Full text] [CrossRef] [Medline]
    20. Hickie IB, Scott EM, Hermens DF, Naismith SL, Guastella AJ, Kaur M, et al. Applying clinical staging to young people who present for mental health care. Early Interv Psychiatry. 2013;7(1):31-43. [CrossRef] [Medline]
    21. Iorfino F, Scott EM, Carpenter JS, Cross SP, Hermens DF, Killedar M, et al. Clinical stage transitions in persons aged 12 to 25 years presenting to early intervention mental health services with anxiety, mood, and psychotic disorders. JAMA Psychiatry. 2019;76(11):1167-1175. [FREE Full text] [CrossRef] [Medline]
    22. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Arlington, VA. American Psychiatric Association; 2013.
    23. International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM). National Center for Health Statistics. 2012. URL: http://www.cdc.gov/nchs/icd/icd10cm.htm [accessed 2022-09-01]
    24. Capon W, Hickie IB, Varidel M, Prodan A, Crouse JJ, Carpenter JS, et al. Clinical staging and the differential risks for clinical and functional outcomes in young people presenting for youth mental health care. BMC Med. 2022;20(1):479. [FREE Full text] [CrossRef] [Medline]
    25. Conus P, Ward J, Lucas N, Cotton S, Yung AR, Berk M, et al. Characterisation of the prodrome to a first episode of psychotic mania: results of a retrospective study. J Affect Disord. 2010;124(3):341-345. [CrossRef] [Medline]
    26. Scott EM, Hermens DF, Naismith SL, Guastella AJ, De Regt T, White D, et al. Distinguishing young people with emerging bipolar disorders from those with unipolar depression. J Affect Disord. 2013;144(3):208-215. [CrossRef] [Medline]
    27. Cross SPM, Scott J, Hickie IB. Predicting early transition from sub-syndromal presentations to major mental disorders. BJPsych Open. 2017;3(5):223-227. [FREE Full text] [CrossRef] [Medline]
    28. McGorry PD, Nelson B, Goldstone S, Yung AR. Clinical staging: a heuristic and practical strategy for new research and better health and social outcomes for psychotic and related mood disorders. Can J Psychiatry. 2010;55(8):486-497. [CrossRef] [Medline]
    29. Shah JL, Scott J, McGorry PD, Cross SPM, Keshavan MS, Nelson B, et al. International Working Group on Transdiagnostic Clinical Staging in Youth Mental Health. Transdiagnostic clinical staging in youth mental health: a first international consensus statement. World Psychiatry. 2020;19(2):233-242. [FREE Full text] [CrossRef] [Medline]
    30. Hickie IB, Scott EM, Cross SP, Iorfino F, Davenport TA, Guastella AJ, et al. Right care, first time: a highly personalised and measurement-based care model to manage youth mental health. Med J Aust. 2019;211(Suppl 9):S3-S46. [CrossRef] [Medline]
    31. Delgadillo J, Ali S, Fleck K, Agnew C, Southgate A, Parkhouse L, et al. Stratified care vs stepped care for depression: a cluster randomized clinical trial. JAMA Psychiatry. 2022;79(2):101-108. [FREE Full text] [CrossRef] [Medline]
    32. Delgadillo J, Moreea O, Lutz W. Different people respond differently to therapy: a demonstration using patient profiling and risk stratification. Behav Res Ther. 2016;79:15-22. [CrossRef] [Medline]
    33. Fletcher S, Spittal MJ, Chondros P, Palmer VJ, Chatterton ML, Densley K, et al. Clinical efficacy of a decision support tool (Link-me) to guide intensity of mental health care in primary practice: a pragmatic stratified randomised controlled trial. Lancet Psychiatry. 2021;8(3):202-214. [CrossRef] [Medline]
    34. Balcombe L, De Leo D. Human-computer interaction in digital mental health. Informatics. 2022;9(1):14. [FREE Full text] [CrossRef]
    35. Iorfino F, Occhipinti JA, Skinner A, Davenport T, Rowe S, Prodan A, et al. The impact of technology-enabled care coordination in a complex mental health system: a local system dynamics model. J Med Internet Res. 2021;23(6):e25331. [FREE Full text] [CrossRef] [Medline]
    36. Balcombe L, De Leo D. Digital mental health challenges and the horizon ahead for solutions. JMIR Ment Health. 2021;8(3):e26811. [FREE Full text] [CrossRef] [Medline]
    37. Srinivasan B, Finkelstein JL, Erickson D, Mehta S. Point-of-care quantification of serum alpha-fetoprotein for screening birth defects in resource-limited settings: proof-of-concept study. JMIR Biomed Eng. 2021;6(1):e23527. [FREE Full text] [CrossRef] [Medline]
    38. Boutilier JJ, Chan TCY, Ranjan M, Deo S. Risk stratification for early detection of diabetes and hypertension in resource-limited settings: machine learning analysis. J Med Internet Res. 2021;23(1):e20123. [FREE Full text] [CrossRef] [Medline]
    39. Jin H, Wu S. Text messaging as a screening tool for depression and related conditions in underserved, predominantly minority safety net primary care patients: validity study. J Med Internet Res. 2020;22(3):e17282. [FREE Full text] [CrossRef] [Medline]
    40. Bradford S, Rickwood D. Young people's views on electronic mental health assessment: prefer to type than talk? J Child Fam Stud. 2015;24(5):1213-1221. [FREE Full text] [CrossRef] [Medline]
    41. Iorfino F, Cheng VWS, Cross SP, Yee HF, Davenport TA, Scott EM, et al. Right care, first time: developing a theory-based automated protocol to help clinically stage young people based on severity and persistence of mental illness. Front Public Health. 2021;9:621862. [FREE Full text] [CrossRef] [Medline]
    42. Iorfino F, Cross SP, Davenport T, Carpenter JS, Scott E, Shiran S, et al. A digital platform designed for youth mental health services to deliver personalized and measurement-based care. Front Psychiatry. 2019;10:595. [FREE Full text] [CrossRef] [Medline]
    43. Kessler RC, Andrews G, Colpe LJ, Hiripi E, Mroczek DK, Normand SLT, et al. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychol Med. 2002;32(6):959-976. [CrossRef] [Medline]
    44. Altman EG, Hedeker D, Peterson JL, Davis JM. The Altman Self-Rating Mania Scale. Biol Psychiatry. 1997;42(10):948-955. [CrossRef] [Medline]
    45. Ising HK, Veling W, Loewy RL, Rietveld MW, Rietdijk J, Dragt S, et al. The validity of the 16-item version of the Prodromal Questionnaire (PQ-16) to screen for ultra high risk of developing psychosis in the general help-seeking population. Schizophr Bull. 2012;38(6):1288-1296. [FREE Full text] [CrossRef] [Medline]
    46. Moore SA, Welch SS, Michonski J, Poquiz J, Osborne TL, Sayrs J, et al. Psychometric evaluation of the Overall Anxiety Severity And Impairment Scale (OASIS) in individuals seeking outpatient specialty treatment for anxiety-related disorders. J Affect Disord. 2015;175:463-470. [CrossRef] [Medline]
    47. Rush AJ, Trivedi M, Carmody TJ, Biggs MM, Shores-Wilson K, Ibrahim H, et al. One-year clinical outcomes of depressed public sector outpatients: a benchmark for subsequent studies. Biol Psychiatry. 2004;56(1):46-53. [CrossRef] [Medline]
    48. van Spijker BAJ, Batterham PJ, Calear AL, Farrer L, Christensen H, Reynolds J, et al. The suicidal ideation attributes scale (SIDAS): community-based validation study of a new scale for the measurement of suicidal ideation. Suicide Life Threat Behav. 2014;44(4):408-419. [CrossRef] [Medline]
    49. Whitlock J, Exner-Cortens D, Purington A. Assessment of nonsuicidal self-injury: development and initial validation of the Non-Suicidal Self-Injury-Assessment Tool (NSSI-AT). Psychol Assess. 2014;26(3):935-946. [CrossRef] [Medline]
    50. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011;168(12):1266-1277. [FREE Full text] [CrossRef] [Medline]
    51. Mundt JC, Marks IM, Shear MK, Greist JH. The Work and Social Adjustment Scale: a simple measure of impairment in functioning. Br J Psychiatry. 2002;180:461-464. [CrossRef] [Medline]
    52. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med. 1998;158(16):1789-1795. [CrossRef] [Medline]
    53. WHO ASSIST Working Group. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): development, reliability and feasibility. Addiction. 2002;97(9):1183-1194. [FREE Full text] [CrossRef] [Medline]
    54. DeWit DJ, Adlaf EM, Offord DR, Ogborne AC. Age at first alcohol use: a risk factor for the development of alcohol disorders. Am J Psychiatry. 2000;157(5):745-750. [CrossRef] [Medline]
    55. Chartier KG, Hesselbrock MN, Hesselbrock VM. Development and vulnerability factors in adolescent alcohol use. Child Adolesc Psychiatr Clin N Am. 2010;19(3):493-504. [FREE Full text] [CrossRef] [Medline]
    56. Connor JP, Haber PS, Hall WD. Alcohol use disorders. The Lancet. 2016;387(10022):988-998. [CrossRef]
    57. Hay PJ, Mond J, Buttner P, Darby A. Eating disorder behaviors are increasing: findings from two sequential community surveys in South Australia. PLoS One. 2008;3(2):e1541. [FREE Full text] [CrossRef] [Medline]
    58. Buysse DJ, Reynolds CF, Monk TH, Berman SR, Kupfer DJ. The Pittsburgh Sleep Quality Index: a new instrument for psychiatric practice and research. Psychiatry Res. 1989;28(2):193-213. [CrossRef] [Medline]
    59. Roenneberg T, Wirz-Justice A, Merrow M. Life between clocks: daily temporal patterns of human chronotypes. J Biol Rhythms. 2003;18(1):80-90. [FREE Full text] [CrossRef] [Medline]
    60. Vernon MK, Dugar A, Revicki D, Treglia M, Buysse D. Measurement of non-restorative sleep in insomnia: a review of the literature. Sleep Med Rev. 2010;14(3):205-212. [CrossRef] [Medline]
    61. RStudio Team. RStudio: Integrated Development for R. R version 4.2.1 (2022-06-23) ed. 2022. URL: https://ptds.samorso.ch/tutorials/r_install/ [accessed 2023-08-03]
    62. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 2016;20(1):37-46. [CrossRef]
    63. Ting KM. Confusion matrix. In: Sammut C, Webb GI, editors. Encyclopedia of Machine Learning and Data Mining. Boston. Springer US; 2017;260.
    64. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360-363. [FREE Full text] [Medline]
    65. White V, Linardon J, Stone JE, Holmes-Truscott E, Olive L, Mikocka-Walus A, et al. Online psychological interventions to reduce symptoms of depression, anxiety, and general distress in those with chronic health conditions: a systematic review and meta-analysis of randomized controlled trials. Psychol Med. 2022;52(3):548-573. [CrossRef] [Medline]
    66. Barak A, Hen L, Boniel-Nissim M, Shapira N. A comprehensive review and a meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. Journal of Technology in Human Services. 2008;26(2-4):109-160. [CrossRef]
    67. Arean PA, Hallgren KA, Jordan JT, Gazzaley A, Atkins DC, Heagerty PJ, et al. The use and effectiveness of mobile apps for depression: results from a fully remote clinical trial. J Med Internet Res. 2016;18(12):e330. [FREE Full text] [CrossRef] [Medline]
    68. Davenport TA, LaMonica HM, Whittle L, English A, Iorfino F, Cross S, et al. Validation of the InnoWell Platform: protocol for a clinical trial. JMIR Res Protoc. 2019;8(5):e13955. [FREE Full text] [CrossRef] [Medline]
    69. Rauseo-Ricupero N, Henson P, Agate-Mays M, Torous J. Case studies from the digital clinic: integrating digital phenotyping and clinical practice into today's world. Int Rev Psychiatry. 2021;33(4):394-403. [CrossRef] [Medline]
    70. Purcell R, Jorm AF, Hickie IB, Yung AR, Pantelis C, Amminger GP, et al. Demographic and clinical characteristics of young people seeking help at youth mental health services: baseline findings of the Transitions Study. Early Interv Psychiatry. 2015;9(6):487-497. [CrossRef] [Medline]
    71. Cross SPM, Hermens DF, Scott J, Salvador-Carulla L, Hickie IB. Differential impact of current diagnosis and clinical stage on attendance at a youth mental health service. Early Interv Psychiatry. 2017;11(3):255-262. [CrossRef] [Medline]
    72. Hamilton BA, Naismith SL, Scott EM, Purcell S, Hickie IB. Disability is already pronounced in young people with early stages of affective disorders: data from an early intervention service. J Affect Disord. 2011;131(1-3):84-91. [CrossRef] [Medline]
    73. Davenport TA, LaMonica HM, Rowe S, Sturgess J, Scott EM, Lee G, et al. Innovative preclinic triage system to guide Australians to the right mental health care first time. Aust Health Rev. 2021:578-583. [CrossRef] [Medline]

    ‎
    K10: Kessler Psychological Distress Scale
    NEET: not in education, employment, or training
    OASIS: Overall Anxiety Severity And Impairment Scale
    PQ-16: Prodromal Questionnaire-16
    QIDS: Quick Inventory Of Depressive Symptomatology-Self-Report
    SIDAS: Suicide Ideation Attributes Scale

    Edited by A Mavragani; submitted 18.12.22; peer-reviewed by L Balcombe, N Kozloff; comments to author 08.04.23; revised version received 31.05.23; accepted 26.06.23; published 08.09.23.

    Copyright

    ©Min K Chong, Ian B Hickie, Shane P Cross, Sarah McKenna, Mathew Varidel, William Capon, Tracey A Davenport, Haley M LaMonica, Vilas Sawrikar, Adam Guastella, Sharon L Naismith, Elizabeth M Scott, Frank Iorfino. Originally published in JMIR Formative Research (https://formative.jmir.org), 08.09.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.