Ethics approval was granted by The University of Melbourne’s Human Research Ethics Committee (ID: 22376). Each DMHS collected administrative service use data using its own consent process. The data analyzed in this study were provided by each DMHS in aggregate form and excluded potentially identifying individual information.

Mental Health Online only collects pre- and posttreatment mental health outcome data for discrete evaluations of various service aspects. Therefore, mental health outcomes collected during one such evaluation from September 2, 2021 to February 2, 2022, were directly provided by Mental Health Online and included in our effectiveness analyses. Details about how Mental Health Online collected these data and their analysis indicating this sample’s representativeness of their wider group of consumers (n=2318) of the clinician-supported (Therapist Assist) program are reported elsewhere [10].

We conducted descriptive analyses of existing routinely collected service use data provided in aggregate form by the 3 DMHSs to report on uptake by service type (assessment or treatment) and treatment type (clinician-supported or self-directed).

To examine the effect of treatment on mental health symptoms or functioning, we imported into Stata (StataCorp LLC) quantitative data from publications on the 3 DMHSs and the comparator programs (sample sizes, means and SDs at baseline and posttreatment, and whether the treatment was self-directed or clinician-supported; supplemented by additional evaluation data from Mental Health Online). Decreases in scores for the original measures of symptoms and disability, and increases in scores for measures of quality of life and functioning, indicated improvement. Because different measures were frequently used, we calculated the effect size as a standardized mean difference (specifically, Cohen d). This enabled us to compare pre- and posttreatment means for specific groups using a common metric. Cohen [23] recommends interpreting effect sizes of Cohen d=0.2 as “small,” 0.5 as “medium,” and 0.8 as “large” effect sizes. For ease of interpretation, all effect size estimates were reported as positive if posttreatment means showed improvements compared with pretreatment means, and negative if posttreatment means showed deterioration compared with pretreatment means.

Next, we generated a pooled effect size for each service’s clinician-supported and self-directed treatments. The pooled mental health effect sizes were compared with effect sizes we calculated for comparison treatments. Individual effect sizes were pooled using a random-effects meta-analysis and estimated using restricted maximum likelihood. All analyses were undertaken in Stata (v16; StataCorp LLC) [24].

Table 1 summarizes key uptake data for the 3 DMHSs across different periods.

Over 7 years from January 2015 to December 2021, 24,495 consumers completed an assessment (with or without treatment) with Mental Health Online. Overall, 17,916 consumers (who had or had not completed an assessment) enrolled in a treatment program, 2319 (13%) of whom registered to receive clinician support. From July 2018 to August 2021 (period during which session attendance was being documented in a readily collatable dataset), 815 consumers were allocated a clinician, 57% (463) of whom completed supported treatment.

For MindSpot over 9 years from January 2013 to December 2021, 133,447 assessments were completed and 27% (35,942) of consumers who completed an assessment enrolled in either clinician-supported or self-directed treatment programs. This should be interpreted in the context that around 67% (65,042/97,127) of MindSpot consumers report that an assessment is their primary need [25]. In the overall period of data provision, most consumers enrolled in clinician-supported treatment (34,390/35,942, 96%). From July 2019, when self-directed treatment was introduced, to December 2020, 83% (7597/9149) of consumers enrolled in clinician-supported treatment. Two-thirds of those who started clinician-supported treatment completed treatment.

Data from THIS WAY UP show that over 6 and a half years from July 2015 to December 2021, 149,002 consumers started the assessment, and 124,270 stage 1 (initial anonymous online questionnaire) and 71,069 stage 2 (clinician-supported) assessments were completed. Approximately 72,000 consumers enrolled in any treatment and 34,048 (47%) in clinician-supported treatment. Three-quarters of all enrollments (54,510/72,007) started treatment. Of consumers who started, 46% (24,989/54,510) completed treatment (ie, at least two-thirds of lessons).

Of consumers who enrolled in MindSpot and THIS WAY UP treatments, 85% (30,384/35,942) and 76% (54,510/72,007), respectively, commenced treatment (equivalent data were not available for Mental Health Online).

Figures 1-3 show the mental health and quality-of-life outcomes produced by Mental Health Online, Mindspot, and THIS WAY UP, respectively.

Figure 1 shows that Mental Health Online clinician-supported treatments produced large (Cohen d=0.95) and self-directed treatments produced medium (Cohen d=0.59) improvements in mental health symptom severity. However, treatment had small effects on quality of life irrespective of whether clinician-supported or self-directed (Cohen d=0.12 and Cohen d=0.28, respectively). There were significant differences in effect sizes between the 4 subgroups (P<.001). Studies in the same subgroup had similar effects (as demonstrated by low heterogeneity, with I²=0%‐35%; and no evidence against homogeneity, all P>.16).

Figure 2 shows that MindSpot clinician-supported treatments produced large to very large improvements in mental health symptom severity for the “all consumers” group. There are significant differences in effect sizes between the subgroups (P<.001), ranging from Cohen d=0.89 for migrants from non–English-speaking European countries to Cohen d=1.59 for Indigenous peoples. Additionally, clinician-supported treatment had small effects on functioning (Cohen d=0.35) as assessed by whole or part days out of role (using the K10+ [Kessler Psychological Distress 10 Plus-Item Scale]) [41]. The level of heterogeneity within subgroups varied substantially and ranged from approximately 0 (for Indigenous people, migrants from English-speaking backgrounds, migrants from the Middle East, older adults, and younger adults, all with low-moderate sample sizes) to extremely high (96% for the overall service user group for functioning). Because the sample sizes in the overall (all) service user groups were very large (>5000), confidence limits are very narrow, so the tests may be less relevant and useful for these groups. For example, the estimated effect sizes for mental health symptoms in the overall service user group were quite consistent, at Cohen d=1.42, Cohen d=1.39, and Cohen d=1.45. Heterogeneity was moderate at around 40%‐60% in subgroups with medium sample sizes, including migrants from Asia, non–English-speaking migrants from English-speaking countries, and migrants from Europe.

Figure 3 shows that THIS WAY UP clinician-supported treatments produced large (Cohen d=1.04) improvements in mental health symptom severity for adult consumers, across different age ranges, disorders (depression and anxiety disorders), and outcome measures. Clinician-supported treatment produced moderate effect size reductions in disability (Cohen d=0.48), which were similar across the adult lifespan and for different disorder types (depression and anxiety).

Significant differences were observed in pooled effect sizes between the mental health symptoms and disability subgroups (P<.001). High heterogeneity across age groups and between studies reporting on symptoms (I²=93% and P<.001) indicates differences in the magnitude of outcomes produced by THIS WAY UP clinician-supported treatment. This is partly due to the outcome measure, with larger effect sizes observed for the K10. Heterogeneity was much lower for the disability subgroup (I²=37% and P=.22), which means the age groups and studies in this subgroup produced outcomes of a similar magnitude.

Figure 4 presents effect sizes we calculated for each of the comparison treatments plotted against the pooled effect sizes we calculated for each of the DMHSs (0.95 for clinician-supported and 0.59 for self-directed treatment by Mental Health Online, 1.42 for clinician-supported treatment by MindSpot, and 1.04 for clinician-supported treatment by THIS WAY UP).

Most effect sizes were large, ranging from 0.95 (Mental Health Online and clinician-supported) to 1.46 (Better Access). Of these, the largest effect sizes (around 1.4) were observed for Better Access (delivered by registered psychologists), MindSpot (clinician-supported treatment), and New Access. One effect size was medium (Mental Health Online, self-directed, Cohen d=0.59) and 3 (all for Link-me) were small (ranging from −0.36 to 0.33). The Link-me effect sizes, which differ from all other services, are likely to have driven the observed heterogeneity. However, the aim of this study was to compare the magnitude of the effects between DMHSs and the comparators rather than to pool the effects across all included services.

Our findings should be interpreted in the context of several caveats. Most of our effectiveness findings were based on analysis of secondary data we extracted from purposively selected key peer-reviewed publications from the wealth of studies published by the DMHSs. However, it was beyond our scope to extract data from all their publications. Even if we used deidentified individual-level consumer data, some data elements are either lacking or not readily available in the data capture systems of the DMHSs (eg, nature and dose of clinician support for all 3 DMHSs and posttreatment outcome data for Mental Health Online are not captured). For example, around one-third of MindSpot consumers who enroll in clinician-supported treatment choose not to take up the clinician support component, and around one-third of consumers who enroll in self-directed treatment end up receiving clinician support, but the data system only captures clinician support at enrollment (N Titov, personal communication, March 15, 2022). Similarly, although in our pooled estimates of THIS WAY UP treatment effects, we labeled all treatments as “clinician-supported,” the extent of clinician support is unknown given that this service element is provided externally by the consumer’s own mental health professional. One study of THIS WAY UP reported the median number of clinician contacts was 1, and more than 50% of patients reported that they had no contact from their clinician while completing their online course [45]. Therefore, differences in findings between clinician-supported and self-directed treatments should be interpreted with caution.

Similarly, because of the real-world nature of the evaluation and the limited time frame available to us, it was not feasible to include a comparison group. As a result, we relied on published findings of diverse comparison mental health treatments and treatment-as-usual control groups who may not be representative of consumers using DMHSs.

Certainty and generalizability of effectiveness estimates are limited by complete-case analysis (excluding participants without pre-post outcome data) and heterogeneity of the pooled effects of each DMHS and comparator treatment. However, heterogeneity was expected given the variability across both the DMHSs and the comparator interventions—spanning a broad range of intensities and levels of mental health need.

Finally, despite the achievements of the 3 DMHSs, their impact has been reported in isolation from community needs analyses, prohibiting comment on the true magnitude of their achievements or their reach within the intended target consumer groups.

A key strength of this study is that it was conducted as one component of a larger evaluation that used multiple additional data sources including an environmental scan [11], an umbrella review [12], and consultations with a large number and broad range of stakeholders (consumers [13], DMHS providers, people with lived experience of mental health problems with or without experience using DMHSs, additional health professionals with or without experience using DMHSs such as GPs and mental health professionals, and other key mental health sector stakeholders) [10]. This enabled us to assess processes and impacts and to triangulate findings from a range of perspectives. Our collaborative approach with the 3 DMHSs and the Australian Department of Health (now the Department of Health, Disability and Ageing) optimizes the potential utilization of our findings.

It is recommended that Australia and other countries with similar health systems (ie, universal public coverage with a parallel private system) foster clinician-supported DMHSs as a treatment option by increasing funding to expand their availability, commissioning ongoing evaluation of their performance, and developing information campaigns for people with lived experience of mental illness and health professionals to increase awareness of clinician-supported DMHSs [8]. It is essential to understand how DMHSs work and sit in the broader mental health service system, particularly in contexts such as the COVID-19 pandemic in which traditional face-to-face services may not be tenable. Multipronged, innovative approaches may facilitate embedding DMHSs in the mental health system. Examples include collaborating and partnering with other parts of the mental health system and tertiary education, or developing on-site spaces for consumers to access DMHSs (eg, health clinics and other face-to-face health and community services) to improve access to DMHSs in low socioeconomic and rural and remote areas where internet connections may be unreliable or unaffordable.

Efforts to enhance and standardize data collection and outcome measurement across services and embedding evaluation as part of a continuous improvement approach within DMHSs will contribute to building the evidence base and improving our understanding of the role of clinician-supported DMHSs in the mental health service system. Evaluation of DMHSs should be ongoing, and findings shared to allow scaling up of service models, or components of service models which are effective. Standardized data collection and outcome measurement will improve the robustness of statistical analysis and maximize comparisons between DMHSs and other areas of the mental health system.

For example, data on dose and nature of clinician support may provide important insights about adherence and effectiveness, but these data are not captured in the existing data repository systems of the services we examined. Our review of reviews highlighted diversity in the type, format, and extent of support or guidance offered; and in the duration of treatment [12].

Assessing consumer outcomes at the commencement and the completion of treatment risks introducing a systematic bias in which people who drop out of treatment (potentially with poorer outcomes) are excluded from effectiveness analyses. However, a key strength of the way in which outcome data are collected by MindSpot and THIS WAY UP is that consumers are assessed using standardized outcome measures at each session. This means that analysis of their outcome data provides the opportunity to examine the effects of treatment for consumers who drop out of treatment.

Although robust evidence exists for the efficacy and effectiveness of using DMHIs to treat depression and anxiety in adults [12], further research is needed to explore the efficacy of these interventions for young people and traditionally underserviced groups (eg, Indigenous peoples, people from culturally and linguistically diverse backgrounds, and people who are LGBTQIA+); and other mental disorders (eg, psychotic disorders, personality disorders, and substance dependence) and comorbid conditions. For example, exploring whether consumers experiencing psychotic disorders, suicidal ideation, and/or more complex presentations benefit from clinician-supported DMHS treatment as a component of a suite of care (eg, medical follow-up and specialist mental health care), which for many does not include any evidence-based psychological treatment [55]. Additionally, the proportion of recovered or improved consumers postintervention should be considered in future studies since our study focused on average effects across samples.

Our environmental scan of DMHSs indicated that they are not suitable or the preferred choice for everyone, with some consumers preferring face-to-face services [11]. It also highlighted many opportunities for improving reach and use of DMHSs, making optimal use of technology, and embedding and sustaining DMHSs in the mental health service system. Therefore, future research is also needed to increase understanding of who engages with, adheres to, and benefits from DMHSs, and why. This work could contribute to comprehensive assessment and screening of consumers to identify consumers suited to and most likely to benefit from DMHSs. It may also help identify which people do not engage with DMHSs and how this might be addressed. Finally, notwithstanding the importance of clinician support in providing effective DMHSs, the role of technological advances, such as AI, in personalizing or tailoring DMHS delivery warrants consideration to address differences in consumer needs, promote adherence and positive outcomes, and potentially increase efficiency.

This study has shown that Mental Health Online, MindSpot, and THIS WAY UP provide consumers with a range of service offerings including assessment and treatment. They have provided these services free of charge to a substantial number of adult consumers and have become an integral part of Australia’s mental health care system. The outcome data, where available, show that treatments delivered by DMHSs are producing significant clinical improvement for consumers. The magnitude of improvement produced via clinician-supported treatment is comparable with more resource-intensive face-to-face treatment options. These services have the potential to be scalable and good value for money.

The DMHSs are largely intended to target consumers with depression and anxiety disorders who choose to use digital mental health treatment or who, for a range of reasons, may have limited or no access to alternative treatment options. DMHSs are not intended to serve Australia’s entire help-seeking population, which may be better served through other components of the mental health system (eg, the larger-scale Better Access program, state-funded public mental health services, and the not-for-profit sector). DMHSs are relatively new innovative elements of the Australian mental health care landscape and will become further embedded with time. DMHSs are contributing to ensuring that consumers get the right care at the right time and, importantly, in accordance with consumer needs and preferences.