Developing a Theory of Change for a Digital Youth Mental Health Service (Moderated Online Social Therapy): Mixed Methods Knowledge Synthesis Study

Introduction

Overview

Poor youth mental health is a global problem. Mental illness is the leading cause of disability worldwide [1], and 75% of all mental disorders have their onset before the age of 25 years, severely disrupting the social, emotional, and vocational transition to adulthood [2], with life-long consequences [3]. While significant progress and investment have been made worldwide in developing a youth mental health service system to meet these challenges, problems remain, including long waits for care, low levels of uptake and engagement once in care, and modest treatment effects for those who overcome access and engagement challenges [4]. Youth mental health services, like all mental health services, face a sequential and compounding series of challenges that ultimately lead to low levels of overall effectiveness relative to community needs.

Fewer than half of all young people with mental health conditions seek or receive evidence-based treatment [5-7]. Among the key structural barriers to effective care is that the demand for care outstrips the supply of qualified mental health professionals, resulting in both excessive wait times and premature discharge. Young people also tend to disengage early from interventions. About 42% of young people drop out of treatment by the third therapy session in face-to-face service settings [8], and those with more complex mental health problems have poor attendance [9]. Even when young people receive evidence-based psychological interventions, their effectiveness is often limited, with effect sizes ranging from small to moderate [10]. A review of 453 randomized controlled trials over 50 years revealed that psychological treatment effectiveness has not improved for youth anxiety and has decreased for youth depression [11]. Furthermore, despite the exceedingly high rates of diagnostic comorbidity and symptom heterogeneity seen in youth mental health services, most evidence-based psychological interventions tend to be single-disorder-focused, one-size-fits-all approaches derived from trials reporting group-level effects from selected participants [12]. In care settings and clinical trials, comorbid conditions are typically ignored, and disorders are treated as a rigid constellation of symptoms [13]. This approach has left us with interventions that work well for the “average” young person with a narrow and specific set of presenting difficulties, leading clinicians and services to personalize interventions through a trial-and-error approach [14]. After more than 50 years of randomized controlled trials focused on group-effect research, we simply do not know what works, for whom, and why [15].

The Role of Digital Technology in Youth Mental Health Care Reform

Digital technologies have the potential to enhance care access, engagement, and treatment effectiveness. Nearly 30 years of internet-delivered treatment research and service delivery for adults have demonstrated that these services significantly improve accessibility while producing engagement and effectiveness outcomes that are roughly equivalent to face-to-face psychological therapy [16-18]. Mental health system reviews have called for digital interventions to be “blended into service delivery,” which has now become a national and international policy priority [19,20]. The recent Australian Royal Commission into Victoria’s Mental Health System found that the mental health system is “antiquated” and “has failed to keep up to date with advances in digital technology.” It mandated a contemporary “system enabled by digital technology” to “improve accessibility and continuity of care” [21].

Much like face-to-face services, however, uptake, engagement, and adherence to digital interventions remain challenges [19]. The problem appears to be exacerbated in young people, where rates of engagement in established digital service offerings are lower than those in adults [18,22]. Additionally, there has been a historical, global failure to integrate digital interventions into real-world mental health services [19]. Innovative mental health solutions with matched implementation and integration plans are urgently required to effectively overcome these challenges.

Moderated Online Social Therapy

Moderated Online Social Therapy (MOST) [23] is a digital mental health platform (both a web interface and a dedicated app) for people aged between 12 and 25 years, designed to be integrated (blended) into established face-to-face youth mental health services, including primary care mental health services like “Headspace” and specialist child and adolescent mental health services [4]. Mental health professionals and researchers can refer young people to the platform to use any of the five interacting components: (1) evidence-based psychotherapeutic content, including psychoeducation, reflective activities, behavioral experiments, and comics; (2) additional remote clinician support; (3) remote peer support; (4) remote vocational or career consultation support; and (5) a peer-led web-based community and social network. These distinct components are designed to work cohesively together, enabling young people and their treating Mental health professionals to choose the kinds of support that best suit their needs. For instance, some young people might only access therapeutic content, while others might only use vocational support. However, the majority of users access more than two or more components, depending on their preferences and needs. In this way, MOST is a flexible service offering that can be adapted to a range of different preferences, needs, and service delivery contexts.

The flexibility of the platform allows MOST to be used differently by young people in youth mental health services at different phases in the care pathway (Figure 1). For instance, at the point of service entry, rather than waiting for a face-to-face mental health professional to be available, MOST can be used by the young person to gain immediate access to evidence-based care and remote professional and social support. At the point of receiving face-to-face care from a mental health professional, personalized clinical and psychosocial content can be recommended, homework can be scheduled, appealing strategies can be reflected on, and social or peer support can be provided between sessions. After care or at discharge, MOST can be used to maintain treatment gains, reduce relapse, or be an ongoing resource for treatment and support for those who drop out of care early. MOST can also be used for young people not connected to the formal care system at all, potentially alleviating demands on the mental health system and facilitating entry into mental health services as needed.

‎

Theory of Change

Improving access, engagement, and effectiveness of youth mental health services is a multifaceted endeavor that likely requires several interacting interventions. To explain how an intervention with multiple interacting components (known as a complex intervention) works, the Medical Research Council’s framework for the evaluation of complex interventions [24,25] recommends developing a theory of change (ToC) or a program logic. As a theory-driven evaluation technique, a ToC seeks to understand how and why a complex intervention works to guide evaluation and refinement [26]. Whereas a logic model is a description of the process of delivering an outcome through connecting inputs, activities, and outputs, a ToC overcomes a weakness of this approach by also articulating the assumptions and the theory or proposed explanations as to how the change process will occur [27]. A ToC entails a process of formulating how a complex intervention is supposed to work, why it works, who benefits and how, and what conditions are required for success. Through participatory techniques, a ToC produces pathways and quantifiable indicators that remain changeable as an intervention unfolds [28]. A ToC is often expressed visually in an accompanying narrative, showing the casual pathways and the evidence that links them, which can come from a variety of sources, including research evidence, behavior change theories, lived experience knowledge, or research conducted as part of the intervention feasibility and piloting stage [29].

Methods

Overview

MOST brings together several theoretical perspectives on behavior and therapeutic change in young people in the context of mental health challenges. Further, the ToC describes how a digital platform like MOST is designed to integrate within established face-to-face youth mental health services, thereby enhancing service provision and ultimately leading to a range of improved outcomes.

The MOST ToC provides a framework to evaluate the real-world impact of digitally enhancing a large nationwide network of youth mental health services and to hypothesize the therapeutic mechanisms and contextual factors that drive effectiveness in clinical settings, thus allowing iteration and refinement of the platform itself.

Over a 12-year period, more than 1000 consultation and co-design sessions have been conducted with young people, clinicians, and youth mental health services, and their findings have been reported within the various studies using the MOST platform as outlined below.

The development of the MOST ToC was conducted in 5 steps, as outlined below.

1. Findings from 2 completed and 4 ongoing randomized control trials and 11 pilot studies across 44 youth mental health services were reviewed. Over time, MOST has been customized to address several mental health conditions and associated challenges, including psychosis [30], ultra-high risk for psychosis [31], depression [32], anxiety [33,34], vocational recovery [35], and suicide risk [36], across all stages of treatment (ie, help-seeking, blended with face-to-face care, and relapse prevention). In summary, results from these trials demonstrate that MOST is (1) engaging, with 50% of young people with psychosis engaged over 9 months or longer and 70%-80% of young people across the diagnostic spectrum engaged for 3 months in shorter-term studies; (2) safe, with no incidents and serious adverse events over 11 years of testing; (3) endorsed by young people (95% of young people would recommend it to others) and clinicians (100% considered it helpful for young people); (4) effective in improving vocational recovery and reducing hospital admissions and visits to emergency services in young people with psychosis; (5) cost-effective and cost-saving [37]; and (6) promising in improving depression, anxiety, psychological distress, social functioning and support, loneliness, and well-being.
2. A review of qualitative findings from young people engaging in MOST in a variety of contexts was done; these included: MOST design and development with young people with first-episode psychosis [38]; MOST social networking use with active suicidal ideation [39]; social anxiety in psychosis [40]; social anxiety [41]; depression relapse prevention [42]; experience of intervention for first-episode psychosis [43,44]; and blended care experiences for first-episode psychosis [45].
3. A review of the recent digital mental health literature and evidence-based theories for guidance on improving digital engagement and adherence, improving outcomes for diagnostically diverse young people, and facilitating digital platform implementation in traditional services.
4. Examples of ToC frameworks in similar yet distinct youth web-based mental health programs [46] were undertaken, leading to the construction of a ToC framework (Textbox 1) that outlines the ToC components and describes how each element logically relates to the preceding and proceeding elements.
5. A final review and feedback from a range of MOST experts (study authors) who have variously been central in the development and service delivery of MOST over the last 12 years.

Ethical Considerations

This study is a synthesis of data from published research, so no ethics review was required.

Results

Overview

The above steps led to the production of the first version of the MOST ToC, which is presented in both narrative form and in a more detailed format (Textbox 2). Figure 1 summarizes the use cases of MOST at various phases of the care pathway.

Narrative Summary of the MOST Theory of Change

The mission of MOST is to integrate effective human professional support, be it remote or face-to-face, with digital health technology to personalize mental health care and services for young people. Technology-enhanced care centered around the needs and preferences of each young person can personalize mental health care.

There is growing interest in developing a new generation of integrated digital services in youth mental health services [4]. Various policy reviews report that the mental health system is fragile and not meeting the needs and preferences of the young people who rely on it [20,21,54]. When it comes to digital integration, these reviews share common findings: that the system is technologically antiquated and that the most obvious and impactful way to achieve true system reform is through the integrated use of digital technology [21]. Several assumptions exist, centered around the readiness and uptake of these integrated digital solutions by young people, clinicians, and service providers [55,56]. There is clearly a demand worldwide for digital mental health solutions [57]. However, young people have comparatively lower rates of uptake of such digital health tools compared to adults [18], and clinicians and service providers experience several barriers to digital adoption in their challenging service delivery environments [58]. These and other external factors include an unprecedented youth mental health need in the community [59], increasing competition in the digital mental health service landscape, with a focus on user engagement over outcomes [60], and a lack of infrastructure and change management for services and clinicians to integrate digital technologies into their clinical practice.

The challenges can be overcome with effective inputs and resources. They include a diverse and highly skilled workforce of clinicians, peer support workers, career consultants, implementation officers, software engineers, user experience and user interface designers, professional writers, artists, graphic designers, researchers, and operational specialists who work together to develop and maintain the service. MOST is delivered with financial support from various Australian state governments, as well as two major national foundations, numerous research grants, and partnership support from a number of national and international research partners.

Our key participants and stakeholders are young people, their families and caregivers, and the service clinicians and services that provide clinical care. Each of these participants faces a set of unique challenges and problems that MOST seeks to address. For young people, these include difficulties navigating the mental health system, a lack of timely access to mental health care and support [59], a dearth of other specialized supports such as peer and vocation support, and a lack of appropriate (matched to needs and preferences) [61], continuous [62], and effective care [11]. Multidisciplinary youth mental health teams, clinicians, and peer and vocational workers experience challenges related to managing large caseloads of young people with complex needs [63], keeping up with administrative and clinical demands, finding time to provide effective care between formal appointments, having effective tools to tailor therapy and track clinical and vocational outcomes [64], and limited integrated digital tools to support scarce service resources and clinical workflow management. As a result, long waiting times [63] and disengagement from care are common [8,9], and a lack of digital integration within and between services causes inefficiencies and additional challenges when implementing new technologies [58].

Theories and approaches guide both the implementation of MOST, that is, its integration within face-to-face services, and its therapeutic foundation. Implementation is guided by the “adaptive, tailored, and behavioral science-informed” (ATLAS) implementation strategy framework, bringing together gold-standard determinants such as Consolidated Framework for Implementation Research (CFIR) [49], Expert Recommendations for Implementing Change (ERIC) [50], and technology implementation (nonadoption, abandonment, scale-up, spread, and sustainability [NASSS]) frameworks [65]. Established theories and approaches to mental health support are evident in the literature, most notably self-determination theory (SDT) [66] and transdiagnostic, mechanistic, process-focused psychological therapy approaches [13,14,67,68]. Derived from these theories and emerging evidence, MOST targets key mechanisms of change, such as behavior change techniques used in the platform and by MOST clinicians, peer and vocational support to improve engagement [52], and transdiagnostic therapy targets, including but not limited to, repetitive negative thinking [69], cognitive and affective biases [70], experiential avoidance [71], and emotional dysregulation [72], to improve effectiveness. These mechanisms of change guide and inform the activities and interventions in MOST that include psychotherapeutic content, clinician support, peer support, a social network, vocational support, and implementation support, which are ultimately designed to interact and make an impact on the problems and challenges that bring a young person to the platform.

Measuring the outputs of these activities ensures that MOST is collecting and monitoring the kinds of data that link the activities with the problem targets. These outcomes are divided into short-, medium-, and long-term outcomes. While each of the activities is listed by component, they share common outputs and outcomes, such that unique combinations of these activities within MOST can achieve a diverse set of outcomes, from symptom reduction, social and functional recovery, and vocational attainment, to improved access, care continuity, service efficiency, and digital integration.

Discussion

Overview

As is common in ToC development, the creation of the first version of the MOST ToC helped us to connect the intended primary set of outcomes for the complex intervention, to articulate the inputs, activities, and outputs, and to make explicit the underlying assumptions and context of the intervention, as well as the theories and underlying mechanisms stakeholders believe will drive meaningful change. As a result, the ToC becomes a framework for ongoing evaluation and testing of MOST, helping identify areas for further development in MOST and iteratively improving the ToC itself, which we outline below.

Evolving the Therapeutic Content to Better Target Transdiagnostic Mechanisms

The construction of the ToC highlighted the need to improve treatment outcomes for young people with a range of different mental health problems and disorders, not just high-prevalence single disorders. This broader approach may be achieved by better targeting “transdiagnostic mechanistic constructs” at the level of the individual [73]. Transdiagnostic approaches may [14,74] be more precise, personalized, easy to use, and can be used with large-scale implementation of complex interventions [13,75]. Multiple meta-analyses also indicate that, relative to diagnostic-specific interventions, transdiagnostic protocols are at least as or more effective at addressing the primary clinical diagnosis, more effective at alleviating comorbid diagnoses, easier to implement, and more engaging [47,76-78].

MOST’s therapeutic content is grouped into web-based guided transdiagnostic or problem-specific “journeys” that contain evidence-based techniques from 2nd- and 3rd-wave cognitive behavioral therapies (CBTs), including CBT [79], acceptance and commitment therapy (ACT) [80], dialectical behavior therapy (DBT) [81], and mindfulness-based cognitive therapy (MCT) [82]. Within these journeys are specific skills-based activities that will better target a range of mechanisms outlined above and are directly supported by MOST and service clinicians, essentially blending their remote or face-to-face care with specific digital therapy content.

Future development of therapeutic content informed by the ToC will also seek to match content to specific transdiagnostic mechanisms as well as present problems. Further, it aims to use a more personalized approach rather than a single diagnosis and a group-informed approach to intervention by targeting individually specific mechanisms or transdiagnostic processes to improve engagement and outcomes, as well as make the intervention more efficient and less burdensome. In a newly funded clinical trial, we will test the effectiveness of this model using a wider range of mechanisms and carefully examine their interactions with individual characteristics, problems, and clinical outcomes in an effort to enhance treatment precision at the individual level.

Enhancing Engagement and Adherence: Developing Peer and Professional Support Alongside the Social Network

Contemporary behavioral science theories such as SDT are increasingly applied to web-based service contexts to enhance engagement and adherence in health behavior change efforts. SDT is the most prominent and empirically supported theory of human motivation that has demonstrated efficacy in predicting motivated behavior in multiple contexts and populations and for a variety of health behaviors such as physical activity, healthy eating, and smoking cessation [83]. It is a theory of human motivation that focuses on the quality of motivation, or the internal reasons that drive behavior. The quality of motivation is influenced by the extent to which individuals experience support for 3 basic psychological needs: autonomy, competence, and relatedness. Therefore, behaviors or messages from agents that support the satisfaction of these needs are likely to promote autonomous motivation and sustained behavior change within the individual, while those that do not may undermine autonomous motivation and lead to negative outcomes, such as disengagement or poor adherence. Teixeira et al [52] outlined 21 behavior change techniques drawn from SDT that can guide change agent behavior to promote sustained engagement in a health intervention, and these techniques shall guide peer, vocational, and clinician interactions with young people as well as the design of MOST, including the therapeutic content.

Aligned with the ToC, future improvements to professional and peer support as well as the social network will include alignment and fidelity to SDT and related motivation and behavior change techniques (MBCT) [52] to provide support that promotes engagement and adherence. Clinicians, peer workers, vocational consultants, and social network moderators will be trained in behavior change techniques, and newly developed fidelity measures will track compliance.

Improving Clinician and Service Adoption Through Implementation Science Frameworks

MOST is intended for integration within traditional youth mental health services to help address the multiple challenges experienced by the broader system. For example, integration of MOST at intake or entry can provide young people with immediate support while they would otherwise be waiting for care, or at discharge planning, where young people have ongoing access to relapse prevention support and content. Importantly, MOST can be used by clinicians in services, working in partnership with young people to augment their shared therapy goals and tasks with specific and relevant clinical content on the platform and concurrent remote peer and vocational support. In this way, MOST is an example of blended care [64]. There is evidence that blended care can be as effective as traditional therapy while requiring fewer sessions [84], potentially making it more scalable. Blended care may also improve engagement and compliance by linking digital tools to the client’s goals and providing a human component for building a therapeutic alliance.

The implementation of MOST in face-to-face services will be guided by a purpose-built framework (ATLAS) that combines several highly researched and validated implementation approaches (Textbox 2). ATLAS will apply CFIR to identify barriers and facilitators to integrating MOST, match determinates to behavior change and contextual constructs, and select operationalized behavioral change strategies matched to ERIC implementation strategies. These steps result in a cohesive implementation intervention delivered to key stakeholders, including service leaders, clinicians, and administrative personnel, and adapted at all levels overtime to address arising needs. Guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) outcome framework [48], these groups have been selected based on the primary implementation outcomes (reach and adoption), with service leaders having to endorse MOST’s adoption and integration, clinicians having to refer young people and adopt MOST in clinical practice, and administrative staff having to integrate MOST into services’ routine processes.

Integrating MOST in established clinical service environments is a key outcome in the ToC. Enhancing clinician and service adoption and the blended use of MOST in standard clinical care, as well as identifying facilitators, barriers, and the value of MOST implementation, are key areas for further development.

Strengths and Limitations

Using a ToC helps articulate the theories, mechanisms, and approaches drawn upon that are hypothesized to facilitate meaningful change. These hypotheses can be tested, allowing the ToC itself to iterate and be constantly aligned with the latest evidence of what is effective as well as discarding what is not effective in driving change. Despite being a time-intensive exercise, the development of a ToC has led to new areas of platform and service development that otherwise may not have been highlighted and has strengthened several other areas, including service quality, clinician fidelity, and the monitoring and evaluation approach. We argue that there is value in other digital services constructing their own ToC frameworks.

While these main strengths support the development of a ToC for MOST, there are also several limitations. First, there is no established or standardized methodology for developing a ToC. Further iterations of the ToC will also help determine the most effective and feasible way to consolidate the vast amounts of knowledge from stakeholders, data from the digital service, and the broader scientific literature. Second, while the value of codeveloping a ToC with stakeholders is self-evident, the costs and time to develop them may be prohibitive to many and also difficult to repeat regularly. Third, choosing between a myriad of theories and approaches for addressing common difficulties in youth mental health service provision may be subject to bias. The theories chosen here were based on those with the most substantial and compelling contemporary evidence that were most adaptable to the digital service model. In time, the data will ultimately support or refute the use of these theories, highlighting the importance of using a ToC framework to articulate how each step in the process will ultimately be evaluated.

Conclusions

MOST has been developed and tested over the past 12 years. By using a ToC framework, key processes within the complex intervention have been articulated that allow for future studies using MOST to capture key data points in their evaluations of effectiveness. Other complex interventions, including other multicomponent digital intervention platforms, may also benefit from developing a ToC to clearly articulate how the intervention is supposed to work, why it works, who benefits and how, and what conditions are required for success.