The in-person delivery of cognitive behavioral therapy (CBT) is an effective, evidence-based approach for treating a wide range of mental health conditions with a substantial amount of research indicating its efficacy for the treatment of anxiety [1] and depression [2]. In Canada, mild to moderate depression and anxiety disorders affect up to 11.7% and 10.8% of the adult population, respectively [3]. Although these mental health challenges affect many Canadians, there are numerous barriers to accessing evidence-based treatment. The availability of treatment is hindered by lengthy wait lists, therapists’ caseload limitations, and the moderate to low availability of clinicians trained in the delivery of evidence-based CBT [4]. The accessibility of treatment is further hindered by the monetary costs of treatment outside the public system and geographic access for those living in rural areas [5]. During the COVID-19 pandemic, additional challenges to accessing in-person treatment included physical distancing and self-isolation measures related to COVID-19, in addition to the sharp increase in stress, anxiety, and depression associated with the pandemic [6,7].

Previous research has found that internet-based CBT (iCBT) interventions provide increased access to evidence-based treatment, which can improve the mental health of the general population by effectively reducing symptoms associated with anxiety and depression [8,9]. However, few web-based interventions have been reviewed by therapy experts and have demonstrated treatment fidelity to core evidence-based CBT treatment protocols. Treatment fidelity is defined as “the extent to which an intervention is delivered as intended by the protocol” [10,11]. For an iCBT program targeting anxiety and depression, treatment fidelity indicates how closely the program follows the standard CBT approach for these mental health challenges and whether the necessary therapeutic interventions (eg, exposure and behavioral activation) are included. Bowie-DaBreo et al [12] described how there is significant uncertainty about the effectiveness of many mental health apps and programs, and much of that uncertainty appears to stem from a lack of information about treatment fidelity. In addition, few programs appeared to follow the guidelines for evidence-based care for specific challenges (eg, anxiety). Furthermore, a recent study examining mobile apps to treat depression [13] found that few programs followed a CBT approach, even though the literature supports CBT as the gold standard for the treatment of anxiety and depression [14]. In 2018, an international group of researchers developed a list of recommendations for the development of iCBT programs [15]—one of their core recommendations was the use of a co-design process (ie, involvement of intended users in the development of the iCBT intervention). Co-design has many benefits (eg, improved program tailoring) [16], yet this approach appears underused in the literature. In sum, there appears to be a gap in the literature pertaining to evidence-based iCBT programs that also follow the iCBT recommendations for development, especially for programs targeting co-occurring depression and anxiety [17].

Tranquility was designed to be an iCBT intervention that increases cognitive and behavioral skills (eg, thought challenging, exposure, and behavioral experiments) of individuals with mild to moderate anxiety. Users learn ways to manage their symptoms in addition to gaining access to personalized support through video, phone, and in-app messaging with a web-based coach. Results from a pilot program evaluation of Tranquility illustrated decreases in anxiety and stress levels in individuals who completed 3 or more modules, meaning that people who engaged with the program were noted to benefit, and most program users found the program to be helpful [18,19]. Though designed by a team including licensed psychologists with advanced skills in CBT techniques, researchers, and people with lived experience, Tranquility has not yet undergone a rigorous evaluation of its fidelity to core CBT components linked to improved client outcomes by a broader team of experts in this field. Furthermore, engaging mental health–informed peers (eg, people with lived experience of mental health challenges or peer counselors) in program development and evaluation could further illuminate usability characteristics (ie, aspects that contribute to the technical effectiveness and efficiency of a program) [20,21] that enable and increase engagement. Finally, Tranquility has the potential to be adapted for dual treatment streams related to anxiety and depression but requires both clinical and first voice input to map development.

This study aims to address these gaps by (1) evaluating Tranquility’s fidelity to CBT principles; (2) assessing CBT experts’ and mental health–informed peers’ perceptions of the usability of the Tranquility anxiety program and the likelihood they would recommend it to others; and (3) co-designing an adaptation of the Tranquility program to include treatment for depression with a group of mental health–informed peers.

To recruit focus group participants, relevant networks (eg, local university programs and peer counseling programs) were contacted and asked to disseminate study information to all local individuals on their newsletter mailing list. The focus group participants had to identify as a first voice advocate or have experience with mental health conditions in a near-peer role where they would have advanced knowledge of needs and experiences of the population for which Tranquility was designed (eg, peer mentors on university campuses). CBT experts were recruited through the authors’ professional networks of CBT clinicians and North American CBT organizations (eg, Canadian Association of Cognitive and Behavioural Therapies and Association for Behavioral and Cognitive Therapies), and individuals who met the study criteria were contacted directly via email. CBT experts had to have 5 years of experience delivering CBT for depression and anxiety in adults and must be licensed by a professional body. All participants were required to be ≥18 years of age.

We recruited 6 mental health–informed peers, referred to as focus group participants throughout the remainder of the paper, and 6 CBT experts to participate in the study. Sociodemographic information of all participants is presented in Table 1. Across both groups, participants were predominately White, heterosexual women. The mean age of the focus group was 39.25 (SD 8.88) years, and the mean age of experts was 47.6 (SD 14.72) years. CBT experts were entered into a draw to win a web-based gift card, while the focus group participants received an honorarium following each focus group. The study procedures were approved by the Research Ethics Board of the Nova Scotia Health Authority (file number 1025561) and conformed to the ethical standards of research set out by the Canadian Tri-Council Policy Statement 2 [22].

Upon completion of informed consent, all participants were provided with a video link outlining how to access and use Tranquility, and participants were then asked to review Tranquility in detail. To fulfill the 3 aims of the study, data collection occurred in 3 phases. In phase 1, following their review of Tranquility for anxiety, the CBT experts and focus group participants completed measures of usability (ie, SUS) and the likelihood to recommend (ie, Likelihood to Recommend scale). In addition, CBT experts evaluated Tranquility’s treatment fidelity to CBT. All measures were completed on the REDCap (Research Electronic Data Capture; Vanderbilt University) platform, a web-based data collection platform [32,33]. In phase 2, focus group participants participated in a co-design meeting with the researchers using the Zoom Professional platform (Zoom Video Communications Inc); discussions focused on existing anxiety components and adaptations for depression. In phase 3, focus group participants were presented with the survey results from phase 1 to provide a foundation for their feedback. The themes that emerged from the focus group meeting in phase 2 were shown to focus group participants to refine this information and correct any discrepancies. Next, focus group participants were presented with the adaptations made to Tranquility following the previous co-design meeting, and final feedback regarding the adaptations and the integration of anxiety and depression interventions in Tranquility was obtained.

As the purpose of the study was not to test a specific hypothesis or compare groups but rather to explore usability, assess treatment fidelity, and engage in the co-design process, data from both groups were pooled for all measures except for CBT fidelity measures only completed by CBT experts. Descriptive statistics were used to summarize the data. Focus group transcripts were analyzed using the principles of thematic content analysis, and we used a data-driven approach to inductively establish themes [34]. The themes were created using the guiding principle of selecting feedback that highlighted the strengths and weaknesses of the Tranquility program. Units of analysis included discrete words, sentences, and paragraphs. In line with the ECOUTER framework, mind maps were used to develop the conceptual schema of themes and subthemes; 2 experienced raters reviewed the anonymized interview transcripts and independently coded the data using Microsoft Excel [35]. Any discrepancies were resolved through discussion.

The participants’ ratings of the usability of Tranquility are presented in Table 2. Data were missing from several focus group members (n=4) on measures of usability, resulting in a final sample of 8 participants for the SUS measure. Most participants agreed that Tranquility was easy to use, and all participants agreed with the statement that they felt confident in their ability to use the Tranquility program and that the functions of Tranquility were well-integrated into the program. In contrast, there was less agreement about whether participants would use the program frequently in the future or whether most people would learn to use Tranquility quickly. When looking at reverse-coded items, it was apparent that most participants believed that program users would not need the support of a technical person nor would they need to learn much before being able to use the Tranquility program. Nearly all participants strongly agreed that inconsistency was not a problem across the Tranquility program. The SUS questionnaire results indicated that the average overall usability score was 76.56 (SD 14.07; range 52.5-97.5), which was above the evidence-based cut-off for program usability. Finally, the mean Likelihood to Recommend scale score was 7 (SD 1.07; range 5-8), indicating user satisfaction; 88% (11/12) of participants rated Tranquility as being within the satisfactory category, while 1 participant’s rating of Tranquility indicated dissatisfaction and a low likelihood to recommend.

CBT experts evaluated the fidelity of Tranquility to CBT for the treatment of anxiety. The average rating for fidelity to core CBT components was 91% (SD 11.14; range 75-100), and half of the CBT experts indicated that Tranquility included 100% of all required CBT components.

In terms of specific components, all 6 experts reported that Tranquility included clear explanations of both anxiety and the CBT model, including cognitive and behavioral techniques, and Tranquility provided formal ratings of anxiety (eg, 0 to 10 scale) to program users. In total, 4 of 5 (80%) CBT experts agreed that Tranquility included specific emotion monitoring, 1 of 5 (20%) therapists agreed that Tranquility included only some emotion monitoring, and 1 expert did not respond. Similarly, 5 of 6 (83%) experts agreed that Tranquility provided specific monitoring of cognitions, and 1 of 6 (17%) therapists indicated that Tranquility included only some monitoring of cognitions. This exact pattern of results was also seen when evaluating whether Tranquility provided a method to monitor behaviors. Only 4 of 6 (67%) CBT experts reported that Tranquility offered a way to monitor specific physical sensations and allowed for adequate case conceptualization of anxiety. For both physical sensation monitoring and case conceptualization, one CBT expert believed there was some tracking of physical sensation monitoring or case conceptualization, while another expert believed there was none.

This study represents an assessment of both usability and treatment fidelity to CBT and a co-design adaptation of an iCBT program. Overall, participants highly rated the usability of Tranquility and indicated satisfaction with the program, and CBT experts provided high ratings for Tranquility’s treatment fidelity to CBT. The co-design adaptation process resulted in several improvements to Tranquility.

Experts agreed that Tranquility had high fidelity to the CBT model and included the most necessary components for the treatment of anxiety. There was high agreement across most components (eg, inclusion of behavioral techniques), although to obtain perfect fidelity, Tranquility would need to make emotion, behavior, physical sensation, and cognition monitoring more explicit. These elevated fidelity ratings were not unexpected, as the Tranquility program development team included a licensed clinical psychologist with extensive expertise in CBT. Notably, a recent functionality analysis of apps for depression [36] found that of a possible 8 CBT components, 22% of apps only included ≥3 CBT components, 68% included only 1 or 2, and 10% of apps did not include any CBT components. Furthermore, only 45% of these apps had expert involvement (eg, health professionals) during the app development process. These findings highlight the gap that Tranquility fills with regard to fidelity to CBT components and the importance of fidelity evaluations by objective experts.

Across vignettes, CBT experts typically agreed on which CBT interventions were necessary to treat the depicted cases. Moreover, both cases required the same set of interventions, except for the addition of safety behavior psychoeducation and substance use tracking for the Helen vignette, given the depicted substance use. It was an expected finding that experts would strongly suggest using interventions such as behavioral activation, given its robust efficacy in depression treatment, including within an iCBT format [37]. The integration of therapeutic strategies for depression within an iCBT program for anxiety is likely to be beneficial, given the overlap between the symptoms and the core therapeutic strategies used to treat each disorder [38]. However, the addition of unique therapeutic elements also used to treat either anxiety or depression can overwhelm program users; individuals may benefit from external support to navigate and select the most helpful strategies. A recent literature review [39] suggested that guided (eg, coaches or therapist support) programs had similar rates of adherence as in-person CBT treatment, which is higher than self-guided programs—guidance appears to positively impact treatment program use [39,40].

Current guidelines for digital mental health interventions strongly suggest using a co-design process [41,42], and the results of this study illustrate the importance of that process—the mental health peers suggested crucial improvements to program customizability, delivery, accessibility, and available interventions. Previous research has found that collaborative study designs, such as the co-design approach (see Hill et al [15] for a review), can have a positive effect on aspects such as user adherence, usability, and uptake by involved stakeholders (eg, program users and clinicians).

Mental health programs targeting depression and anxiety need to have both high fidelity to the CBT treatment model and be engaging, flexible, and allow user personalization. Similarly, usability testing is critical, and Kushniruk [43] found that usability testing with small groups (ie, 8-10 users) can drastically reduce the number of usability issues experienced by users in the future. Usability testing within this study revealed areas upon which Tranquility program developers can improve, such as increasing user engagement to encourage users to continue to use this program frequently and reducing the initial learning curve of how to use Tranquility.

The findings of this study are in line with much of the iCBT literature—users want engagement with and control over their web-based therapeutic experiences. Stawarz et al [36] noted that although approximately 58% of CBT apps included at least 1 engagement strategy, they used a less-varied complement of engagement strategies compared with other kinds of mental health apps. Most CBT apps included strategies such as visual aids (eg, graphs and charts) but few included more complex strategies, for example, gamification and coaching or therapist chat functions. Stawarz et al [36] also conducted a qualitative analysis of app reviews, which highlighted the value of leveraging engagement strategies (eg, personalization) to increase user satisfaction with mental health programs [44]. Similarly, a qualitative study of engagement with an iCBT program found that participants wanted to choose what information to learn, accessible content (eg, audiovisual content and appropriate reading level), and a more tailored coaching experience [16,45]. In sum, users want personalized content and more complex engagement strategies.

The ECOUTER framework provided an accessible approach that centered potential users in the feedback process and provided a procedure guide, which was beneficial when recruiting and working collaboratively with focus group participants as well as during data analysis. Engaging in an adaptation co-design process significantly improved program design and delivery for this iCBT program because it permitted the integration of researchers’ expertise pertaining to the design and delivery of iCBT and participants’ expertise pertaining to mental health both as a clinician and as a client. Moreover, seeking usability feedback from all participants aided in the rigor of this mixed methods evaluation and co-design adaptation of Tranquility because it allowed for multiple perspectives to be considered as CBT experts and focus group members may have different experiences and expectations of iCBT programs and may place value on differing program elements measured within the usability questionnaire we used, the SUS. However, although this approach had many benefits, it did include challenges such as recruiting CBT experts. Many of the approached individuals declined to participate, likely due to demands in excess of their available time, and scheduling focus groups was especially challenging. The usability of the design used in this study, although challenging at times, provides a greater real-world approach to mental health program design and both fidelity and usability evaluation.

These findings have therapeutic implications for the iCBT literature. Taken together, future research and iCBT program development should consider the involvement of program users and clinical experts to ensure that fidelity to CBT and engagement strategies are implemented, given its association with adherence and program effectiveness [46]. The effectiveness of gamification and incentives offered within eHealth tools, including within iCBT programs, remains an understudied element of the literature [15], and future work should aim to assess the effectiveness of these user engagement strategies in addition to measuring client satisfaction. In addition, future work is needed to examine the effectiveness of iCBT interventions for comorbid mental health challenges, such as anxiety and depression, and the addition of iCBT protocols to address other challenges (insomnia, trauma, etc) may also help meet treatment demands for other comorbid presentations (eg, anxiety and insomnia).

Similarly, many studies, including this one, included predominantly White, heterosexual women. As a result, there are additional considerations related to race, gender, and sexual orientation that were not captured in the participant feedback or in the resultant changes to Tranquility (eg, using acceptance or values-based strategies vs cognitive restructuring for microaggressions; greater role for family or community members).

Understanding the perspectives and unique needs of marginalized groups is necessary to increase iCBT treatment accessibility and effectiveness—efforts should be made to include participants who are racialized and 2SLGBTQ+ (ie, two-spirit, lesbian, gay, bisexual, transgender, queer, and all other members of this community) and individuals who live in rural areas or are older adults. Efforts to recruit a larger and more diverse sample will afford a greater range of perspectives. Furthermore, this study included a small sample of CBT experts and focus group participants. This smaller sample was a significant strength of the study in that it afforded the focus group participants an opportunity to provide rich and in-depth perspectives on Tranquility and suggestions for adaptation. However, we recognize that the sample size does impact the generalizability of these findings to individuals who were underrepresented in the participant pool (eg, men, nonbinary individuals, or Black people) and who may or may not have been represented at all (eg, people with disabilities). Finally, it is of note that technological confidence and knowledge were not measured; therefore, it is possible that the ratings of this program were influenced by this variable.

CBT experts and mental health peers agreed that Tranquility, a web-based program treating anxiety and depression, had high usability, and both groups would be likely to recommend this program to others. CBT experts scored Tranquility as having high fidelity to CBT, and nearly all intervention components needed to treat depression were included as a part of Tranquility. Finally, the co-design process was key to refining the existing anxiety content and for the creation and integration of the new depression content. These results provide a preliminary evaluation of the Tranquility program, and they may provide user-centered engagement strategies that may help increase adherence and effectiveness for the iCBT treatment of anxiety and depression.