This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Internet-based cognitive behavioral therapy (iCBT) is a necessary step toward increasing the accessibility of mental health services. Yet, few iCBT programs have been evaluated for their fidelity to the therapeutic principles of cognitive behavioral therapy (CBT) or usability standards. In addition, many existing iCBT programs do not include treatments targeting both anxiety and depression, which are commonly co-occurring conditions.
This study aims to evaluate the usability of Tranquility—a novel iCBT program for anxiety—and its fidelity to CBT principles. This study also aims to engage in a co-design process to adapt Tranquility to include treatment elements for depression.
CBT experts (n=6) and mental health–informed peers (n=6) reviewed the iCBT program Tranquility. CBT experts assessed Tranquility’s fidelity to CBT principles and were asked to identify necessary interventions for depression by using 2 simulated client case examples. Mental health–informed peers engaged in 2 co-design focus groups to discuss adaptations to the existing anxiety program and the integration of interventions for depression. Both groups completed web-based surveys assessing the usability of Tranquility and the likelihood that they would recommend the program.
The CBT experts’ mean rating of Tranquility’s fidelity to CBT principles was 91%, indicating a high fidelity to CBT. Further, 5 out of 6 CBT experts and all mental health–informed peers (all participants: 11/12, 88%) rated Tranquility as satisfactory, indicating that they may recommend Tranquility to others, and they rated its usability highly (mean 76.56, SD 14.07). Mental health–informed peers provided suggestions on how to leverage engagement with Tranquility (eg, adding incentives and notification control).
This preliminary study demonstrated the strong fidelity of Tranquility to CBT and usability standards. The results highlight the importance of involving stakeholders in the co-design process and future opportunities to increase engagement.
The in-person delivery of cognitive behavioral therapy (CBT) is an effective, evidence-based approach for treating a wide range of mental health conditions with a substantial amount of research indicating its efficacy for the treatment of anxiety [
Previous research has found that internet-based CBT (iCBT) interventions provide increased access to evidence-based treatment, which can improve the mental health of the general population by effectively reducing symptoms associated with anxiety and depression [
Tranquility was designed to be an iCBT intervention that increases cognitive and behavioral skills (eg, thought challenging, exposure, and behavioral experiments) of individuals with mild to moderate anxiety. Users learn ways to manage their symptoms in addition to gaining access to personalized support through video, phone, and in-app messaging with a web-based coach. Results from a pilot program evaluation of Tranquility illustrated decreases in anxiety and stress levels in individuals who completed 3 or more modules, meaning that people who engaged with the program were noted to benefit, and most program users found the program to be helpful [
This study aims to address these gaps by (1) evaluating Tranquility’s fidelity to CBT principles; (2) assessing CBT experts’ and mental health–informed peers’ perceptions of the usability of the Tranquility anxiety program and the likelihood they would recommend it to others; and (3) co-designing an adaptation of the Tranquility program to include treatment for depression with a group of mental health–informed peers.
To recruit focus group participants, relevant networks (eg, local university programs and peer counseling programs) were contacted and asked to disseminate study information to all local individuals on their newsletter mailing list. The focus group participants had to identify as a first voice advocate or have experience with mental health conditions in a near-peer role where they would have advanced knowledge of needs and experiences of the population for which Tranquility was designed (eg, peer mentors on university campuses). CBT experts were recruited through the authors’ professional networks of CBT clinicians and North American CBT organizations (eg, Canadian Association of Cognitive and Behavioural Therapies and Association for Behavioral and Cognitive Therapies), and individuals who met the study criteria were contacted directly via email. CBT experts had to have 5 years of experience delivering CBT for depression and anxiety in adults and must be licensed by a professional body. All participants were required to be ≥18 years of age.
We recruited 6 mental health–informed peers, referred to as focus group participants throughout the remainder of the paper, and 6 CBT experts to participate in the study. Sociodemographic information of all participants is presented in
Participant demographics.
Demographics | Focus group participants (n=6) | CBTa experts (n=6) | |||
Age (years), mean (SD) | 39.25 (8.88) | 47.6 (14.72) | |||
|
|||||
|
Man | —c | 1 (17) | ||
|
Woman | 5 (83) | 5 (83) | ||
|
Preferred not to say | 1 (17) | — | ||
|
|||||
|
Heterosexual | 4 (66) | 6 (100) | ||
|
Bisexual | 1 (17) | — | ||
|
Preferred not to say | 1 (17) | — | ||
|
|||||
|
White | 5 (83) | 6 (100) | ||
|
Preferred not to say | 1 (17) | — |
aCBT: cognitive behavioral therapy.
bParticipants were provided with a comprehensive list of gender identities, sexual orientations, and racial identities from which to choose when reporting their demographic information—only those selected by at least 1 participant are listed in this table.
cNot available. These categories were not used by the focus group or cognitive behavioral therapy experts.
Mental health–informed peers were involved in 2 focus groups to co-design adaptations to Tranquility for both anxiety and depression and the benefit of these changes. A script with a combination of question types (eg, open-ended and close-ended questions) for the focus group was developed by the research team and can be found in
The System Usability Scale (SUS) [
Participants rated how likely they were to recommend Tranquility via a 1-item measure: “How likely are you to recommend Tranquility to your friends, family, or associates?” [
CBT experts completed questionnaires assessing demographics, the SUS, the Likelihood to Recommend scale, and Tranquility’s level of fidelity to CBT protocols for anxiety. They also followed clinical case vignettes through the program to detail the therapeutic components of CBT that are necessary to treat the depicted cases of depression.
To assess fidelity to CBT components for the treatment of anxiety, we adapted the evaluation criteria for web-based apps treating depression by Huguet et al [
We adapted 2 vignettes of individuals with depression (
CBT experts were asked which CBT components Tranquility should include to offer treatment for depression for the individual in the vignette. Each vignette had the same list of 19 components (eg, behavioral activation and mood tracking;
Upon completion of informed consent, all participants were provided with a video link outlining how to access and use Tranquility, and participants were then asked to review Tranquility in detail. To fulfill the 3 aims of the study, data collection occurred in 3 phases. In phase 1, following their review of Tranquility for anxiety, the CBT experts and focus group participants completed measures of usability (ie, SUS) and the likelihood to recommend (ie, Likelihood to Recommend scale). In addition, CBT experts evaluated Tranquility’s treatment fidelity to CBT. All measures were completed on the REDCap (Research Electronic Data Capture; Vanderbilt University) platform, a web-based data collection platform [
As the purpose of the study was not to test a specific hypothesis or compare groups but rather to explore usability, assess treatment fidelity, and engage in the co-design process, data from both groups were pooled for all measures except for CBT fidelity measures only completed by CBT experts. Descriptive statistics were used to summarize the data. Focus group transcripts were analyzed using the principles of thematic content analysis, and we used a data-driven approach to inductively establish themes [
The participants’ ratings of the usability of Tranquility are presented in
Usability ratings of the Tranquility program.
Usability componenta | Rating, mean (SD; range) | Agreement with this statement, n (%) | Disagreement with this statementb, n (%) |
I think that I would like to use the Tranquility program frequently | 3.75 (0.71; 3-5) | 5 (63) | N/Ac |
I thought that the Tranquility program was easy to use | 4.29 (0.76; 3-5) | 6 (86)d | N/A |
I found the various functions in the Tranquility program were well integrated | 4.25 (0.46; 4-5) | 8 (100) | N/A |
I would imagine that most people would learn to use the Tranquility program very quickly | 3.88 (0.99; 2-5) | 6 (75) | N/A |
I felt very confident using the Tranquility program | 4.13 (0.35; 4-5) | 8 (100) | N/A |
|
1.75 (1.04; 1-4) | N/A | 7 (86) |
|
2.13 (0.99; 1-4) | N/A | 6 (75) |
|
1.5 (0.53; 1-2) | N/A | 8 (100) |
|
1.75 (0.46; 1-2) | N/A | 8 (100) |
|
2.13 (0.83; 1-4) | N/A | 7 (86) |
aRatings ranged from
bReverse-coded items.
cN/A: not applicable.
dThere was missing data for 1 participant (n=7).
eItems that are reverse-scored (ie, low scores mean higher usability) are italicized.
Focus group feedback was organized into 8 themes and 8 subthemes (
As a part of the co-design process, changes were made to Tranquility across the existing anxiety components and the new depression components (
The focus group participants suggested providing the program user with more control over their experience and more support for engagement, resulting in increased control over aspects of notifications (eg, notification type), and ongoing developments for Tranquility include gamification and new incentives to foster engagement. Focus group participants asked that coaches initiate coaching appointments, help personalize the messaging experience with clients, and aid users in selecting content that is best suited to their needs. All requested changes applicable to coaching were made to foster connections with the coach and enrich the experience with Tranquility.
The focus group also encouraged the addition of information about program fit before beginning the program; Tranquility now begins with a screening assessment to ensure a good fit between the user’s identified needs and the Tranquility program before payment, and feedback is provided to the user during onboarding. The focus group participants said that there was too much psychoeducation, and the language level was too high for most laypeople; all language was adjusted by increasing layperson terminology and reducing jargon. Finally, the group discussed program tailoring, the addition of treatment targets, and more accurate advertising. As a result, Tranquility can be tailored to begin with either anxiety or depression content, depending on user needs; now includes quality of life and well-being tracking; and is more specifically advertised as a daily use program targeting mild to moderate anxiety and depression.
CBT experts evaluated the fidelity of Tranquility to CBT for the treatment of anxiety. The average rating for fidelity to core CBT components was 91% (SD 11.14; range 75-100), and half of the CBT experts indicated that Tranquility included 100% of all required CBT components.
In terms of specific components, all 6 experts reported that Tranquility included clear explanations of both anxiety and the CBT model, including cognitive and behavioral techniques, and Tranquility provided formal ratings of anxiety (eg, 0 to 10 scale) to program users. In total, 4 of 5 (80%) CBT experts agreed that Tranquility included specific emotion monitoring, 1 of 5 (20%) therapists agreed that Tranquility included only some emotion monitoring, and 1 expert did not respond. Similarly, 5 of 6 (83%) experts agreed that Tranquility provided specific monitoring of cognitions, and 1 of 6 (17%) therapists indicated that Tranquility included only some monitoring of cognitions. This exact pattern of results was also seen when evaluating whether Tranquility provided a method to monitor behaviors. Only 4 of 6 (67%) CBT experts reported that Tranquility offered a way to monitor specific physical sensations and allowed for adequate case conceptualization of anxiety. For both physical sensation monitoring and case conceptualization, one CBT expert believed there was some tracking of physical sensation monitoring or case conceptualization, while another expert believed there was none.
All CBT experts indicated that the following components should be included to treat both individuals depicted in the vignettes: psychoeducation about depression, behavioral activation, mood tracking, and case conceptualization. In addition, 5 of 6 (83%) experts agreed that mood ratings (eg, rating mood from 0 to 10), behavioral experiments, and problem-solving skills would also be important to include in Tranquility to treat both cases, while 4 of 6 (67%) experts agreed that coping strategies and sleep hygiene information should be included.
All 6 (100%) experts agreed that the Diane case would also require pleasant activity scheduling and symptom or outcome tracking. In total, 5 of 6 (83%) experts indicated that it would be helpful to include thought records, while 4 of 6 (67%) experts believed that the identification of cognitive distortions was important to include. Only 2 of 6 (33%) experts indicated that it was important to include motivational interviewing, physical symptom monitoring, substance use tracking, exposure stepladders, or psychoeducation about safety behaviors. Experts also suggested that the following components should also be added: mindfulness, meditation, open journaling, and CBT for insomnia.
There was unanimous agreement that Helen would benefit from the inclusion of thought records within Tranquility (6/6, 100%). In total, 5 of 6 (83%) experts indicated their belief that pleasant activity scheduling and the identification of cognitive distortions should also be used within Tranquility to treat depression affecting Helen. In total, 4 of 6 (67%) experts believed that substance use tracking, psychoeducation about safety behaviors, and symptom or outcome tracking should also be added. Half of the experts (3/6, 50%) thought motivational interviewing could be beneficial to include, while one-third (2/6, 33%) thought that physical symptom tracking should be included. No CBT expert endorsed the addition of exposure stepladders. Experts also suggested that the following components be added: safety planning, list of crisis resources, mindfulness, meditation, open journaling, and CBT for insomnia.
This study represents an assessment of both usability and treatment fidelity to CBT and a co-design adaptation of an iCBT program. Overall, participants highly rated the usability of Tranquility and indicated satisfaction with the program, and CBT experts provided high ratings for Tranquility’s treatment fidelity to CBT. The co-design adaptation process resulted in several improvements to Tranquility.
Experts agreed that Tranquility had high fidelity to the CBT model and included the most necessary components for the treatment of anxiety. There was high agreement across most components (eg, inclusion of behavioral techniques), although to obtain perfect fidelity, Tranquility would need to make emotion, behavior, physical sensation, and cognition monitoring more explicit. These elevated fidelity ratings were not unexpected, as the Tranquility program development team included a licensed clinical psychologist with extensive expertise in CBT. Notably, a recent functionality analysis of apps for depression [
Across vignettes, CBT experts typically agreed on which CBT interventions were necessary to treat the depicted cases. Moreover, both cases required the same set of interventions, except for the addition of safety behavior psychoeducation and substance use tracking for the Helen vignette, given the depicted substance use. It was an expected finding that experts would strongly suggest using interventions such as behavioral activation, given its robust efficacy in depression treatment, including within an iCBT format [
Current guidelines for digital mental health interventions strongly suggest using a co-design process [
Mental health programs targeting depression and anxiety need to have both high fidelity to the CBT treatment model and be engaging, flexible, and allow user personalization. Similarly, usability testing is critical, and Kushniruk [
The findings of this study are in line with much of the iCBT literature—users want engagement with and control over their web-based therapeutic experiences. Stawarz et al [
The ECOUTER framework provided an accessible approach that centered potential users in the feedback process and provided a procedure guide, which was beneficial when recruiting and working collaboratively with focus group participants as well as during data analysis. Engaging in an adaptation co-design process significantly improved program design and delivery for this iCBT program because it permitted the integration of researchers’ expertise pertaining to the design and delivery of iCBT and participants’ expertise pertaining to mental health both as a clinician and as a client. Moreover, seeking usability feedback from all participants aided in the rigor of this mixed methods evaluation and co-design adaptation of Tranquility because it allowed for multiple perspectives to be considered as CBT experts and focus group members may have different experiences and expectations of iCBT programs and may place value on differing program elements measured within the usability questionnaire we used, the SUS. However, although this approach had many benefits, it did include challenges such as recruiting CBT experts. Many of the approached individuals declined to participate, likely due to demands in excess of their available time, and scheduling focus groups was especially challenging. The usability of the design used in this study, although challenging at times, provides a greater
These findings have therapeutic implications for the iCBT literature. Taken together, future research and iCBT program development should consider the involvement of program users and clinical experts to ensure that fidelity to CBT and engagement strategies are implemented, given its association with adherence and program effectiveness [
Similarly, many studies, including this one, included predominantly White, heterosexual women. As a result, there are additional considerations related to race, gender, and sexual orientation that were not captured in the participant feedback or in the resultant changes to Tranquility (eg, using acceptance or values-based strategies vs cognitive restructuring for microaggressions; greater role for family or community members).
Understanding the perspectives and unique needs of marginalized groups is necessary to increase iCBT treatment accessibility and effectiveness—efforts should be made to include participants who are racialized and 2SLGBTQ+ (ie, two-spirit, lesbian, gay, bisexual, transgender, queer, and all other members of this community) and individuals who live in rural areas or are older adults. Efforts to recruit a larger and more diverse sample will afford a greater range of perspectives. Furthermore, this study included a small sample of CBT experts and focus group participants. This smaller sample was a significant strength of the study in that it afforded the focus group participants an opportunity to provide rich and in-depth perspectives on Tranquility and suggestions for adaptation. However, we recognize that the sample size does impact the generalizability of these findings to individuals who were underrepresented in the participant pool (eg, men, nonbinary individuals, or Black people) and who may or may not have been represented at all (eg, people with disabilities). Finally, it is of note that technological confidence and knowledge were not measured; therefore, it is possible that the ratings of this program were influenced by this variable.
CBT experts and mental health peers agreed that Tranquility, a web-based program treating anxiety and depression, had high usability, and both groups would be likely to recommend this program to others. CBT experts scored Tranquility as having high fidelity to CBT, and nearly all intervention components needed to treat depression were included as a part of Tranquility. Finally, the co-design process was key to refining the existing anxiety content and for the creation and integration of the new depression content. These results provide a preliminary evaluation of the Tranquility program, and they may provide user-centered engagement strategies that may help increase adherence and effectiveness for the iCBT treatment of anxiety and depression.
Cognitive behavioral therapy expert and focus group question list and case vignettes.
Focus group feedback and changes to the Tranquility program.
cognitive behavioral therapy
Employing Conceptual Schema for Policy and Translation Engagement in Research
internet cognitive behavioral therapy
Research Electronic Data Capture
System Usability Scale
two-spirit, lesbian, gay, bisexual, transgender, queer, and all other members of this community
The authors are grateful to the individuals who participated in the focus groups and to the cognitive behavioral therapy experts who committed time to review this program and provide valuable feedback. They would also like to thank Stephanie Sellars for helping with the focus groups.
AP, LW, and VCP conceptualized the study. VCP created project materials and recruited and interviewed participants with support from LW and MAR. VCP and MAR wrote the manuscript with support from AP and LW.
AP is a cofounder and senior scientific director of Tranquility, along with being a paid consultant. AP had no direct contact with the study participants during this study and did not participate in the data analysis process.