Background: Although evidence suggests that digital mental health interventions (DMHIs) are effective alternatives to traditional mental health care, participant engagement continues to be an issue, especially for pediatric DMHIs. Extant studies of DMHIs among adults suggest that participants’ satisfaction, perceived usefulness, and therapeutic alliance are closely tied to engagement. However, these associations have not been investigated among children and adolescents involved in DMHIs.
Objective: To address these gaps in extant DMHI research, the purpose of this study was to (1) develop and implement a measure to assess satisfaction, perceived usefulness, and therapeutic alliance among children and adolescents participating in a DMHI and (2) investigate satisfaction, perceived usefulness, and therapeutic alliance as correlates of children’s and adolescents’ engagement in the DMHI.
Methods: Members (children and adolescents) of a pediatric DMHI who had completed at least one session with a care provider (eg, coach or therapist) were eligible for inclusion in the study. Adolescent members and caregivers of children completed a survey assessing satisfaction with service, perceived usefulness of care, and therapeutic alliance with care team members.
Results: This study provides evidence for the reliability and validity of an adolescent- and caregiver-reported user experience assessment in a pediatric DMHI. Moreover, our findings suggest that adolescents' and caregivers’ satisfaction and perceived usefulness are salient correlates of youths’ engagement with a DMHI.
Conclusions: This study provides valuable preliminary evidence that caregivers’ satisfaction and perceived usefulness are salient correlates of youths’ engagement with a DMHI. Although further research is required, these findings offer preliminary evidence that caregivers play a critical role in effectively increasing engagement among children and adolescents involved in DMHIs.
In recent years, digital mental health interventions (DMHIs), such as internet-based coaching and therapy, have emerged as effective and scalable alternatives to in-person mental health care. However, low participant engagement, especially among children and adolescents, continues to limit the therapeutic benefits of DMHIs [- ]. Should these issues of engagement remain unaddressed, DMHIs may lose their long-term viability as mental health treatments for young people. As such, the identification of factors associated with participant engagement is paramount for DMHIs, enabling the tailoring of interventions to maximize therapeutic effects for participants.
Extant studies of DMHIs among adults suggest that participants’ satisfaction, perceived usefulness, and therapeutic alliance are closely tied to engagement. DMHI participants who experience low satisfaction, often due to a complicated or dysfunctional user experience, are more likely to drop out prematurely [, ]. Conversely, participants who report higher usefulness of a DMHI, such as improvements in their well-being and mental health symptoms, often show increased engagement [ , , ]. Finally, therapeutic alliance, or the working relationship between participants and care providers, is an often-identified predictor of compliance and engagement in traditional therapeutic interventions, such as face-to-face therapy [ , ]. Nascent evidence suggests positive but inconsistent links between therapeutic alliance and engagement in DMHIs, highlighting the need for further study [ ]. As such, understanding how satisfaction, perceived usefulness, and therapeutic alliance contribute to DMHI engagement among children and adolescents is crucial to the continued effectiveness of digitally delivered mental health care.
To address these gaps in extant DMHI research, the purpose of this study was to (1) develop and implement a measure to assess satisfaction, perceived usefulness, and therapeutic alliance among children and adolescents participating in a DMHI and (2) investigate satisfaction, perceived usefulness, and therapeutic alliance as correlates of children’s and adolescents’ engagement in the DMHI.
Design and Treatment
Members (ages 2-17 years) enrolled in care with Bend Health, Inc by December 7, 2022, who had completed at least one care session were eligible for inclusion in the study. Survey data were collected between December 2022 and January 2023. Members and their caregivers were invited via email (sent up to 5 times to each member) to complete a survey assessing the following: satisfaction with service, perception of the usefulness of care, and therapeutic alliance with care team members. Members were given a US $5 electronic gift card upon completion of the survey.
Treatment with the DMHI, Bend Health, Inc, has been described in detail previously [, ]. In brief, each member has a behavioral care manager (BCM) who coordinates and oversees their care and, in synchronous virtual sessions, coaches or licensed therapists deliver care plans based on cognitive behavioral therapy, behavioral activation, parent management training, mindfulness-based cognitive therapy, motivational interviewing, and mindfulness-based stress reduction. Members may also have sessions with a prescriber. Members may attend up to 5 synchronous virtual sessions per month with their BCMs, coaches, therapists, or prescribers.
Study procedures were approved by an independent institutional review board, Biomedical Research Alliance of New York (BRANY IRB; study identification number 23-12-010-1374).
The survey was designed based on prior work assessing therapeutic alliance and service satisfaction among youths participating in digital mental health services [- ]. The survey assessed the following domains: satisfaction with service, perceived usefulness, and therapeutic alliance. Satisfaction (5 items) assessed the acceptability of the DMHI interface and care services. Perceived improvement (5 items) assessed perceptions of improvements in emotions, emotional skills, and relationships. Therapeutic alliance (12 items) assessed members’ feelings of hope and optimism after sessions, being understood by their care team, and alignment of goals and tasks. Responses were on a 5-item Likert scale (ranging from “strongly disagree” to “strongly agree”). All items are listed in Table S1 and Figures S2-S4 in . Caregivers of children (ages 2-12 years) responded to the survey on behalf of their child (caregiver report). Adolescents (ages 13-17 years) responded to the survey on behalf of themselves (self-report).
To determine whether the subscales’ respective items measured the same underlying construct, we conducted an exploratory factor analysis, followed by 3 confirmatory factor analyses (one per subscale). All factor analyses were conducted in R 4.2.0 (R Core Team) using the lavaan package [, ]. After establishing the dimensionality of subscales, Cronbach alphas were computed for each subscale to determine reliability. Due to power limitations, mean values of subscale items were used for subsequent analyses (maximum mean score 5).
For initial descriptive statistics, survey respondents were differentiated by caregiver report and self-report. Caregiver-reported and self-reported survey scores were compared for each subscale by paired 2-tailed t tests. Each responding member’s duration in care was assessed as the number of months between their first synchronous session with a Bend Health care provider (eg, BCM, coach, or therapist) and their survey completion date. Descriptive statistics of care with Bend Health, Inc are reported for 2022. Survey respondents’ demographic information and patterns of participation were then compared to those who were sent the survey but did not respond using paired t tests and chi-square tests.
Finally, correlation analyses were used to assess whether there is a significant positive association between engagement (eg, duration of care and the number of total sessions with a coach or therapist) and subscale scores. Each subscale was assessed independently.
Respondents Versus Nonrespondents
Of the 213 members eligible for the study, 52 (24.1%) responded (caregiver report: n=33 and self-report: n=19). Comprehensive demographics for all groups are reported in. Chi-square tests suggested no significant demographic differences between respondents and nonrespondents (Table S5 in ). Respondents completed the survey, on average, 2.92 (SD 1.39) months after they began care with the DMHI and had completed an average of 3.62 coaching or therapy sessions prior to responding ( ). Nonrespondents had completed significantly fewer coaching or therapy sessions ( ).
|Demographics||Caregiver report (children; n=33)||Self-report (adolescents; n=19)||Nonrespondents (n=161)|
|Age (years), mean (SD)||8.0 (2.3)||14.4 (1.2)||11.0 (3.7)|
|Sex, n (%)|
|Female||20 (60.6)||17 (89.5)||88 (54.7)|
|Male||13 (39.4)||2 (10.5)||72 (44.7)|
|Gender or sex conformity, n (%)|
|Conforming||38 (100)||18 (94.7)||159 (98.8)|
|Nonconforming||0 (0)||1 (5.3)||2 (1.2)|
|Race or ethnicity, n (%)|
|Asian||1 (3)||1 (5.3)||0 ( 0)|
|Black or African American||0 (0)||0 (0)||1 (0.6)|
|Hispanic or Latino||2 (6.1)||1 (5.3)||14 (8.7)|
|Other or multiracial||8 (24.2)||10 (52.6)||6 (3.7)|
|Unknown||0 (0)||0 (0)||0 (0)|
|White||22 (66.7)||7 (36.8)||55 (34.2)|
|Participation measure||Respondents (n=52), mean (SD)||Nonrespondents (n=166), mean (SD)||Comparison|
|t test (df)||P value|
|Total BCMa sessions||1.3 (1.2)||0.9 (1.2)||–2.4 (211)||.02|
|Total coaching sessions||2.1 (3.2)||1.2 (2.3)||–2.0 (68.3)||.05|
|Total therapy sessions||1.0 (1.3)||1.0 (1.5)||–0.1 (211)||.94|
|Total coaching and therapy sessions||3.1 (3.1)||2.2 (2.5)||–2.1 (75.2)||.04|
|Total prescriber sessions||0.9 (0.9)||0.6 (0.9)||–2.1 (211)||.04|
aBCM: behavioral care manager.
Measurement Model and Reliability
Using exploratory factor analysis, we identified a 3-factor solution as most appropriate (). This solution aligned closely with our expected subscales of satisfaction, perceived usefulness, and alliance, with the exception of 2 items from the alliance subscale that loaded onto the satisfaction factor and 3 items from the alliance subscale that loaded onto the perceived usefulness factor (Figures S1-S3 in present more details).
Using confirmatory factor analysis, we then confirmed this solution using 3 separate 1-factor models (a 3-factor model was not feasible due to lower power ). All models showed acceptable fit based on the comparative fit index (CFI), Tucker-Lewis index (TLI), and standardized root mean square residual (SRMR) fit indices (CFI>0.90; TLI>0.90; and SRMR<0.08 [ ]). Factors were highly correlated with each other (alliance with satisfaction=0.90; P<.001; alliance with perceived usefulness=0.805; P<.001; and satisfaction with perceived usefulness=0.847; P<.001). The results of the final measurement models were then used to create mean subscales ( - ; mean values of each subscale are found in Table S4 in ). For each subscale, Cronbach alpha was calculated to assess reliability. All subscales indicated high reliability (alliance α=.96, satisfaction α=.95, and perceived usefulness α=.97).
Correlation of Survey Scores and Engagement
No significant associations were found between survey scores and care duration. However, satisfaction and perceived usefulness scores were both significantly correlated with the total number of therapy or coaching sessions (satisfaction: r=0.28; P=.046 and perceived usefulness: r=0.36; P=.01;and ). Coefficients and P values of all correlation analyses are included in Table S6 in .
By surveying adolescent members and caregivers of children engaged in care with a DMHI, this study (1) developed a novel measure to assess satisfaction, perceived usefulness, and therapeutic alliance among children and adolescents participating in a DMHI and (2) investigated member-reported satisfaction, perceived usefulness, and therapeutic alliance as correlates of care duration and total sessions with a care provider. Results from our factor analyses revealed that several items intended to assess therapeutic alliance were more closely correlated with members’ satisfaction and perceived usefulness of the DMHI. We found that satisfaction and perceived usefulness were positively linked to total care sessions attended, while therapeutic alliance was not significantly associated with either measure of engagement. These findings suggest that caregivers’ and youths’ positive feelings about the functionality and utility of the services provided may be central to youths’ engagement in the DMHI.
Our final measurement models showed good psychometric properties of a caregiver- and self-reported user experience measure characterized by dimensions of satisfaction, perceived usefulness, and therapeutic alliance. These findings are largely consistent with those by Rickwood et al , who identified a similar 3-factor model characterized by service satisfaction (including practical aspects of the web-based and in-session user experience as well as members’ likelihood of recommending and returning to the DMHI), potential outcomes (including the emotional, behavioral, and relational changes associated with participation in the DMHI), and session satisfaction (including elements of the therapeutic alliance, such as feeling listened to and understood). This study largely replicates those findings and builds upon them by not only establishing similar dimensionality using caregiver-reported data but also exploring the subscales’ correlation with indices of engagement.
Compared to satisfaction and therapeutic alliance, we found that perceived usefulness was most strongly associated with the total number of coaching and therapy sessions attended by children and adolescents. Several extant meta-analyses and systematic reviews have identified perceived usefulness as a particularly salient predictor of engagement among adults [, , ]. These results suggest a similar trend in the context of a pediatric DMHI, namely that caregivers’ and youths’ beliefs about experiencing tangible change due to the intervention are more important for engagement than their perceptions about the practical user experience and their relationship with the care team practitioners. Although further study is necessary, findings like these offer insights for digital health companies that can inform the development and design of the user experience, which may in turn facilitate engagement, improve patient symptoms, and increase revenue generation.
This study has several limitations. Given the cross-sectional nature of the study, we cannot assess the directionality of these effects. As such, it may be that increased care sessions lead to higher levels of satisfaction and perceived usefulness. Our previous studies have indeed demonstrated time-dependent improvements in mental health outcomes of children and adolescents as well as improvements in the well-being of their caregivers, who are participating in coaching and therapy with Bend Health, Inc [, , ]. Future research is necessary to determine whether caregivers’ satisfaction and perceived usefulness improve concurrently with their children’s mental health symptoms over the course of care with a pediatric DMHI.
In addition, the survey completion rate was quite low, limiting our sample size and consequent power to investigate and detect group-level differences in reporter, care type, and symptom type. This issue also highlights the difficulty of voluntary data collection among DMHI participants [- ]. Future studies should explore methods for more successfully incentivizing DMHI participants to complete optional assessments aimed at improving the user experience. Moreover, given that our survey used convenience sampling of active members only, it is unclear how our survey results are impacted by social desirability bias or selection effects. A random sampling of both active and nonactive members would mitigate this bias and, thus, better characterize users’ experiences.
Notwithstanding these limitations, this study provides valuable preliminary evidence that adolescents' and caregivers’ satisfaction and perceived usefulness are salient correlates of youths’ engagement with a DMHI. As mental health care becomes more digitalized, the development and standardization of user experience assessments will become increasingly central to DMHI product design, quality assurance, and revenue generation. Our findings speak to this growing need by providing preliminary evidence for the reliability, validity, and implications of a multidimensional user experience assessment administered in the context of a pediatric DMHI.
LGH and DLS conceptualized the study, designed the methodology, conducted formal analyses, wrote the original draft, reviewed and edited the manuscript, and contributed to the visualization of the study. JH, CB, and RG conceptualized the study and helped in the writing of the original draft as well as reviewing and editing of the manuscript. JH and CB also supervised the study. MR, KR, and AP contributed to the writing, reviewing, and editing of the manuscript. MR and KR were also in charge of the funding acquisition.
Conflicts of Interest
All authors are employed or contracted with Bend Health, Inc, which delivered the treatment used in this retrospective study. However, authors’ employment status and salary are not dependent upon the results of their research.
Survey responses and factor analysis details.DOCX File , 255 KB
- Borghouts J, Eikey E, Mark G, De Leon C, Schueller SM, Schneider M, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res 2021 Mar 24;23(3):e24387 [https://www.jmir.org/2021/3/e24387/] [CrossRef] [Medline]
- Garrido S, Millington C, Cheers D, Boydell K, Schubert E, Meade T, et al. What works and what doesn't work? A systematic review of digital mental health interventions for depression and anxiety in young people. Front Psychiatry 2019;10:759 [https://europepmc.org/abstract/MED/31798468] [CrossRef] [Medline]
- Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [https://www.jmir.org/2019/9/e14567/] [CrossRef] [Medline]
- Liverpool S, Mota CP, Sales CMD, Čuš A, Carletto S, Hancheva C, et al. Engaging children and young people in digital mental health interventions: systematic review of modes of delivery, facilitators, and barriers. J Med Internet Res 2020 Mar 12:e16317 [https://doi.org/10.2196/16317] [CrossRef] [Medline]
- Lehtimaki S, Martic J, Wahl B, Foster KT, Schwalbe N. Evidence on digital mental health interventions for adolescents and young people: systematic overview. JMIR Ment Health 2021 Apr 29;8(4):e25847 [https://mental.jmir.org/2021/4/e25847/] [CrossRef] [Medline]
- Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health 2018 Aug 05;21(3):116-119 [CrossRef] [Medline]
- Wasil A, Taylor M, Franzen R, Steinberg J, DeRubeis R. Promoting graduate student mental health during COVID-19: acceptability, feasibility, and perceived utility of an online single-session intervention. Front Psychol 2021;12:569785 [https://europepmc.org/abstract/MED/33897511] [CrossRef] [Medline]
- Bourke E, Barker C, Fornells-Ambrojo M. Systematic review and meta-analysis of therapeutic alliance, engagement, and outcome in psychological therapies for psychosis. Psychol Psychother 2021 Sep 10;94(3):822-853 [CrossRef] [Medline]
- Sotero L, Relvas A. Dropout versus retention in family therapy: how are they associated with behavioral manifestations of the therapeutic alliance? Contemp Fam Ther 2021 Oct 19;43(4):320-328 [CrossRef]
- Tremain H, McEnery C, Fletcher K, Murray G. The therapeutic alliance in digital mental health interventions for serious mental illnesses: narrative review. JMIR Ment Health 2020 Aug 07;7(8):e17204 [https://mental.jmir.org/2020/8/e17204/] [CrossRef] [Medline]
- Huffman L, Lawrence-Sidebottom D, Huberty J, Roots M, Roots K, Parikh A, et al. Using digital measurement-based care for the treatment of anxiety and depression in children and adolescents: observational retrospective analysis of bend health data. JMIR Pediatr Parent 2023 Apr 20;6:e46154 [https://pediatrics.jmir.org/2023//e46154/] [CrossRef] [Medline]
- Lawrence-Sidebottom D, Huffman L, Huberty J, Beatty C, Roots M, Roots K, et al. Using digital measurement-based care to address symptoms of inattention, hyperactivity, and opposition in youth: retrospective analysis of bend health. JMIR Form Res 2023 Apr 26;7:e46578 [https://formative.jmir.org/2023//e46578/] [CrossRef] [Medline]
- Horvath AO, Greenberg LS. Development and validation of the working alliance inventory. J Couns Psychol 1989 Apr;36(2):223-233 [CrossRef]
- Marmar CR, Horowitz MJ, Weiss DS, Marziali E. The development of the Therapeutic Alliance Rating System. The Psychotherapeutic Process: A Research Handbook 1986:367-390 [https://psycnet.apa.org/record/1987-97275-010]
- Rickwood D, Wallace A, Kennedy V, O'Sullivan S, Telford N, Leicester S. Young people's satisfaction with the online mental health service eheadspace: development and implementation of a service satisfaction measure. JMIR Ment Health 2019 Apr 17;6(4):e12169 [https://mental.jmir.org/2019/4/e12169/] [CrossRef] [Medline]
- The most trusted IDE for open source data science. Posit. URL: https://posit.co/products/open-source/rstudio/ [accessed 2023-08-14]
- Rosseel Y. Lavaan: an R package for structural equation modeling. J Stat Softw 2012;48:36
- Wolf EJ, Harrington KM, Clark SL, Miller MW. Sample size requirements for structural equation models: an evaluation of power, bias, and solution propriety. Educ Psychol Meas 2013 Dec;76(6):913-934 [https://europepmc.org/abstract/MED/25705052] [CrossRef] [Medline]
- Kenny DA. Measuring Model Fit. 2020. URL: https://davidakenny.net/cm/fit.htm [accessed 2023-08-16]
- Huffman L, Lawrence-Sidebottom D, Huberty J, Beatty C, Guerra R, Roots K, et al. Sleep problems and parental stress among caregivers of children and adolescents enrolled in a digital mental health intervention. JMIR Preprints. Preprint posted online on August 10, 2023 [CrossRef]
- Ng MM, Firth J, Minen M, Torous J. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv 2019 Jul 01;70(7):538-544 [https://europepmc.org/abstract/MED/30914003] [CrossRef] [Medline]
- Mailey E, Wójcicki TR, Motl R, Hu L, Strauser D, Collins K, et al. Internet-delivered physical activity intervention for college students with mental health disorders: a randomized pilot trial. Psychol Health Med 2010 Dec;15(6):646-659 [CrossRef] [Medline]
- Lintvedt O, Griffiths K, Sørensen K, Østvik AR, Wang C, Eisemann M, et al. Evaluating the effectiveness and efficacy of unguided internet-based self-help intervention for the prevention of depression: a randomized controlled trial. Clin Psychol Psychother 2011;20(1):10-27 [CrossRef] [Medline]
|BCM: behavioral care manager|
|CFI: comparative fit index|
|DMHI: digital mental health intervention|
|SRMR: standardized root mean square residual|
|TLI: Tucker-Lewis index|
Edited by A Mavragani; submitted 30.05.23; peer-reviewed by T Baranowski; comments to author 01.07.23; revised version received 08.08.23; accepted 10.08.23; published 06.09.23Copyright
©Landry Huffman, Darian Lawrence-Sidebottom, Jennifer Huberty, Clare Beatty, Monika Roots, Kurt Roots, Amit Parikh, Rachael Guerra. Originally published in JMIR Formative Research (https://formative.jmir.org), 06.09.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.