Original Paper
Abstract
Background: Resilience is defined as the ability to rely on internal characteristics and external strengths to adapt to adverse events. Although universal resilience-enhancing programs are effective for adolescents, there is a need for interventions that are more easily accessible and can be customized for individual teens. Phone apps are easy to use, can be tailored to individuals, and have demonstrated positive effects for mental health outcomes.
Objective: This study aimed to examine the feasibility and acceptability of a resilience app for adolescents. This app aimed to enhance resilience through modules focused on depression prevention, stress management, and healthy lifestyle approaches containing videos, measures, and practice suggestions. Furthermore, the study aimed to evaluate the effect of short-term app use on changes in resilience.
Methods: In study 1, individual interviews and focus groups were conducted with adolescents, parents, teachers, and clinicians to discuss possible incentives for using a mental health app, the benefits of app use, and concerns associated with app use. Feedback from study 1 led to ideas for the prototype. In study 2, individual interviews and focus groups were conducted with adolescents, parents, teachers, and clinicians to gather feedback about the resilience app prototype. Feedback from study 2 led to changes in the prototype, although not all suggestions could be implemented. In study 3, 40 adolescents used the app for 30 days to determine feasibility and acceptability. Additionally, resilience and secondary mental health outcomes were measured before and after app use. Dependent samples 2-tailed t tests were conducted to determine whether there were changes in resilience and secondary mental health outcomes among the adolescents before and after app use.
Results: Multiple themes were identified through study 1 individual interviews and focus groups, including app content, features, engagement, benefits, concerns, and improvement. Specifically, the adolescents provided helpful suggestions for making the prototype more appealing and functional for teen users. Study 2 adolescents and adults reported that the prototype was feasible and acceptable through the Computer System Usability Questionnaire (mean 6.30, SD 1.03) and Mobile App Rating Scale (mean 4.08, SD 0.61). In study 2, there were no significant differences in resilience and mental health outcomes after using the app for 30 days. There was variation between the participants in the extent to which they used the app, which may have led to variation in the results. The users appeared to prefer the depression module and survey sections, which provided mental health feedback.
Conclusions: Qualitative and quantitative data provide evidence that youth are interested in a resilience mental health app and that the current prototype is feasible. Although there were no significant mental health changes in study 3 users, practical implications and future directions are discussed for mental health app research.
doi:10.2196/38042
Keywords
Introduction
Background
Adolescence is an important period for brain development, identity formation, and the development of autonomy. Approximately 10% to 20% of youth experience mental health problems, which can cause physical, emotional, or social impairment [,]. Adolescent mental health problems have worsened since the COVID-19 pandemic started in March 2020 []. Although many interventions focus on the reduction of psychological symptoms, only a few focus on the prevention of mental illness and bolstering resilience [].
Werner and Smith [] first used the term resilience to describe the ability of children who experienced poverty and domestic violence to bounce back and live stable lives as adults. Werner and Smith [] found initial protective factors (eg, a stable adult role model), which helped increase youth’s resilience in the presence of stressors. There are multiple definitions of resilience, but resilience is defined in this study as “the ability, when faced with stress or adversity to actively use individual traits (internal factors) and wider social, community, and environmental supports (external factors) to return or maintain a positive state of mental health and functioning” []. However, some individuals may not recover to their original mental health state [], whereas others may experience posttraumatic growth in the presence of the stressor [].
Internal factors include personality characteristics and personal strengths, and external factors include the ability to rely on family, friends, school, and community []. Adolescents’ resilience increases when they use the combination of internal and external strengths to avoid negative outcomes in the presence of risks []. Previous models informed the conceptualization of resilience by Dray et al []. Lynch and Cicchetti [] developed an ecological transactional model that supported a resilience framework. This framework emphasized specific levels of proximity in order from lowest to highest: society and culture, community settings, family environment, and the individual []. Lynch and Cicchetti [] placed greater emphasis on internal resilience factors, but they acknowledged that the individual and environment constantly influence one another and could lead to changes within certain resilience domains. Gartland et al [] built on the Lynch and Cicchetti [] model and labeled these internal and external domains as separate from one another. Gartland et al [] agreed with Lynch and Cicchetti [] in that these factors could impact each other. Valuing the importance of individual and external factors, resilience is characterized as a dynamic, multidimensional concept that can change throughout one’s life span []. These internal and external factors have been targeted by resilience-enhancing programs [,].
Resilience-enhancing programs are mental health awareness programs that focus on social and emotional learning, mindfulness, and stress reduction []. Most of these school-based programs have shown positive intervention effects for depressive symptoms, internalizing problems, externalizing problems, and psychological distress []. Mental health promotion programs implemented at schools have demonstrated positive changes in resilience [,].
Prior Work
Although universal resilience-enhancing programs are effective for adolescents, there would be more benefits for interventions that are easily accessible and specific to the individual user. Phone apps can serve as such interventions, as they are easy to use, can be tailored to individuals, and have shown positive effects for mental health outcomes. Furthermore, phone apps have demonstrated efficacy in positively impacting mental health, specifically adolescent anxiety and depression []. Most studies have focused on the feasibility of mental health apps, concluding that specific factors such as esthetics, ease of usability, and components of the apps are important in their adoption and use [-].
Regarding youth mental health apps, parents and other adults have reported concerns about safety and privacy [-]. Overall, 70% of the people who use mental health apps reported the importance of privacy and data encryption []. However, most mental health apps do not have any regulatory restrictions or address safety concerns []. Given these concerns, it is particularly important to include the consumers (adolescents) and stakeholders (parents, teachers, and clinicians) in the development and testing of these mental health apps [].
Meta-analyses have provided evidence that phone-based mental health interventions that include self-monitoring components, are tailored to individuals, and increase social interaction are significantly more effective for adolescents than apps that do not use these features [,-]. Self-monitoring involves the ability to log behaviors through the app []. Tailoring involves individualizing the experience for the user based on demographic information []. This differs from personalization, which is not as in-depth as tailoring in its approach to an individual’s characteristics []. Automatic dialogue components produced by avatars can increase social connection and have beneficial effects on outcomes []. However, although recent studies have identified features related to app efficacy, there are still no consensus guidelines for the core set of app features [].
Youth apps have shown efficacy in reducing negative mental health outcomes []. The use of Ibobbly, an app aimed at reducing depression in adolescents and young adults [], resulted in decreases in suicidal ideation, depression symptoms, and distress. Mobiletype [] demonstrated an increase in the coordination of care between adolescent patients and pediatricians. At the time of this study, there were only 13 mental health apps whose use among adolescents was extensively studied (Table S1 in [,-]), and none of them focused primarily on resilience. Out of these 13 apps, 5 (38%) focused on mental health treatment (eg, coping strategies for anxiety and decreasing suicidal ideation), 4 (31%) focused on self-assessment (eg, mood rating scales and tracking sleep quality), and 4 (31%) focused on mental health illness prevention (eg, prevention of anxiety disorders and prevention of psychosis). Among these 13 apps, 11 (85%) were in the formative and pilot stages, and 2 (15%) were specifically evaluated for mental health outcomes [,]. However, a meta-analysis stated that more mental health apps need to be tested among adolescents to determine their efficacy []. There is a need for a mental health app that incorporates these aspects and promotes the importance of resilience.
Adolescents and adults have identified barriers to using mental health apps in the past, so there is a need to gather input from these groups in designing a novel resilience-focused app. Conducting individual interviews and focus groups will assist in incorporating the app components that adolescents find helpful and will address some of the concerns related to using an app.
Goals of This Study
This study aimed to develop and test the feasibility of a mental health and resilience-promotion app for youth. The study was conducted in 3 parts, with studies 1 (formative) and 2 (pilot) aimed at the development of the app and study 3 (evaluative) aimed at testing the app. The aims of studies 1 and 2 were to incorporate input from adolescents and stakeholders in the development of a feasible resilience app. Adolescents and stakeholders stated that the prototype was adequate and acceptable. This research adds to the literature by examining the core aspects that adolescents and stakeholders believe should be considered when developing a resilience app.
Methods
Recruitment
Study 1 was conducted from August 2019 to October 2019. Study 2 was conducted from November 2019 to January 2020. Study 3 was conducted from February 2020 to May 2020. For the participants who provided consent after the COVID-19 lockdown in March 2020, the consent form was discussed over the phone, and written consent was obtained electronically through a secure web-based platform. The participants were recruited through existing programs at an academic medical center using a combined convenience and purposeful sampling approach []. Given that adolescents were the target demographic, a more purposeful sampling approach was used to maximize demographic variety. Adult stakeholders were recruited primarily through convenience sampling. Inclusion criteria were as follows: adolescents who are aged 12 to 18 years, who can speak English, whose legal guardian can provide written consent, who can provide written informed assent, and who have a mobile phone. Exclusion criteria were as follows: children aged <12 years and adults aged >18 years, non–English-speaking adolescents, and adolescents without a mobile phone. Parental or guardian consent and participant assent were obtained before conducting any study procedures.
Inclusion criteria for parents, teachers, and clinicians were as follows: parents, teachers, and clinicians who can speak in English and have a mobile phone. Exclusion criteria were as follows: children aged <12 years and adults aged >18 years, non–English-speaking adolescents, and adolescents without a mobile phone. Some of the parents recruited were related to adolescents who participated in the study, which may limit the generalizability of the results. None of the interview content was shared with anyone outside the approved research staff per the institutional review board–approved protocol.
Study Design
Studies 1 and 2 mainly used qualitative data. In study 1, interviews and focus groups were conducted to guide the development of preliminary content for prototype 1.0. Individual interviews can provide qualitative information about “why” people feel or answered in a certain way, which cannot always be captured through quantitative approaches []. Focus groups can be important as an adjunct to individual interviews because of the interactive component of having multiple people in the room and can generate new ideas that might not be captured within individual interviews []. The goal was to conduct at least 6 individual interviews to reach information saturation, as identified in previous research []. Overall, 5 interviews with adolescents, 3 interviews with parents, 3 interviews with teachers, and 3 interviews with clinicians were conducted. Each individual interview lasted 20 to 50 minutes. In addition, 1 focus group was led by 2 team members and included 6 adolescents. Previous qualitative studies encourage having at least 2 focus groups []; however, for pragmatic considerations, there was only 1 focus group per phase. The focus group interview lasted approximately 60 minutes. Individual interviews and focus groups followed a semistructured design. The interview guide was semistructured but also reflected key questions, themes, and concepts that the researchers determined a priori to be important for the development of prototype 1.0. To maximize interaction during focus groups, the team prepared primes and potential prompts to elicit response and interaction between participants, as well as anticipating techniques such as mirroring one participant’s comment when questioning subsequent participants. Study 1 interview questions are included in .
In study 2, semistructured interviews and focus groups were conducted to ask the participants about the prototype. Given the iterative app development process, an updated prototype was developed during study 2. Prototype 1.0 was used for the individual interviews, and prototype 2.0 was used for the focus group. In total, 5 interviews with adolescents, 1 interview with a parent, 1 interview with a teacher, and 2 interviews with clinicians were conducted. Each individual interview lasted 25 to 50 minutes and involved the participant using the first version of the app (prototype 1.0) for at least 15 minutes on a tablet (iPad). One focus group was led by 2 team members and included 4 adolescents. Given the small size of this focus group, focus group coleaders called upon the participants to respond to questions to facilitate increased interaction and feedback about the prototype. The focus group interview lasted approximately 60 minutes, during which the group members were asked to explore an updated version of the app (prototype 2.0) on their phone for at least 15 minutes. Exploratory and confirmatory questions were used to help further refine the app before testing it for initial efficacy. Study 2 interview questions are included in .
The participants were given US $25 for participation in the focus groups and US $50 for participation in the individual interviews. They completed 2 questionnaires (Computer System Usability Questionnaire [CSUQ] [] and Mobile App Rating Scale [MARS] []), which obtained quantitative data about the participants’ thoughts surrounding app feasibility.
For study 3, our research team recruited 40 adolescents to use the app for 1 month. The adolescents completed questionnaires that measured their levels of resilience (Adolescent Resilience Questionnaire) [], self-efficacy (Generalized Self-Efficacy Scale) [], happiness (General Happiness Scale) [], emotional awareness (Emotional Awareness Questionnaire) [], coping skills (BRIEF COPE) [], and perceived social support (Interpersonal Social Support) []. They were given weekly reminders via email to use the app with the hope of increasing their engagement, as shown to be effective within the Mobile Mood Diary []. They completed questionnaires before and within 3 days after using the app for 1 month. They were provided a compensation of US $50 for using the app and completing the prestudy and poststudy questionnaires.
App
As referenced in the Introduction section, resilience-bolstering interventions increase the awareness about mental health concerns and focus on social and emotional learning, mindfulness, and stress reduction []. This app focused on enhancing resilience via 3 modules: depression prevention, stress management, and healthy lifestyle approaches. Each module follows a stepwise approach that includes the following questions: What is it? What does it look like? Where do I rate? How do I change it? and How can I practice? Each module includes videos from a teen and a clinician along with interactive games to learn skills within each section. The depression module teaches adolescents about the difference between sadness and depression. The module places emphasis on the symptoms that adolescents would experience if they were depressed. Adolescents can rate their depressive symptoms using a simple questionnaire (Patient Health Questionnaire-9) to obtain a total score regarding their depression severity (from none to severe). Adolescents can learn about specific strategies that can be helpful when they are depressed (eg, do an activity they enjoy) and can incorporate these strategies in the practice section (eg, do specific activity for certain number of minutes and rate mood before and after).
The stress module places emphasis on what stress consists of for an adolescent. The module provides psychoeducation about negative stress, positive stress, and recognizing when stress can lead to increased difficulties and anxiety. The module provides important information about the various ways in which stress presents in the body. Adolescents can rate anxiety symptoms with a simple questionnaire (Generalized Anxiety Disorder Scale-7) to obtain a total score regarding anxiety severity (from none to severe). Similar to the mood module, adolescents can learn ways to decrease their stress (eg, practicing relaxation) and can focus on this in the practice section (eg, taking a warm shower or bath).
The lifestyle module discusses the importance of self-care and the impact it can have on the quality of life. The module focuses on physical activity, nutrition, sleep, and mindfulness. Adolescents can rate themselves across these areas to see whether they are in a healthy range or if changes need to be made. Adolescents can learn about strategies to help improve their quality of life across these areas (eg, nutrition—think of food alternatives—turkey burger instead of regular burger) and can incorporate some of these changes in the practice section (eg, nutrition—listing times of mindful vs mindless eating).
Ethics Approval
The University of Texas Southwestern Medical Center Institutional Review Board approved the study before the project commenced (STU2018-0304). Written informed consent and assent were obtained before any participant data were collected. All study participants had their protected health information deidentified, and the study data were only accessible by approved members on the research team.
Data Analyses
A total of 4 independent raters and 1 qualitative expert were involved in transcribing and coding the audio recordings into the NVivo software (QSR International) []. The qualitative data were analyzed using a mixed applied thematic analysis []. An interview guide was created to reflect themes that had been referenced in previous mental health app research []. Kenny et al [] identified multiple themes after targeted youth focus groups for their mental health app (CopeSmart). Their themes helped form our interview questions and initial coding list []. A deductive coding list was developed from the interview guide in consultation with a qualitative expert and the research team through a consensus discussion. Team members who served as independent raters first participated in a co-rating exercise for a set of 2 test set transcripts. Discrepancies among the team members about codes were highlighted and reviewed by the team to reach a consensus. After reviewing the test set of 2 transcripts, inductive codes were added (ie, social interaction and time) based on team member agreement. Each individual interview transcript and focus group transcript were reviewed by 2 independent raters. Once calibration had been achieved, 2 members of the study team coded each transcript and met to ensure consistency in coding. The NVivo (version 10.0) software was used to organize all coded material such that the material could be easily analyzed. This process was performed for both studies 1 and 2.
Our research team obtained means (and SDs) for feasibility through the CSUQ. Scores >4 provide evidence that the resilience app is acceptable. We also obtained the means (and SDs) for feasibility through the MARS. Scores >3 provide evidence that the resilience app is acceptable. Dependent samples 2-tailed t tests were conducted to determine whether there were any changes in resilience and secondary mental health outcomes among the adolescents before and after they used the app. P-P plots were constructed to test the assumption of normality for all the dependent 2-tailed t tests. Our research team obtained frequencies for the number of users who viewed the entire app, sections that the users viewed, and completion of specific modules.
Results
Overview
In study 1, we conducted 14 individual interviews (5 with adolescents, 3 with parents, 3 with teachers, and 3 with clinicians) along with 1 focus group (6 adolescents). In study 2, we conducted 9 individual interviews (5 with adolescents, 2 with clinicians, 1 with a parent, and 1 with a teacher) along with 1 focus group (4 adolescents). Additional demographic information is listed in 
Tables 1 and 2.
Information about the themes (with quotes) identified during studies 1 and 2 are presented in Tables 3 and 4.
| Variables | Values, n (%) | ||
| Adolescent individual interview sample demographics (n=5) | |||
| Sex | |||
| Female | 3 (60) | ||
| Male | 2 (40) | ||
| Ethnicity | |||
| Non-Hispanic | 2 (40) | ||
| Hispanic | 3 (60) | ||
| Race | |||
| White | 2 (40) | ||
| Asian | 2 (40) | ||
| American Indian | 1 (20) | ||
| Grade | |||
| 7th grade | 1 (20) | ||
| 8th grade | 1 (20) | ||
| 9th grade | 2 (40) | ||
| 10th grade | 1 (20) | ||
| Adolescent focus group sample demographics (n=6) | |||
| Sex | |||
| Female | 1 (17) | ||
| Male | 4 (67) | ||
| Other | 1 (17) | ||
| Ethnicity | |||
| Non-Hispanic | 4 (67) | ||
| Hispanic | 1 (17) | ||
| Unknown | 1 (17) | ||
| Race | |||
| White | 2 (33) | ||
| African American | 1 (17) | ||
| Native Hawaiian | 1 (17) | ||
| >1 race | 1 (17) | ||
| Other | 1 (17) | ||
| Grade | |||
| 7th grade | 1 (17) | ||
| 9th grade | 1 (17) | ||
| 10th grade | 1 (17) | ||
| 11th grade | 2 (33) | ||
| 12th grade | 1 (17) | ||
| Adult individual interview sample demographics (n=9) | |||
| Sex | |||
| Female | 8 (89) | ||
| Male | 1 (11) | ||
| Ethnicity | |||
| Non-Hispanic | 6 (67) | ||
| Hispanic | 3 (33) | ||
| Race | |||
| White | 7 (78) | ||
| American Indian | 1 (11) | ||
| >1 race | 1 (11) | ||
| Stakeholder | |||
| Parent | 3 (33) | ||
| Teacher or school personnel | 3 (33) | ||
| Pediatric clinician | 3 (33) | ||
| Variables | Values, n (%) | ||
| Adolescent individual interview sample demographics (n=5) | |||
| Sex | |||
| Female | 2 (40) | ||
| Male | 2 (40) | ||
| Transgender | 1 (20) | ||
| Ethnicity | |||
| Non-Hispanic | 5 (100) | ||
| Race | |||
| White | 3 (60) | ||
| African American | 1 (20) | ||
| Asian | 1 (20) | ||
| Grade | |||
| 8th grade | 2 (40) | ||
| 12th grade | 3 (60) | ||
| Adolescent focus group sample demographics (n=4) | |||
| Sex | |||
| Female | 2 (50) | ||
| Male | 2 (50) | ||
| Ethnicity | |||
| Non-Hispanic | 4 (100) | ||
| Race | |||
| White | 2 (50) | ||
| African American | 1 (25) | ||
| >1 race | 1 (25) | ||
| Grade | |||
| 9th grade | 1 (25) | ||
| 10th grade | 2 (50) | ||
| 11th grade | 1 (25) | ||
| Adult individual interview sample demographics (n=4) | |||
| Sex | |||
| Female | 2 (50) | ||
| Male | 2 (50) | ||
| Ethnicity | |||
| Non-Hispanic | 4 (100) | ||
| Race | |||
| White | 3 (75) | ||
| Asian | 1 (25) | ||
| Stakeholder | |||
| Parent | 1 (25) | ||
| Teacher or school personnel | 1 (25) | ||
| Pediatric clinician | 2 (50) | ||
| Theme | Adolescents | Adult stakeholders |
| Content |
|
|
| Features |
|
|
| Concerns |
|
|
| Benefits |
|
|
| Most important thing |
|
|
| Theme | Adolescents | Adult stakeholders |
| Content |
|
|
| Features |
|
|
| Concerns |
|
|
| Engagement |
|
|
| Improvement |
|
|
| Frequency of use |
|
|
| Recommendations |
|
|
Study 2 Quantitative Results
Changes from prototype 1.0 to prototype 2.0 included the removal of the “Rate Your Mood” function, changes in the depression module game (content and graphic updates), and improved smartphone adaptability. The participants rated the app as acceptable in the CSUQ (mean 6.30, SD 1.03) and MARS (mean 4.08, SD 0.61). Further analysis determined that the participants viewed the app as adequately engaging, functional, esthetically pleasing, informative, and impactful in the MARS. These results are presented in Table 5.
Exploratory analyses examined the participants’ ratings of the app and whether they would recommend the app to others. The participants gave the prototype a rating close to 4 stars after reviewing it for a short period (mean 3.85, SD 0.69). Of the 13 participants, 8 (62%) stated that they would recommend the app to “many people” or “everyone.” These findings are also presented in Table 5.
| Variables | Values | |
| Computer System Usability Questionnaire (Likert scale from 1 to 7), mean (SD) | 6.30 (1.03) | |
| Mobile App Rating Scale (Likert scale from 1 to 5), mean (SD) | 4.08 (0.61) | |
| App engagement | 3.85 (0.67) | |
| App functionality | 4.54 (0.60) | |
| App esthetics | 4.28 (0.57) | |
| App information | 3.90 (0.80) | |
| App impact on behavior | 4.29 (0.59) | |
| App rating (Likert scale from 1 to 5), mean (SD) | 3.85 (0.69) | |
| Frequency of use over a year (Likert scale from 1 to 5), mean (SD) | 3.62 (0.96) | |
| 1-2 times, n (%) | 2 (15) | |
| 3-10 times, n (%) | 3 (23) | |
| 10-50 times, n (%) | 6 (46) | |
| >50 times, n (%) | 2 (15) | |
| App recommendation (Likert scale from 1 to 5), mean (SD) | 3.84 (0.98) | |
| Few people, n (%) | 1 (8) | |
| Several people, n (%) | 4 (31) | |
| Many people, n (%) | 4 (31) | |
| Everyone, n (%) | 4 (31) | |
Study 3 Results
A total of 40 participants provided consent for study 3 to use the app for 30 days. Overall, 60% (24/40) of the participants identified as female, 35% (14/40) as male, 2.5% (1/40) as transgender, and 2.5% (1/40) as nonbinary. Approximately half (19/40, 48%) of the participants identified as White, 20% (8/40) as Asian, 15% (6/40) as African American, and 5% (2/40) as American Indian. The nonusers (7/40, 18%), defined as the participants who never logged into the app, were removed from the analysis. Exploratory analyses revealed that there were no significant differences between nonusers and users in baseline resilience scores and demographic factors (eg, sex, race, and grade). An additional participant (1/40, 2.5%) who did not complete the poststudy questionnaires was removed from the analysis. The final analysis included 32 participants.
A total of 25% (10/40) of participants completed the entire intervention, and 55% (22/40) of participants completed some portions of the intervention. The depression module was completed by more users (18/40, 45%) than the stress module (13/40, 32%) and lifestyle module (10/40, 25%). As for each specific component, the “Where do I rate?” section was most used for the depression module (29/40, 72%), followed by the stress module (25/40, 62%) and lifestyle module (18/40, 45%). Adolescent use data are presented in Table 6.
The adolescents stated that the app was feasible overall in the CSUQ (mean 6.14, SD 0.84) and MARS (mean 3.94, SD 0.62). Further analysis determined that the adolescents viewed the app as adequately engaging, functional, esthetically pleasing, informative, and impactful in the MARS. The results are presented in Table 7.
No statistically significant improvements were found between the baseline and follow-up assessments in levels of resilience (mean 325.66, SD 59.35 vs mean 323.66, SD 60.30; t31=−0.96; P=.34) or levels of individual factors of resilience (mean 143.35, SD 28.94 vs mean 143.78, SD 27.58; t31=0.31; P=.76). The adolescents did not show statistically significant changes in any of the secondary mental health outcomes. Exploratory analyses provided evidence that the number of user log-ins was positively correlated with increases in individual resilience (r=0.39; P=.03).
| Variable | Values | |
| App completion, n (%) | ||
| None | 7 (18) | |
| Some | 23 (58) | |
| All | 10 (25) | |
| Log-insa | ||
| Number of log-ins, average (SD) | 3.83 (3.13) | |
| Number of log-ins among only users (n=33), average (SD) | 4.64 (2.84) | |
| Number of log-in among users, range | 0-14 | |
| Module completion, n (%) | ||
| Depression | 18 (45) | |
| Stress | 13 (32) | |
| Lifestyle | 10 (25) | |
| Module use, n (%) | ||
| Depression | 29 (73) | |
| Stress | 25 (63) | |
| Lifestyle | 22 (55) | |
| Depression subsection completion, n (%) | ||
| Introduction | 20 (50) | |
| What is it? | 20 (50) | |
| What does it look like? | 19 (48) | |
| Where do I rate? | 29 (73) | |
| How do I change it? | 18 (45) | |
| How can I practice this? | 19 (48) | |
| Stress subsection completion, n (%) | ||
| What is it? | 19 (48) | |
| What does it look like? | 18 (45) | |
| Where do I rate? | 25 (63) | |
| How do I change it? | 16 (40) | |
| How can I practice this? | 14 (35) | |
| Lifestyle subsection completion, n (%) | ||
| What is it? | 17 (43) | |
| What does it look like? | 15 (38) | |
| Where do I rate? | 18 (45) | |
| How do I change it? | 10 (25) | |
| How can I practice this? | 13 (33) | |
aTotal number of log-ins: 153.
| Variables | Values | |
| Computer System Usability Questionnaire (Likert scale from 1 to 7), mean (SD) | 6.14 (0.84) | |
| Mobile App Rating Scale (Likert scale from 1 to 5), mean (SD) | 3.94 (0.62) | |
| App engagement | 3.49 (0.82) | |
| App functionality | 4.41 (0.65) | |
| App esthetics | 4.17 (0.67) | |
| App information | 3.90 (0.72) | |
| App impact on behavior | 3.89 (0.90) | |
| App rating (Likert scale from 1 to 5), mean (SD) | 3.37 (0.83) | |
| Frequency of use over a year (Likert scale from 1 to 5), mean (SD) | 2.91 (0.96) | |
| None, n (%) | 4 (13) | |
| 1-2 times, n (%) | 4 (13) | |
| 3-10 times, n (%) | 15 (47) | |
| 10-50 times, n (%) | 9 (28) | |
| App recommendation (Likert scale from 1 to 5), mean (SD) | 3.53 (1.24) | |
| Not at all, n (%) | 2 (6) | |
| Few people, n (%) | 4 (13) | |
| Several people, n (%) | 11 (34) | |
| Many people, n (%) | 5 (15) | |
| Everyone, n (%) | 10 (31) | |
Discussion
Principal Findings
The aims of studies 1 and 2 were to incorporate qualitative and quantitative inputs from adolescents and stakeholders in the development of a feasible resilience app. When discussing the mental health apps, youth emphasized features over content. This finding is not surprising, given that previous apps with appealing esthetics have been viewed favorably by users [-]. It was also challenging for adolescents and adults to list ideas for content during study 1 interviews. For future studies, it may be important to provide general examples of content that could be included within the app to help generate further ideas from interviewees. The participants discussed improvements to facilitate user interest and engagement, which included logging features to track changes in mood. Turvey and Roberts [] found that user interest in mental health apps increases with the option to track mood. Clinicians focused on improving engagement by incorporating interesting activities and a reward system. The lack of engagement has previously deterred youth from gaining the benefits of mental health apps [,].
The adolescents wanted the app to be personalized and suggested that there could be a “Get to Know You” section where users provide initial demographic information. Entering demographic and behavioral information has improved users’ app experiences []. The adolescents and clinicians discussed social connection as a vital feature to include within the app. Increased social interaction can lead to improved mental health app outcomes []. Morrison et al [] stated that this can include increasing the variety of avatar automatic dialogue and mimicking peer-to-peer social interactions.
The users provided preferences for the resilience app content. The adolescents reported being happy that the prototype discussed relaxation skills. Youth mental health apps that teach coping skills have been viewed as more effective [,]. The adolescents enjoyed the surveys and discussed including components to track feelings through journaling. Previous mental health apps have emphasized the importance of self-monitoring components [,].
Although the prototype identified content strengths, the users were interested in specific changes. The interview participants inquired about including dilemmas to provide youth with opportunities to determine what they would do in complex situations. Wasserman et al [] incorporated reviewing dilemmas within the Youth Aware of Mental Health program and found significant decreases in mental health symptoms.
The adolescents stated that an app would be beneficial if it could decrease their school stress. Youth stress from school has increased over the years and occurs from a multitude of interpersonal and academic pressures []. The participants expressed doubts about app use among youth because there was minimal incentive to continue using the app prototype. Any mental health app that is developed would benefit from including rewards, or adolescents may stop using it [].
There was a consensus that multiple changes would make the app more appealing. With the ever-expanding world of technology, adolescents are more interested in apps with dynamic features. If we were able to incorporate all the feedback from study 2 to our study 3 prototype, we would have included the rate your mood function, improved navigation to engaging app functions, added youth dilemmas, provided app incentives, and added a variety of characters to increase social engagement.
The adolescents stated that the apps should include self-assessment and coping strategies. Morrison et al [] and Whiteside [] supported the notion that users enjoy immediate feedback from surveys. Fenwick-Smith et al [] found initial evidence that adolescents benefit from programs that focus on enhancing their coping skills. Youth also reported interest in apps that facilitate social communication. Including automatic dialogue through avatars could increase adolescents’ connection with the app []. Mental health app developers should continue incorporating these areas in future apps.
Mental health app designers should be cautious when developing universal apps. Patwardhan et al [] provided evidence that youth are more interested in mental health apps that can be tailored to their experience. Werner-Seidler et al [] provided evidence that stepped-care approaches can be more personalized and helpful for youth. Even with universal apps, being able to tailor to the app to individuals can lead to beneficial user experience.
In general, apps should be used for resilience interventions to reach adolescents who may not receive this information elsewhere. In-person prevention programs have demonstrated efficacious changes for mental health outcomes [,,]; however, many resources are needed for these programs. Youth mental health apps could be a target universal approach and decrease the burden on schools and primary care clinics, which have rigorous curriculum and health care demands.
The aims of study 3 were to investigate the feasibility of the mental health app and determine whether its use would lead to changes in resilience and secondary mental health outcomes. Although there were no statistically significant changes, the adolescents determined that the app was adequate, and use data provided important information about practical implications for future mental health app research.
The participants reported that the app was feasible through ratings in the CSUQ and MARS after 30 days of app use. These findings are a strength because the users found the app to be adequately engaging, functional, esthetically pleasing, informative, and helpful after prolonged use. This is an important first step, as apps must be rated as feasible before they can become effective [-].
There were no significant differences in resilience and other mental health outcomes after the adolescents used the app for 30 days. There are various reasons why there might have been no changes in user mental health outcomes. One of the possibilities is that the participants did not use the app enough, or the time given (ie, 30 days) was not enough to lead to mental health changes. Our exploratory analyses demonstrated a positive correlation between the number of times the users logged in and increases in internal resilience factors. Therefore, more app use could lead to changes in specific mental health outcomes. The users logged into the app once per week on average, but the researchers were unable to approximate the amount of time that the participants spent on the app. In addition, only 25% (10/40) of users completed all the app modules. In comparison, in the study by Tighe et al [], 34 (56%) out of 61 users completed all the modules, which may be necessary for statistically significant mental health changes.
Specific features were missing from the app, which may have decreased app use. The app did not provide the participants the option to rate their mood or send them notifications. The researchers sent weekly email reminders to encourage the use of the app; however, the app was not designed to use SMS push notifications. Previous youth mental health apps have reported higher feasibility ratings and effectiveness when incorporating these components [,,], so not having these features could decrease youth engagement in the app.
The adolescents appeared to prefer the depression module (18/40, 45%) over the stress (13/40, 33%) and lifestyle modules (10/40, 25%). One of the reasons for this is sample selection bias. This academic center had ongoing studies of depression in at-risk youth. Youth participants with a previous or current history of depression could account for the preference for the depression module. Adolescent depression prevalence is higher than adolescent anxiety rates [], which could be the reason for the higher user interest.
Adolescents preferred the “Where do I rate?” section, which provided surveys that users could complete to receive feedback about possible depression symptoms, anxiety symptoms, and physical activities (eating, sleeping, and exercising). Whiteside [] found that youth were more likely to complete self-assessments in comparison with practicing new coping skills. Surveys are more interactive and engaging to maintain youth’s attention, which is an important component for mental health apps [,].
Limitations
Some limitations should be acknowledged within our sample. First, all the adolescent participants were recruited from an academic medical center setting. Recruitment from this setting may have created a sample selection bias, which could make findings less generalizable. Future app studies should focus on recruitment through schools and community centers to create diverse samples representing spectrum of mental health awareness. Although study 2 focus group participants were able to review prototype 2.0, our sample was small. More participants should have been recruited to make these findings for prototype 2.0 generalizable to adolescents. In addition, the users were only able to review the prototype for approximately 15 minutes.
Several limitations are the same for study 3. The researchers were unable to make significant changes from the prototype to the final version of the app; therefore, qualitative feedback from the users could not be fully incorporated. Study 3 participants did not participate in the interviews and focus groups after using the app for 30 days. Gaining qualitative feedback after study 3 was beyond the scope of the study, but these interviews would have provided more in-depth information about feasibility ratings and use rates for the app.
Future Directions
Weekly use and nonsignificant changes in study outcomes provide support that this app could be paired in conjunction with an in-person component. School and primary care settings could have teachers and providers use the app videos to provide youth with psychoeducation about resilience and other mental health areas. In-person psychoeducation could be paired with asking youth to complete the interactive features of the app as homework within school settings. Pramana et al [] received positive feedback from youth by pairing cognitive behavioral therapy and SmartCAT. The current version of the app may be more effective when combined with in-person interventions.
Exploratory analyses revealed a positive correlation between the number of times the users logged into the app and internal resilience changes. Increased use may lead to changes in resilience. Future mental health app developers could incorporate features and content that provide an interapp incentive to increase app use among youth. Pramana et al [] suggested that goal setting with rewards can increase app use among youth.
The depression module was preferred among the users. Future mental health apps could incorporate psychoeducation and coping skills to alleviate depression, as adolescents may be more interested than stress or physical mental health factors. Tighe et al [] reported that Ibobbly was effective in reducing depressive symptoms among older adolescents.
Study 3 participants preferred completing the “Where do I rate?” (eg, symptom surveys) section across the 3 modules. Whiteside [] provided evidence that youth enjoy completing self-assessment in comparison with completing coping skill activities. Future mental health app developers could incorporate surveys and interactivity to increase user interest.
Future studies should obtain interview feedback from adolescents after they use the app for 30 days. Poststudy interviews could help researchers obtain insights into app features and content that impacted app use among adolescents. Although the app did not lead to changes in resilience and mental health outcomes, future studies should focus on the efficacy of pairing the content in additional settings. Apps have previously been viewed as efficacious in combination with mental health interventions [,]. Using this app with mental health treatment could lead to an increase in resilience.
Conclusions
Studies 1 and 2 provided new information about feature and content development for a resilience app prototype. Practical implications for future mental health apps include incorporating adolescent feedback, making the app more engaging and personalized, and addressing any safety concerns from adolescent and adult stakeholders.
The adolescents found the resilience mental health app to be feasible after 30 days of use. Although there were no significant changes in mental health outcomes, the users logged into the app weekly and preferred the depression module and app surveys. These findings provide important practical implications to make future mental health apps more appealing and effective for adolescents.
This study is the first to test the feasibility and efficacy of a youth resilience app. New research has been conducted on specific features and content that adolescents would want to be incorporated into mental health apps. Although there were no significant changes in user mental health outcomes, the participants found the app to be functionally acceptable after using it for 30 days. The findings of this study provide researchers with important information for the development of youth mental health apps. Future studies should examine the efficacy of the app in conjunction with in-person mental health interventions.
Acknowledgments
The authors would like to acknowledge Adora Choquette and Blake Martin for their assistance with transcribing and coding the qualitative interviews. Their work as part of the research team was important for the completion of data analysis.
The authors would like to acknowledge the Jerry M. Lewis Foundation for providing grant funding to financially support this study. This study was also funded by the Center for Depression Research and Clinical Care at the University of Texas Southwestern and The Rees-Jones Foundation (MHT, principal investigator).
Data Availability
The data sets generated during or analyzed during this study are available from the corresponding author upon request.
Conflicts of Interest
None declared.
References
- Kieling C, Baker-Henningham H, Belfer M, Conti G, Ertem I, Omigbodun O, et al. Child and adolescent mental health worldwide: evidence for action. Lancet 2011 Oct;378(9801):1515-1525. [CrossRef]
- Aldridge J, McChesney K. The relationships between school climate and adolescent mental health and wellbeing: a systematic literature review. Int J Educ Res 2018 Mar;88:121-145 [FREE Full text] [CrossRef]
- Panchal U, Salazar de Pablo G, Franco M, Moreno C, Parellada M, Arango C, et al. The impact of COVID-19 lockdown on child and adolescent mental health: systematic review. Eur Child Adolesc Psychiatry 2021 Aug 18:1-27 [FREE Full text] [CrossRef] [Medline]
- Skala K, Bruckner T. Beating the odds: an approach to the topic of resilience in children and adolescents. Neuropsychiatr 2014 Nov 21;28(4):208-217. [CrossRef] [Medline]
- Werner E, Smith RS. Journeys from Childhood to Midlife Risk, Resilience, and Recovery. Ithaca, New York, United States: Cornell University Press; 2001.
- Dray J, Bowman J, Freund M, Campbell E, Wolfenden L, Hodder RK, et al. Improving adolescent mental health and resilience through a resilience-based intervention in schools: study protocol for a randomised controlled trial. Trials 2014 Jul 18;15(1):289 [FREE Full text] [CrossRef] [Medline]
- Fergusson DM, McLeod GF, Horwood LJ. Childhood sexual abuse and adult developmental outcomes: findings from a 30-year longitudinal study in New Zealand. Child Abuse Negl 2013 Sep;37(9):664-674. [CrossRef] [Medline]
- Vloet TD, Vloet A, Burger A, Romanos M. Post-traumatic growth in children and adolescents. J Trauma Stress Disor Treat 2017;06(04). [CrossRef]
- Dray J, Bowman J, Campbell E, Freund M, Wolfenden L, Hodder RK, et al. Systematic review of universal resilience-focused interventions targeting child and adolescent mental health in the school setting. J Am Acad Child Adolesc Psychiatry 2017 Oct;56(10):813-824 [FREE Full text] [CrossRef] [Medline]
- Lynch M, Cicchetti D. An ecological-transactional analysis of children and contexts: the longitudinal interplay among child maltreatment, community violence, and children's symptomatology. Dev Psychopathol 1998 Jun 01;10(2):235-257. [CrossRef] [Medline]
- Gartland D, Bond L, Olsson CA, Buzwell S, Sawyer SM. Development of a multi-dimensional measure of resilience in adolescents: the Adolescent Resilience Questionnaire. BMC Med Res Methodol 2011 Oct 05;11(1):134 [FREE Full text] [CrossRef] [Medline]
- Fenwick-Smith A, Dahlberg EE, Thompson SC. Systematic review of resilience-enhancing, universal, primary school-based mental health promotion programs. BMC Psychol 2018 Jul 05;6(1):30 [FREE Full text] [CrossRef] [Medline]
- Dray J, Bowman J, Campbell E, Freund M, Hodder R, Wolfenden L, et al. Effectiveness of a pragmatic school-based universal intervention targeting student resilience protective factors in reducing mental health problems in adolescents. J Adolesc 2017 Jun 03;57(1):74-89 [FREE Full text] [CrossRef] [Medline]
- Grist R, Porter J, Stallard P. Mental health mobile apps for preadolescents and adolescents: a systematic review. J Med Internet Res 2017 May 25;19(5):e176 [FREE Full text] [CrossRef] [Medline]
- Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Ment Health 2016 Mar 01;3(1):e7 [FREE Full text] [CrossRef] [Medline]
- Kenny R, Dooley B, Fitzgerald A. Developing mental health mobile apps: exploring adolescents' perspectives. Health Informatics J 2016 Jun;22(2):265-275 [FREE Full text] [CrossRef] [Medline]
- East ML, Havard BC. Mental health mobile apps: from infusion to diffusion in the mental health social system. JMIR Ment Health 2015 Mar 31;2(1):e10 [FREE Full text] [CrossRef] [Medline]
- Freeman J, Caldwell PH, Scott KM. How adolescents trust health information on social media: a systematic review. Acad Pediatr 2022 Dec 26:S1876-2859(22)00637-4 [FREE Full text] [CrossRef] [Medline]
- Martinez-Martin N. Chapter Three - Trusting the bot: addressing the ethical challenges of consumer digital mental health therapy. In: Developments in Neuroethics and Bioethics. Cambridge, Massachusetts, United States: Academic Press; 2020.
- Schueller SM, Neary M, O'Loughlin K, Adkins EC. Discovery of and interest in health apps among those with mental health needs: survey and focus group study. J Med Internet Res 2018 Jun 11;20(6):e10141 [FREE Full text] [CrossRef] [Medline]
- O'Loughlin K, Neary M, Adkins EC, Schueller SM. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv 2019 Mar;15:110-115 [FREE Full text] [CrossRef] [Medline]
- Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health interventions? A review using techniques from Critical Interpretive Synthesis. Telemed J E Health 2012 Mar;18(2):137-144 [FREE Full text] [CrossRef] [Medline]
- Kenny R, Dooley B, Fitzgerald A. Feasibility of "CopeSmart": a telemental health app for adolescents. JMIR Ment Health 2015 Aug 10;2(3):e22 [FREE Full text] [CrossRef] [Medline]
- Turvey CL, Roberts LJ. Recent developments in the use of online resources and mobile technologies to support mental health care. Int Rev Psychiatry 2015 Nov 02;27(6):547-557. [CrossRef] [Medline]
- Wisniewski H, Liu G, Henson P, Vaidyam A, Hajratalli NK, Onnela J, et al. Understanding the quality, effectiveness and attributes of top-rated smartphone health apps. Evid Based Ment Health 2019 Feb;22(1):4-9. [CrossRef] [Medline]
- Reid SC, Kauer SD, Hearps SJ, Crooke AH, Khor AS, Sanci LA, et al. A mobile phone application for the assessment and management of youth mental health problems in primary care: health service outcomes from a randomised controlled trial of mobiletype. BMC Fam Pract 2013 Jun 19;14(1):84 [FREE Full text] [CrossRef] [Medline]
- Tighe J, Shand F, Ridani R, Mackinnon A, De La Mata N, Christensen H. Ibobbly mobile health intervention for suicide prevention in Australian Indigenous youth: a pilot randomised controlled trial. BMJ Open 2017 Jan 27;7(1):e013518 [FREE Full text] [CrossRef] [Medline]
- Matthews M, Doherty G. In the mood: engaging teenagers in psychotherapy using mobile phones. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2011 Presented at: CHI '11: CHI Conference on Human Factors in Computing Systems; May 7 - 12, 2011; Vancouver BC Canada URL: https://doi.org/10.1145/1978942.1979379 [CrossRef]
- O'Brien SM, Scully P, Fitzgerald P, Scott LV, Dinan TG. Plasma cytokine profiles in depressed patients who fail to respond to selective serotonin reuptake inhibitor therapy. J Psychiatr Res 2007 Apr;41(3-4):326-331. [CrossRef] [Medline]
- Whiteside SP. Mobile device-based applications for childhood anxiety disorders. J Child Adolesc Psychopharmacol 2016 Apr;26(3):246-251. [CrossRef] [Medline]
- Patwardhan M, Stoll R, Hamel D, Amresh A, Gary KA, Pina A. Designing a mobile application to support the indicated prevention and early intervention of childhood anxiety. In: Proceedings of the conference on Wireless Health. 2015 Presented at: WH '15: Wireless Health 2015 Conference; Oct 14 - 16, 2015; Bethesda Maryland URL: https://doi.org/10.1145/2811780.2811954 [CrossRef]
- Pramana G, Parmanto B, Kendall PC, Silk JS. The SmartCAT: an m-health platform for ecological momentary intervention in child anxiety treatment. Telemed J E Health 2014 May;20(5):419-427 [FREE Full text] [CrossRef] [Medline]
- Traber-Walker N, Gerstenberg M, Metzler S, Joris MR, Karr M, Studer N, et al. Evaluation of the combined treatment approach "Robin" (standardized manual and smartphone app) for adolescents at clinical high risk for psychosis. Front Psychiatry 2019 Jun 6;10:384 [FREE Full text] [CrossRef] [Medline]
- Kennard BD, Biernesser C, Wolfe KL, Foxwell AA, Craddock Lee SJ, Rial KV, et al. Developing a brief suicide prevention intervention and mobile phone application: a qualitative report. J Technol Hum Serv 2015 Oct 01;33(4):345-357 [FREE Full text] [CrossRef] [Medline]
- Løventoft P, Nørregaard L, Frøkjær E. Designing daybuilder: an experimental app to support people with depression. In: Proceedings of the 12th Participatory Design Conference: Exploratory Papers, Workshop Descriptions, Industry Cases - Volume 2. 2012 Presented at: PDC '12: 12th Participatory Design Conference; Aug 12 - 16, 2012; Roskilde Denmark URL: https://doi.org/10.1145/2348144.2348146 [CrossRef]
- Whitehouse SR, Lam P, Balka E, McLellan S, Deevska M, Penn D, et al. Co-creation with TickiT: designing and evaluating a clinical eHealth platform for youth. JMIR Res Protoc 2013 Oct 18;2(2):e42 [FREE Full text] [CrossRef] [Medline]
- Hides L, Dingle G, Quinn C, Stoyanov SR, Zelenko O, Tjondronegoro D, et al. Efficacy and outcomes of a music-based emotion regulation mobile app in distressed young people: randomized controlled trial. JMIR Mhealth Uhealth 2019 Jan 16;7(1):e11482 [FREE Full text] [CrossRef] [Medline]
- Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health 2015 Sep 6;42(5):533-544 [FREE Full text] [CrossRef] [Medline]
- Brinkmann S. Unstructured and Semi-Structured Interviewing. In: The Oxford Handbook of Qualitative Research. Oxford, United Kingdom: Oxford Academic Press; 2014.
- Guest G, Bunce A, Johnson L. How many interviews are enough?: an experiment with data saturation and variability. Field Method 2016 Jul 21;18(1):59-82. [CrossRef]
- Guest G, Namey E, McKenna K. How many focus groups are enough? Building an evidence base for nonprobability sample sizes. Field Method 2016 Jul 24;29(1):3-22. [CrossRef]
- Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact 1995 Jan;7(1):57-78. [CrossRef]
- Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
- Schwarzer R, Jerusalem M. Generalized self-efficacy scale. In: Measures in Health Psychology: A User’s Portfolio. Windsor, UK: NFER-NELSON; 1995.
- Lyubomirsky S, Lepper HS. A measure of subjective happiness: preliminary reliability and construct validation. Social Indicator Res 1999;46:137-155. [CrossRef]
- Rieffe C, Oosterveld P, Miers AC, Meerum Terwogt M, Ly V. Emotion awareness and internalising symptoms in children and adolescents: The Emotion Awareness Questionnaire revised. Personality Indiv Diff 2008 Dec;45(8):756-761. [CrossRef]
- Carver CS. You want to measure coping but your protocol's too long: consider the brief COPE. Int J Behav Med 1997;4(1):92-100. [CrossRef] [Medline]
- Cohen S, Mermelstein R, Kamarck T, Hoberman HM. Measuring the functional components of social support. In: Social Support: Theory, Research and Applications. Dordrecht: Springer; 1985.
- NVivo version 10. Lumivero. URL: https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home?_ga=2.209968086.259553662.1645663266-711678620.1645663266 [accessed 2019-10-01]
- Guest G, MacQueen KM, Namey EE. Applied Thematic Analysis. Thousand Oaks, California, United States: SAGE Publications; 2012.
- Wasserman D, Hoven CW, Wasserman C, Wall M, Eisenberg R, Hadlaczky G, et al. School-based suicide prevention programmes: the SEYLE cluster-randomised, controlled trial. Lancet 2015 Apr 18;385(9977):1536-1544. [CrossRef] [Medline]
- Bester G. Stress experienced by adolescents in school: the importance of personality and interpersonal relationships. J Child Adolesc Ment Health 2019 May 25;31(1):25-37. [CrossRef] [Medline]
- Schueller SM, Begale M, Penedo FJ, Mohr DC. Purple: a modular system for developing and deploying behavioral intervention technologies. J Med Internet Res 2014 Jul 30;16(7):e181 [FREE Full text] [CrossRef] [Medline]
- Werner-Seidler A, Perry Y, Calear AL, Newby JM, Christensen H. School-based depression and anxiety prevention programs for young people: a systematic review and meta-analysis. Clin Psychol Rev 2017 Feb;51:30-47 [FREE Full text] [CrossRef] [Medline]
- Van Voorhees BW, Fogel J, Reinecke MA, Gladstone T, Stuart S, Gollan J, et al. Randomized clinical trial of an internet-based depression prevention program for adolescents (Project CATCH-IT) in primary care: 12-week outcomes. J Dev Behav Pediatr 2009 Feb;30(1):23-37 [FREE Full text] [CrossRef] [Medline]
- Bruffaerts R, Mortier P, Auerbach RP, Alonso J, Hermosillo De la Torre AE, Cuijpers P, WHO WMH-ICS Collaborators. Lifetime and 12-month treatment for mental disorders and suicidal thoughts and behaviors among first year college students. Int J Methods Psychiatr Res 2019 Jun;28(2):e1764 [FREE Full text] [CrossRef] [Medline]
Abbreviations
| CSUQ: Computer System Usability Questionnaire |
| MARS: Mobile App Rating Scale |
Edited by A Mavragani; submitted 16.03.22; peer-reviewed by F Somville, MA Bin Ibrahim, D Levine, K Winskell; comments to author 06.12.22; revised version received 12.01.23; accepted 29.01.23; published 22.03.23
Copyright©Daniel K Elledge, Simon Craddock Lee, Sunita M Stewart, Radu Pop, Madhukar H Trivedi, Jennifer L Hughes. Originally published in JMIR Formative Research (https://formative.jmir.org), 22.03.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.