JFR JMIR Form Res JMIR Formative Research 2561-326X JMIR Publications Toronto, Canada v6i4e36794 35436218 10.2196/36794 Original Paper Original Paper Feasibility, Acceptability, and Preliminary Outcomes of a Cognitive Behavioral Therapy–Based Mobile Mental Well-being Program (Noom Mood): Single-Arm Prospective Cohort Study Mavragani Amaryllis EIsenstadt Mia Teles Ariel McCallum Meaghan PhD 1 https://orcid.org/0000-0003-0677-5619 Ho Annabell Suh PhD 1 https://orcid.org/0000-0002-0972-0181 Mitchell Ellen Siobhan PhD 1
Academic Research Noom Inc 229 W 28th St, Fl 9 New York, NY, 10001 United States 1 631 938 1248 siobhan@noom.com
https://orcid.org/0000-0001-7851-6283 May Christine N PhD 1 https://orcid.org/0000-0002-4367-5310 Behr Heather PhD 1 2 https://orcid.org/0000-0002-6410-0815 Ritschel Lorie PhD 3 4 https://orcid.org/0000-0001-5984-4046 Mochrie Kirk PhD 3 https://orcid.org/0000-0003-2727-985X Michaelides Andreas PhD 1 https://orcid.org/0000-0003-4983-5001 Academic Research Noom Inc New York, NY United States Department of Integrative Health Saybrook University Pasadena, CA United States Triangle Area Psychology Clinic Durham, NC United States School of Medicine University of North Carolina Chapel Hill, NC United States Corresponding Author: Ellen Siobhan Mitchell siobhan@noom.com 4 2022 15 4 2022 6 4 e36794 26 1 2022 25 2 2022 8 3 2022 23 3 2022 ©Meaghan McCallum, Annabell Suh Ho, Ellen Siobhan Mitchell, Christine N May, Heather Behr, Lorie Ritschel, Kirk Mochrie, Andreas Michaelides. Originally published in JMIR Formative Research (https://formative.jmir.org), 15.04.2022. 2022

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

 Background

The prevalence of anxiety, depression, and general distress has risen in recent years. Mobile mental health programs have been found to provide support to nonclinical populations and may overcome some of the barriers associated with traditional in-person treatment; however, researchers have voiced concerns that many publicly available mobile mental health programs lack evidence-based theoretical foundations, peer-reviewed research, and sufficient engagement from the public.

 Objective

This study aimed to evaluate the feasibility, acceptability, and preliminary outcomes of Noom Mood, a commercial mobile cognitive behavioral therapy– and mindfulness-based program.

 Methods

In this single-arm prospective cohort study, individuals who joined Noom Mood between August and October 2021 completed surveys at baseline and 4-week follow-up. Per-protocol analyses included those who completed both surveys (n=113), and intention-to-treat analyses included all participants (N=185).

 Results

A majority of the sample reported that the program is easy to use, they felt confident recommending the program to a friend, and they perceived the program to be effective at improving stress and anxiety. There were significant improvements in anxiety symptoms, perceived stress, depressive feelings, emotion regulation, and optimism in both the per-protocol and intention-to-treat analyses (all P<.001). Participants reported benefiting most from learning skills (eg, breathing and cognitive reframing techniques), interacting with the program features, and gaining awareness of their emotions and thought patterns. Participants also made a number of suggestions to improve product functionality and usability.

 Conclusions

Results suggest that Noom Mood is feasible and acceptable to participants, with promising preliminary outcomes. Future studies should build on these results to evaluate the effects of Noom Mood using more rigorous designs.

 mHealth mobile mental health mental health stress anxiety Introduction

The World Health Organization stresses the importance of mental health, which they broadly define as a state of “well-being in which an individual realizes his or her own abilities and can cope with the normal stresses of life” [1]. Many individuals are affected by difficulties with mental health [2]; for example, anxiety disorders are highly prevalent worldwide and are estimated to affect 18% of individuals in the United States alone [3,4]. Lifetime prevalence for depression is approximately 17% [2]. Furthermore, it is increasingly recognized that the general population can benefit from mental health support, regardless of whether clinical thresholds for mental illness are met [5,6]: as many as 57% to 84% of US adults have reported subclinical but substantial amounts of stress or worry in recent years [7,8]. Estimates suggest that anxiety, depression, and stress are associated with greater risk of mortality and hundreds of billions of dollars in economic burden per year [9,10]. 

Although a number of empirically supported treatments for mental health difficulties are available, myriad barriers exist that make it difficult for many people to access traditional in-person support, including cost, long waiting times to see providers, and limited provider availability, especially for individuals living in remote areas [11-15]. The COVID-19 pandemic has also increased barriers to accessing in-person support, potentially increasing willingness to seek digital support [16,17]. In addition, many individuals avoid seeking treatment due to stigma or to mistrust of the mental health system more generally [11,13].

In recent years, there has been a proliferation of interest in and development of mobile mental health programs. Use of these programs has tripled in recent years [18], and multiple reviews suggest that mobile mental health apps have the capacity to improve mental health and emotion regulation in the general population [19,20]. Mobile mental health has the potential to address many of the aforementioned barriers to treatment [21,22]; perhaps most importantly, mobile mental health allows for support or psychoeducation that is not restricted by time, location, or provider availability. In addition, digital (ie, via smartphone) delivery increases accessibility and autonomy in allowing for largely self-directed care [5,23]. Such programs facilitate self-monitoring of mood or activity, a well-known strategy to change undesired behaviors [5]. Lastly, mobile platforms allow for objective measurement of behavioral indicators, such as the number of articles read, and, therefore, allow individuals to track which strategies are most effective in helping them achieve behavioral change.

Despite this proliferation of readily accessible mobile mental health programs, researchers have raised several concerns that merit attention and that can be viewed through the lens of implementation science (see Proctor et al [24] for an in-depth discussion of implementation science variables as they apply to outcome studies). First, many mobile health (mHealth) programs available to the public are not based on evidence-based theoretical frameworks [25]. Moreover, users are self-selected, meaning that the problems they are experiencing may or may not map onto the content including the mobile app (ie, problems with appropriateness) [25]. Second, whether evidence based or not, many programs are used briefly and then discarded (ie, problems with adoption) or do not reach a broad enough segment of the population to be useful (ie, problems with penetration) [18,26]. Research has found that thousands of programs have been released on app stores that retain a very limited number of active users over time; for example, studies have shown that 97% of users do not use these mental health apps at day 15 [26,27]. This represents an obvious challenge for mental health programs, as intervention engagement has been associated with better outcomes in a multitude of studies [28-30]. Lastly, few of these mobile mental health programs include a research component to evaluate feasibility, acceptability, or outcomes of any sort; of programs based on theoretical frameworks, only approximately 6.2% have associated peer-reviewed research [25,31,32].

As such, this study was designed to address these gaps in the literature by examining the feasibility, acceptability, and preliminary outcomes of Noom Mood, a widely available commercial mHealth program that incorporates evidence-based recommendations for mobile mental health programs [5]. In particular, this study aims to contribute to the substantial gap in the evidence base, identified by implementation science researchers and review papers on mobile mental health, in data from commercial programs [25,31,33,34]. Another contribution of this study stems from Noom Mood’s inclusion of personal coaching for guidance and implementation of cognitive behavioral therapy (CBT) techniques, but not clinical therapy. Few studies have examined widely available mental health programs guided by personal coaching; many existing studies examine mental health programs that are entirely self-guided (ie, without individualized coaching support), are designed to provide clinical therapy or serve as an adjunct to therapy, or provide personalized coaching in other contexts (eg, employer-provided coaching or for specific conditions) [35-38]. 

Noom Mood is a structured, skills-based approach to stress and anxiety management. Noom Mood uses strategies from empirically supported treatments that have been shown to improve mental health outcomes, such as anxiety, depression, and stress (eg, CBT, dialectical behavior therapy [DBT], acceptance and commitment therapy [ACT], and mindfulness-based stress reduction [MBSR]) [39-43]. Importantly, preliminary evidence has shown that CBT and MBSR can be deployed on a mobile platform and that these programs are associated with improvements in mental well-being in nonclinical and clinical populations [23,44]; however, as described previously, more empirical evaluation is needed of evidence-based, commercial programs. Program components include the following: (1) a daily curriculum consisting of psychoeducational articles for users to read, (2) individualized coaching offered through in-app messaging, (3) weekly skills-based activities, and (4) a mood-logging feature. All four components are expected to improve mental well-being (eg, reduce perceived anxiety and depressive symptoms and perceived stress). The curriculum, activities, and coaching were derived from evidence-based frameworks (ie, CBT, DBT, ACT, and MBSR) that have been shown to be effective in improving these outcomes, so these three components would be expected to be most directly related to outcomes. The fourth component of mood logging is based on behavior change techniques of self-monitoring, helping users to build self-awareness of their mood and associated behaviors [45]. More specifically, the daily curriculum was developed in collaboration with clinical psychologists and was designed to translate evidence-based treatments and psychoeducation into a format that is useful for individuals within a self-help framework. For example, each day, participants are presented with a short article that explains conceptual terms and principles (eg, cognitive defusion from ACT), provides practical tips and quizzes to build knowledge, and guides users through a relevant practical activity (eg, how to practice cognitive defusion over the next week; Figure 1). Because of the utility of skills-training activities that help to apply evidence-based principles into daily life [5,46,47], Noom Mood introduces individuals to a short 10- to 15-minute practical activity based on evidence-based frameworks, such as breathing techniques and cognitive reframing at the beginning of each week. The activity is implemented for 1 week, with a practice on day 7 in which individuals reflect on the skill learned and how well it worked for them (Figure 1). Lastly, Noom Mood includes a messaging feature that allows participants to communicate directly with health coaches (Figure 1). Coaches help users to understand and engage in activities, encourage reflection and awareness of patterns, and provide validation for emotional experiences based on CBT techniques. Coaching protocols were adapted to this mental well-being context from the Noom weight management program, for which coaching has been refined and tested and shown to provide guidance on activities, emotional self-awareness, and emotional validation [48]. Noom Mood coaches are trained in CBT techniques but are not licensed clinicians, as Noom Mood does not provide clinical assessment, diagnoses, or treatment and is not a replacement for therapy. The coaching feature was included to address concerns that have been cited in previous studies of evidence-based programs [48-50]. Specifically, human contact from remote coaches within otherwise self-guided digital programs may encourage engagement and improve outcomes [48,51,52]. One randomized controlled trial (RCT) found that engagement check-ins from coaches improved engagement in a web-based depression program [53].

Screenshots of the Noom Mood program.

The first step in evaluating any new mHealth platform is to investigate stakeholders’ views on the feasibility and acceptability of the proposed product [24,50]. Feasibility is defined as the extent to which end users feel that they could and would use the product in their lives for the purposes for which it was designed [54]. Acceptability is defined as the extent to which stakeholders find the product satisfactory with regard to its content and perceived credibility [54]. Results from feasibility and acceptability testing are then used to refine and update the platform to align with stakeholders’ suggestions more closely. 

The primary goal of this study was to evaluate the feasibility and acceptability of Noom Mood, as well as to gather preliminary data on whether the program might be associated with improved well-being. We hypothesized that users would find the platform to be feasible and acceptable. Furthermore, we hypothesized that participants who used the program would report some benefit in terms of improved anxiety symptoms, stress, depressive feelings, emotion regulation, and optimism by the end of the 4-week study.

 Methods

A single-arm prospective cohort design was used to test feasibility and acceptability of Noom Mood, as well as initial symptom and well-being outcomes.

Ethics Approval

The study was approved by the Advarra Institutional Review Board (protocol No. 00055306).

 Procedure and Participants

Participants were recruited from the pool of individuals who had voluntarily signed up for the Noom Mood program. A randomly selected subset of adults who voluntarily enrolled in the Noom Mood program between August and October 2021 were invited to participate. All participants provided informed consent prior to participation. Inclusion criteria for participants were as follows: located within the United States, English speaking, and aged 18 years or older. Participants were invited to complete the baseline questionnaire within 1 business day of signing up for the Noom Mood program. Those who completed the baseline questionnaire were invited to complete the follow-up survey 4 weeks later. Study completers were compensated with a US $20 gift card for their participation. Participants did not receive the program for free during or after the study. The entire study occurred remotely, including online administration of surveys via email.

 Noom Mood Program

The Noom Mood program was deployed as described above. At the time of this study, approximately 15 psychoeducational articles were presented to participants each week. In addition to the curriculum, participants had access to mood-logging features, and they were encouraged by coaches to engage in the curriculum and to log their mood once per day.

 Measures Feasibility

Feasibility was assessed at 4-week follow-up.

System Usability Scale

The System Usability Scale (SUS) [55] is a 10-item scale assessing stakeholders’ views of ease of use. Items were modified to substitute “program” for “system.” Participants were asked to rate their agreement with each usability statement (eg, “I thought the program was easy to use”) on a scale of 1 (“strongly disagree”) to 5 (“strongly agree”). After reverse-scoring relevant items, sum scores were multiplied by 2.5 to create a final score ranging from 0 to 100. Research indicates that SUS scores above 68 are considered above average and scores below 68 are below average. Internal reliability for the SUS was excellent (α=.90).

 Program Engagement Data

As in past work [56], feasibility was also evaluated via the amount of time participants spent engaging with the program. Engagement data consisted of usage and self-report data recorded by the program for 4 weeks. Self-report and usage data were collected by the mobile program and stored on a secured cloud server from Amazon Web Services [57]. Data were deidentified prior to extraction from the database. Engagement measures included the frequency with which participants completed mood logs, number of times the app was opened, number of articles read, number of messages sent to the coach, and number of activities completed. Data were also extracted to evaluate the number of days the user was active, which was defined as the number of days with at least one in-app action. In order to measure real-world engagement, participants were not given specific minimum engagement requirements to remain in the study.

 Acceptability

Acceptability was assessed at 4-week follow-up. 

Credibility and Expectancy Questionnaire

The Credibility and Expectancy Questionnaire (CEQ) [58] is a 6-item scale that was originally designed to assess perceptions of treatment credibility and expectancy for improvement in psychotherapy. To render the scale more appropriate for use in this study, questionnaire items were modified slightly (ie, “program” was substituted for “therapy” and “stress and anxiety” was substituted for “symptoms”). Items in the CEQ range either from 1 to 9 or from 0 to 100, depending on the item. In line with the CEQ’s factor structure and following previous work [59], we computed average credibility and expectancy scores reflected by the first three and last three items of the scale, respectively. Internal reliability was excellent (credibility subscale: α=.90; expectancy subscale: α=.93).

 Program Satisfaction Questionnaire

We asked the following open-ended questions: (1) What is the main benefit you received from Noom’s stress and anxiety management program? (2) How can we improve Noom’s stress and anxiety management program for you? (3) What was the most helpful part of the program? and (4) What was the least helpful part of the program? Because of the variety of answers possible, content analysis was used to code each response into categories and calculate the percentage of responses allocated to each category. The categories were created using latent Dirichlet allocation (LDA), a machine learning approach for automatic clustering of text data [60]. LDA is an unsupervised approach that automatically identifies latent clusters of words (ie, categories) that cluster within unclassified data. Each word cluster was assigned a label, or category name, by a master coder with experience with the program. For each question, each participant response was given a score (0 or 1) for each category since one response could apply to multiple categories. Interrater reliability between the master coder and another coder blind to the study’s hypotheses and design ranged from 0.72 to 1.0 for all categories, suggesting good to excellent reliability [61].

 Symptom and Well-being Outcomes

Symptom and well-being outcomes were assessed at baseline and 4-week follow-up.

7-Item Generalized Anxiety Disorder Scale

The 7-item Generalized Anxiety Disorder scale (GAD-7) [62] is a 7-item scale that assesses the extent to which individuals experience symptoms of anxiety (eg, “Feeling nervous, anxious, or on edge”) on a scale of 0 (“not at all”) to 3 (“nearly every day”). Internal reliability for the GAD-7 was good (α=.82 and α=.87 for baseline and follow-up, respectively).

 4-Item Perceived Stress Scale

The 4-item Perceived Stress Scale (PSS-4) [63] is a 4-item scale assessing the frequency with which individuals experience various symptoms of stress (eg, “How often have you felt that you were unable to control the important things in your life?”) on a scale of 0 (“never”) to 4 (“very often”). Internal reliability for the PSS-4 was adequate (α=.68 and α=.69 for baseline and follow-up, respectively).

 8-Item Patient Health Questionnaire Depression Scale

The 8-item Patient Health Questionnaire depression scale (PHQ-8) [64] is an 8-item scale that assesses the extent to which participants experience feelings of depression (eg, “feeling down, depressed, or hopeless” or “little interest or pleasure in doing things”) on a scale of 0 (“not at all”) to 3 (“nearly every day”). Internal reliability for the PHQ-8 was good (α=.84 and α=.85 for baseline and follow-up, respectively).

 Difficulties in Emotion Regulation Scale–Short Form

The Difficulties in Emotion Regulation Scale–Short Form (DERS-SF) [65,66] is an 18-item scale assessing emotion dysregulation. It comprises six subscales: emotional awareness, clarity about the nature of one’s emotions, acceptance of one’s emotions, access to effective emotion regulation strategies, ability to engage in goal-directed activities while experiencing negative emotions, and ability to manage one’s impulses during negative emotions. These subscales (α=.74-.91 and α=.76-.91) and the DERS-SF total score (α=.89 at both time points) demonstrated good internal consistency at baseline and follow-up, respectively.

 Life Orientation Test–Revised

The Life Orientation Test–Revised (LOT-R) [67] is a 10-item scale that assesses trait optimism. Individuals are asked to rate their agreement with each statement (eg, “In uncertain times, I usually expect the best.”) on a scale of 0 (“strongly disagree”) to 4 (“strongly agree”). Internal reliability for the LOT-R was good (α=.86 and α=.85 at baseline and follow-up, respectively).

 Statistical Analysis

Analyses were conducted in SPSS software (version 27; IBM Corp). For acceptability and feasibility, survey responses were descriptively analyzed with mean scores and percentages of participants that chose each response. For open-ended acceptability responses, content-analyzed categories are presented descriptively with the percentage of responses that fall into each category. Descriptive statistics were also conducted for engagement measures to evaluate feasibility. For preliminary outcomes, paired 2-tailed t tests were conducted to evaluate changes on all quantitative variables from baseline to week 4. Both per-protocol and intention-to-treat analyses were conducted. The per-protocol sample consisted of participants who completed both assessments (n=113) and included those who started the program but stopped using it. Intention-to-treat analyses included data from all participants who began the study (N=185); baseline scores were carried forward for participants who did not complete the week-4 assessment. Effect sizes were calculated using Cohen d [68].

 Results Participant Characteristics

Participants’ demographic characteristics are presented in Table 1. A total of 185 unique Noom Mood users enrolled in the study and completed the baseline survey. Of these, 113 (62.1%) participants completed the follow-up survey. Participants who completed both baseline and follow-up surveys did not differ significantly from those who completed only the baseline survey in terms of any demographic variables or baseline survey values.

Participant characteristics.

Demographics Per-protocol sample (n=113) Intention-to-treat sample (N=185)
Age (years), mean (SD) 36.8 (9.8) 37.3 (10.4)
Gender, n (%)
Male 15 (13.3) 32 (17.3)
Female 94 (83.2) 141 (76.2)
Other 2 (1.8) 3 (1.6)
Prefer not to say or N/Aa 2 (1.8) 9 (4.9)
Ethnicity, n (%)
Hispanic 13 (11.5) 20 (10.8)
Not Hispanic 97 (85.8) 153 (82.7)
Prefer not to say or N/A 3 (2.7) 12 (6.5)
Race, n (%)
White 99 (87.6) 153 (82.7)
Black or African American 5 (4.4) 7 (3.8)
Asian or Pacific Islander 3 (2.7) 11 (5.9)
Other 0 (0) 1 (0.5)
Prefer not to say or N/A 6 (5.3) 13 (7.0)
Employment status, n (%)
Employed  88 (77.8) 144 (77.8)
Not employed 12 (10.6) 18 (9.7)
Retired 1 (0.9) 2 (1.1)
Disabled 5 (4.4) 6 (3.2)
Student 5 (4.4) 6 (3.2)
Prefer not to say or N/A 2 (1.8) 9 (4.9)
Education, n (%)
High school, GEDb, or less education 7 (6.2) 10 (5.4)
Some college or associate degree 24 (21.2) 37 (20.0)
College graduate 45 (39.8) 67 (36.2)
Graduate degree  35 (31.0) 62 (33.5)
Prefer not to say or N/A 2 (1.8) 9 (4.9)

aN/A: not applicable.

bGED: General Education Development.

 Feasibility

Responses to the SUS are presented in Table 2. As noted above, scores of 68 or higher on the SUS indicate above-average ratings of system usability. A majority (79/109, 72.5%) of participants had overall system usability scores of 68 or higher (mean 77.40, SD 19.45), which is considered an indication of good usability [59]. Most participants reported that the program was easy to use (85/110, 77.3%), and they thought that other people would be able to learn to use the program very quickly (93/109, 85.3%).

Program engagement data are presented in Table 3. Engagement data are presented as weekly averages (ie, the number of times the participant engaged in the behavior over the course of the study divided by the total number of weeks). Participants engaged within the app several times per week on average. Over 4 weeks, the per-protocol sample averaged 14.1 (SD 9.02) app opens, with 2 mean app opens per week. They had an average of 12.1 days with an in-app action, amounting to 1.7 active days per week. The intention-to-treat sample opened the app, on average, 13.7 (SD 8.6) times over 4 weeks, with an average of 1.96 app opens per week. They completed at least one in-app action on an average of 11.2 (SD 8.7) days, which amounted to 1.6 active days per week.

Participants reporting good feasibility and acceptability.

Survey measurea Value
System Usability Scale item, n (%)
I would like to use this program frequently. (n=109) 62 (56.9)
I found the program unnecessarily complex.b (n=110) 81 (73.6)
I thought the program was easy to use. (n=110) 85 (77.3)
I would need the support of a technical person to be able to use this program.b (n=108) 95 (88.0)
I found the various functions in this program were well integrated. (n=109) 76 (69.7)
I thought there was too much inconsistency in the program.b (n=109) 87 (79.8)
I would imagine that most people would learn to use this program very quickly. (n=109) 93 (85.3)
I found the program very cumbersome to use.b (n=109) 77 (70.6)
I felt very confident using the program. (n=108) 78 (72.2)
I needed to learn a lot of things before I could get going with the program.b (n=107) 89 (83.2)
System Usability Scale score of 68 or higher (n=109), n (%) 79 (72.5)
System Usability Scale overall score, mean (SD) 77.4 (19.4)
Credibility and Expectancy Questionnaire item, n (%)
At this point, how logical does the program seem to you? (n=110) 101 (91.8)
How successful do you think this program was in reducing stress and anxiety? (n=109) 83 (76.1)
How confident would you be in recommending this program to a friend who experiences stress and anxiety? (n=108) 87 (80.6)
By the end of the program, how much improvement in stress and anxiety do you think will occur?c (n=108) 63 (58.3)
At this point, how much do you really feel that the program will help to reduce stress and anxiety? (n=108) 85 (78.7)
By the end of this program, how much improvement in stress and anxiety do you feel will occur?c (n=107) 62 (57.9)

aThe table includes participants who chose 4 or greater (out of 5) on the System Usability Scale or 5 or greater (out of 9) on the Credibility and Expectancy Questionnaire, except where indicated.

bThese participants chose 2 or less (out of 5) on the System Usability Scale.

cThese participants chose at least 50% out of 100%.

Average total engagement over 4 weeks.

Type of engagement Per-protocol sample (n=110)a, mean (SD) Intention-to-treat sample (n=181)a, mean (SD)
App opens 14.11 (9.02) 13.72 (8.60)
Articles read 37.12 (28.87) 34.39 (27.44)
Mood logs 10.59 (9.54) 10.00 (10.79)
Days with one in-app action 12.14 (9.03) 11.24 (8.68)
Messages sent to coach 9.71 (10.33) 8.24 (9.09)
Activitiesb 1.33 (2.68)  1.30 (2.54)

aSample sizes represent all participants for whom matching data from the database could be identified.

bActivities were calculated over 3 weeks because one offline activity was not tracked by the program.

 Acceptability

Responses to the CEQ are presented in Table 2. Of note, the table displays the frequency and percentage of participants who chose at least a 5 (“somewhat”) out of 9 (“very much”) on the CEQ. The vast majority of participants (101/110, 91.8%) rated the program as at least somewhat logical (mean 7.1, SD 1.9, range 1-9). Most (83/109, 76.1%) thought the program was at least somewhat successful at reducing stress and anxiety (mean 5.6, SD 2.2, range 1-9). Many participants (87/108, 80.6%) also felt at least somewhat confident in recommending the program to a friend (mean 6.1, SD 2.3, range 1-9). Most participants (85/108, 78.7%) felt the program would help to reduce stress and anxiety at least somewhat (mean 5.7, SD 2.3), with more than half (63/108, 58.3%) expecting it to reduce their stress or anxiety by 50% or more (mean 4.9, SD 2.5, with 0 referring to 0% and 10 referring to 100%).

Responses to the Program Satisfaction Questionnaire are presented in Table 4. Participants reported benefiting most from the skills and techniques they learned or practiced (eg, breathing techniques and thought reframing; 38/106, 35.8%). Participants also reported benefiting from the program’s features or capabilities (eg, mood tracking and articles; 31/106, 29.2%) and greater awareness (eg, learning and reflection) encouraged by the program (30/106, 28.3%). Specifically, participants found the articles (18/106, 17.0%), coaching (18/106, 17.0%), and qualities of the program (eg, manageable, convenient, and “great” attitude; 16/106, 15.1%) to be the most helpful parts of Noom Mood.

For potential areas of improvement, most participants did not provide a response or indicated that they had no suggested improvements (37/106, 34.9%). The next most common response was “other” (21/106, 19.8%), or participants requested a new feature or program idea (19/106, 17.9%). “Other” responses included increasing the frequency of reminders, expanding areas of content (eg, support for procrastination), and slowing the pace of tasks. Participants also preferred a lower cost (16/106, 15.1%), with some mentioning the potential to be reimbursed, as well as a more personalized experience (9/106, 8.5%) and greater flexibility (9/106, 8.5%), such as the ability to progress while skipping articles, accessing future articles, or repeating an activity for another week.

When asked to describe the least helpful parts of Noom Mood, most participants did not provide a response (40/106, 37.7%). The next most common response was “other” (21/106, 19.8%); responses noted that the program contained too much repetition and that the pacing of the program needed improvement. Lastly, some participants (17/106, 16.0%) described coaching as the least helpful aspect of the program, noting that they would prefer to interact with a coach with specialized expertise or to receive more personalized responses.

Response frequencies in each category for the Program Satisfaction Questionnaire.

Category Participant responses (n=106), n (%)a
Main benefit of Noom Mood
Skills and techniques 38 (35.8)
Program features or capabilities 31 (29.2)
Awareness (ie, learning and reflection) 30 (28.3)
Emotional experience and management 27 (25.5)
Other 18 (17.0)
None or no response 15 (14.2)
Areas to improve
None or no response 37 (34.9)
Other 21 (19.8)
New feature or program idea 19 (17.9)
Cost 16 (15.1)
Coaching 15 (14.2)
Personalization 9 (8.5)
Flexibility 9 (8.5)
Articles 6 (5.7)
Activities 3 (2.8)
Most helpful part of Noom Mood
Coaching 18 (17.0)
Articles 18 (17.0)
None or no response 17 (16.0)
Qualities of the program 16 (15.1)
Skills and techniques 15 (14.2)
Activities 14 (13.2)
Awareness (ie, learning and reflection) 9 (8.5)
Other 9 (8.5)
Mood tracking 5 (4.7)
Everything 3 (2.8)
Least helpful part of Noom Mood
None or no response 40 (37.7)
Other 21 (19.8)
Coaching 17 (16.0)
Activities 9 (8.5)
Mood tracking 7 (6.6)
Articles 6 (5.7)
Personalization and interactivity 5 (4.7)
Cost 4 (3.8)
Everything 2 (1.9)

aEach response could be placed in more than one category. Categories were derived from individuals’ open-ended responses.

 Symptom and Well-being Outcomes

From baseline to 4 weeks, there was a significant reduction in anxiety symptoms for both per-protocol samples (Table 5; t112=10.92, P<.001, d=1.03) and intention-to-treat samples (t184=9.48, P<.001, d=0.70) with large and medium effect sizes, respectively. There was also a significant improvement in perceived stress (per-protocol sample: t112=7.69, P<.001, d=0.72; intention-to-treat sample: t184=7.09, P<.001, d=0.52) and depressive feelings (per-protocol sample: t110=7.88, P<.001, d=0.75; intention-to-treat sample: t181=7.40, P<.001, d=0.55) with medium effect sizes. Finally, there were significant improvements in emotion regulation (per-protocol sample: t105=5.93, P<.001, d=0.58; intention-to-treat sample: t178=5.79, P<.001, d=0.43) and optimism (per-protocol sample: t104=–5.04, P<.001, d=–0.49; intention-to-treat sample: t175=–5.15, P<.001, d=–0.39) with small to medium effect sizes. 

Symptom and well-being outcomes from baseline to 4 weeks.

Outcome Per-protocol sample (n=113)a Intention-to-treat sample (N=185)b
Baseline,  mean (SD) 4 weeks, mean (SD) ΔMean (% change)c P value Effect sized Baseline, mean (SD) 4 weeks, mean (SD) ΔMean (% change)c P value Effect sized
Anxiety symptoms (GAD-7e) 13.30 (4.31) 8.54 (4.61) –4.76 (–35.81) <.001 1.03 13.28 (4.39) 10.18 (5.14) –3.10 (–23.32) <.001 0.70
Perceived stress (PSS-4f) 8.96 (2.39) 7.08 (2.29) –1.88 (–21.03) <.001 0.72 8.89 (2.41) 7.73 (2.48) –1.16 (–13.07) <.001 0.52
Depressive feelings (PHQ-8g) 11.67 (5.47) 7.77 (4.98) –3.90 (–33.39) <.001 0.75 11.99 (5.55) 9.40 (5.66) –2.59 (–21.61) <.001 0.55
Emotion regulation (DERS-SFh) 45.97 (11.86) 39.39 (11.30) –6.57 (–14.30)  <.001 0.58 47.01 (13.09) 42.95 (13.49) –4.05 (–8.63)  <.001 0.43
Optimism (LOT-Ri) 7.05 (3.49) 8.16 (3.15) 1.11 (15.75) <.001 0.49 7.33 (3.57) 8.09 (3.39) 0.75 (10.30)  <.001 0.39

aPer-protocol analyses only included participants who completed both survey assessments.

bFor intention-to-treat analyses, baseline responses were carried forward for nonresponders.

cNegative values indicate decreases compared to baseline.

dEffect sizes constitute Cohen d.

eGAD-7: 7-item Generalized Anxiety Disorder scale.

fPSS-4: 4-item Perceived Stress Scale.

gPHQ-8: 8-item Patient Health Questionnaire depression scale.

hDERS-SF: Difficulties in Emotion Regulation Scale–Short Form; negative values on the DERS-SF indicate better emotional regulation (ie, fewer difficulties with emotional regulation).

iLOT-R: Life Orientation Test–Revised; positive values on the LOT-R indicate more optimism.

 Discussion Principal Findings

In reviews of mental health programs, researchers have voiced concerns about limited published research on commercial programs, and that programs either have limited public engagement or are not based on evidence-based theory [18,25-27,31,32]. Given the identified need for evidence from this type of commercial program [25,31], this pilot study evaluated the feasibility, acceptability, and preliminary outcomes of Noom Mood, which is widely publicly available, based on CBT and MBSR techniques, designed to encourage engagement among the general public, and includes personal coaching. Our results suggest that the program was usable, feasible, and acceptable to participants. In addition, self-reported anxiety symptoms, stress, depressive feelings, emotion regulation, and optimism improved from baseline to 4 weeks.

 Feasibility and Acceptability Feasibility

Overall, participants rated the program as feasible. The average system usability score was 77.4, which surpasses the threshold for good usability [69], and more than 75% of participants reported that the program was easy to use. These scores are in line with feasibility and usability scores from other mobile programs [70-73]. Similar to levels of engagement reported in studies of comparable mobile mental health programs [46,70,74], participants in this study engaged with Noom Mood regularly, opening the program approximately two times per week and performing an action within the app once every 2 to 3 days (11 of 28 days). Participants engaged most with the articles and least with activities. Of note, it is possible that participants completed activities offline throughout the week, which is how they were designed, but did not mark them as complete in the app. As such, it is likely that the data collected on activities underestimate participant engagement in this aspect of Noom Mood, given that many activities focus on offline experiences (eg, practicing breathing exercises or grounding techniques). Future studies will aim to assess actions completed offline in relationship to symptom outcomes.

 Acceptability

The vast majority of participants found the program to be logical (92%) and effective at reducing stress and anxiety (76%). Importantly, 81% of participants felt confident in recommending the program to a friend. These findings are similar to other studies of mobile mental health programs and suggest that the program was perceived to be acceptable to users [35,70,73]. Additionally, at the follow-up assessment, more than half of the participants reported that they expected that the program would eventually reduce their stress or anxiety by an additional 50% or more. Future work should investigate long-term outcomes and whether these participant expectations are borne out.

Participants reported benefiting most from skills training; program features such as articles, activities, and coaching; learning to better manage their emotions; and reflective processes such as learning, reflecting, and increasing their awareness. Participants reported benefiting from taking the time to reflect on how they were feeling and increasingly becoming aware of their emotions and thought patterns. Many participants also mentioned benefiting from the structure and accountability of a designated program. Participants appreciated the overall tenor of the program; one participant reflected that “the attitude it strikes is a great balance of cheeky humor but realistic so it’s not overly strict nor overly cheesy. Makes me connect with it well and stick with it.” Other participants, however, reported that they hoped for a more serious tone to the articles. At the time of the study, the program incorporated jokes and hashtags for the sake of relatability, and has since been modified in response to participant feedback. 

Participants also indicated that the program could be improved to better help individuals progress in a way that best suits an individual’s idiosyncratic wants or needs. For example, some participants wanted a slower pace, whereas others requested more daily reminders. Additionally, some participants provided feedback that they wanted more specialized interactions with coaches. While individuals were informed that Noom Mood is not a replacement for therapy and does not provide clinical assessment or treatment, it is possible that participants were expecting the coaching feature to function more similarly to therapy. However, some participants provided feedback stating that responses given by coaches did not feel personalized and felt too generic. It is also possible that some participants may not have been good candidates for a self-help approach. As mentioned previously, in the literature, there is limited understanding of how participants would experience a commercial mobile mental health program with personal coaching, rather than therapy. This study contributes initial understanding that, in this context, coaching can be helpful, but it can also raise confusion about the role of a coach when providing guidance and support rather than therapy. Future iterations of the program should, thus, be sure to set expectations for this feature clearly. 

Participants also relayed some suggestions for program improvements that would provide support in varying environments or situations, such as support for moms with young children, skills to reduce procrastination, video and audio recordings, and easily accessible summaries of activities or articles, all of which should be considered in future programs. Since the time of the study, audio recordings have been added to the program. Some participants reported that they would prefer that the program be offered at a lower cost, and some mentioned they would like the program to be covered by health insurance plans. In order to increase accessibility, future initiatives and programs should consider efforts to provide reimbursable experiences (eg, through employee wellness initiatives).

 Preliminary Outcomes Anxiety Symptoms, Perceived Stress, and Depressive Feelings

From baseline to 4 weeks, anxiety symptoms improved by 36% (d=1.03) in per-protocol analyses and 23% (d=0.70) in intention-to-treat analyses. In addition, stress reductions were 21% (d=0.72, per-protocol analysis) and 13% (d=0.52, intention-to-treat analysis), and depressive feelings decreased by 33% (d=0.75, per-protocol analysis) and 22% (d=0.55, intention-to-treat analysis). These effect sizes are comparable to those reported in studies of other mobile mental health programs with the same study length and outcome measures [44,75-80]. Specifically, anxiety and stress decreased in ways that were comparable to or greater than anxiety reductions shown in previous studies, whereas depression showed comparable, though smaller, effect sizes [44,75,78,79]. Of course, this may reflect the fact that the program focuses more on stress and anxiety management than on depression. Of all our outcome measures, anxiety showed the biggest effect sizes, which contrasts with some studies that have found that anxiety scores did not improve as much as other symptom measures, such as depression [75,80].

 Emotion Regulation and Optimism 

In this study, we found that emotion regulation improved by 14% (d=0.58, per-protocol analysis) and 8.6% (d=0.43, intention-to-treat analysis). Emotion dysregulation is hypothesized to underpin a wide range of psychological difficulties [81]; in fact, transdiagnostic interventions, such as DBT or the Unified Protocol [82], focus on emotion dysregulation as the primary treatment target. Notably, however, emotion regulation is rarely included as an outcome variable in mobile mental health programs, despite its empirical and theoretical relevance to mental health and well-being [19]. In two studies of mHealth programs conducted with young adults [83] and homeless youth [84] that measured emotion regulation as an outcome variable, results showed no significant improvements in emotion regulation capacity. 

We found significantly higher optimism at 4 weeks compared to baseline (15.7% or d=0.49, per-protocol analysis; 10% or d=0.39, intention-to-treat analysis). To our knowledge, this is the first mobile mental health study to measure changes in optimism, though some studies of mobile mental health programs have found improvements in other positive psychological constructs, such as life satisfaction, general mental well-being, or quality of life [46,85-88]. A robust literature base demonstrates that optimism is inversely correlated with depression and anxiety and positively correlated with measures of life satisfaction and self-reported health variables [89,90]. Importantly, optimism may influence physical and mental health by encouraging adaptive coping [85]. Consistent with previous findings, both baseline and 4-week optimism scores were significantly negatively correlated with time-matched anxiety symptoms, stress, and depressive feelings, and optimism scores were positively correlated with emotion regulation (ie, higher optimism is correlated with greater capacity to regulate one’s emotions). Future studies should evaluate optimism and its associations with other mental health outcomes.

 Limitations

This pilot study had several limitations. First, without a control group, it was not possible to separate the effects of the program itself from improvement over time (ie, regression to the mean and maturation). In addition, other interventions were uncontrolled; that is, program participants may have been participating in active therapy or may have been taking psychotropic medications while they were participating in this study. Nevertheless, it is unlikely that these findings are purely spurious, as the effect sizes are similar to those found in active treatment groups in RCTs, and they are much larger than those found in control groups (eg, see Bakker et al [75]). Now that preliminary feasibility and acceptability have been established, future studies should use randomized designs to confirm that these results were due to the program itself. Also, the study was conducted over 4 weeks, and it is unclear whether results would change over longer periods of time. Further, the study examined the program as a whole, making it difficult to isolate which specific program components led to changes in outcomes. Future studies should use causal methods to explore this further. In addition, the sample was primarily female, White, and highly educated, which is typical of studies of mobile mental health programs [19]. Future research should evaluate to what extent these results would generalize to other populations and actively recruit from hard-to-reach populations. Lastly, this study did not assess other variables that may have caused improvement in symptoms, such as psychiatric services, individual or group therapy, and participants’ use of other self-help materials.

 Conclusions

In this study, we explored the usability, feasibility, acceptability, and preliminary effectiveness of Noom Mood, a publicly available, mobile mental well-being program based on CBT and MBSR with personal coaching. The program follows 11 of Bakker et al’s [5] evidence-based recommendations for mobile mental health programs: it is based on CBT; addresses both anxiety and low mood; is designed for use by nonclinical populations; includes reporting of thoughts, feelings, and behaviors; recommends activities; provides mental health information; encourages non–technology-based activities; includes gamification or intrinsic motivation to engage; shows logs of past app use (eg, patterns of logged mood); uses reminders to engage (eg, messages from the coach); and provides a simple and intuitive interface and interactions. Our results suggest that Noom Mood was usable, feasible, and acceptable to participants, with promising preliminary improvements in anxiety symptoms, stress, depressive feelings, emotion regulation, and optimism. Future directions should include (1) the incorporation of changes suggested by participants in this study and (2) more rigorous testing of outcome variables, such as through randomized designs.

 Abbreviations ACT

acceptance and commitment therapy

 CBT

cognitive behavioral therapy

 CEQ

Credibility and Expectancy Questionnaire

 DBT

dialectical behavior therapy

 DERS-SF

Difficulties in Emotion Regulation Scale–Short Form

 GAD-7

7-item Generalized Anxiety Disorder scale

 LDA

latent Dirichlet allocation

 LOT-R

Life Orientation Test–Revised

 MBSR

mindfulness-based stress reduction

 mHealth

mobile health

 PHQ-8

8-item Patient Health Questionnaire depression scale

 PSS-4

4-item Perceived Stress Scale

 RCT

randomized controlled trial

 SUS

System Usability Scale

Authors MM, ASH, ESM, CNM, HB, and AM are employees at Noom, Inc, and have received salary and stock options for their employment. LR received payment from Noom, Inc, for their role as a consultant on this project and for their contribution as an author on this paper. LR is also a co-owner of the Triangle Area Psychology Clinic; a consultant to, and a DBT trainer for, Behavioral Tech, LLC; and an employee of University of North Carolina School of Medicine. There are no specific conflicts to report with those entities, and none of those entities were involved in their contribution to this project.

 Mental health: Strengthening our response World Health Organization 2018 03 30 2021-12-01 https://www.who.int/news-room/fact-sheets/detail/mental-health-strengthening-our-response Kessler RC Berglund P Demler O Jin R Merikangas KR Walters EE Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication Arch Gen Psychiatry 2005 06 62 6 593 602 10.1001/archpsyc.62.6.593 15939837 62/6/593 Remes O Brayne C van der Linde R Lafortune L A systematic review of reviews on the prevalence of anxiety disorders in adult populations Brain Behav 2016 07 6 7 e00497 10.1002/brb3.497 27458547 BRB3497 PMC4951626 Demyttenaere K Bruffaerts R Posada-Villa J Gasquet I Kovess V Lepine JP Angermeyer MC Bernert S de GG Morosini P Polidori G Kikkawa T Kawakami N Ono Y Takeshima T Uda H Karam EG Fayyad JA Karam AN Mneimneh ZN Medina-Mora ME Borges G Lara C de GR Ormel J Gureje O Shen Y Huang Y Zhang M Alonso J Haro JM Vilagut G Bromet EJ Gluzman S Webb C Kessler RC Merikangas KR Anthony JC Von KMR Wang PS Brugha TS Aguilar-Gaxiola S Lee S Heeringa S Pennell B Zaslavsky AM Ustun TB Chatterji S WHO World Mental Health Survey Consortium Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys JAMA 2004 06 02 291 21 2581 2590 10.1001/jama.291.21.2581 15173149 291/21/2581 Bakker D Kazantzis N Rickwood D Rickard N Mental health smartphone apps: Review and evidence-based recommendations for future developments JMIR Ment Health 2016 03 3 1 e7 10.2196/mental.4984 26932350 v3i1e7 PMC4795320 Arango C Díaz-Caneja CM McGorry PD Rapoport J Sommer IE Vorstman JA McDaid D Marín O Serrano-Drozdowskyj E Freedman R Carpenter W Preventive strategies for mental health Lancet Psychiatry 2018 07 5 7 591 604 10.1016/S2215-0366(18)30057-9 29773478 S2215-0366(18)30057-9 Canady VA APA stress report amid COVID-19 points to parental challenges Mental Health Weekly 2020 05 29 2022-04-09 https://onlinelibrary.wiley.com/doi/10.1002/mhw.32385 Witters D Harter J Worry and stress fuel record drop in US life satisfaction Gallup 2020 05 08 2021-12-01 https://news.gallup.com/poll/310250/worry-stress-fuel-record-drop-life-satisfaction.aspx Walker ER McGee RE Druss BG Mortality in mental disorders and global disease burden implications: A systematic review and meta-analysis JAMA Psychiatry 2015 04 72 4 334 341 10.1001/jamapsychiatry.2014.2502 25671328 2110027 PMC4461039 Whiteford A Degenhardt L Rehm J Baxter AJ Ferrari AJ Erskine HE Charlson FJ Norman RE Flaxman AD Johns N Burstein R Murray CJL Vos T Global burden of disease attributable to mental and substance use disorders: Findings from the Global Burden of Disease Study 2010 Lancet 2013 11 09 382 9904 1575 1586 10.1016/S0140-6736(13)61611-6 23993280 S0140-6736(13)61611-6 Andrade LH Alonso J Mneimneh Z Wells JE Al-Hamzawi A Borges G Bromet E Bruffaerts R de Girolamo G de Graaf R Florescu S Gureje O Hinkov HR Hu C Huang Y Hwang I Jin R Karam EG Kovess-Masfety V Levinson D Matschinger H O'Neill S Posada-Villa J Sagar R Sampson NA Sasu C Stein DJ Takeshima T Viana MC Xavier M Kessler RC Barriers to mental health treatment: Results from the WHO World Mental Health surveys Psychol Med 2014 04 44 6 1303 1317 10.1017/S0033291713001943 23931656 S0033291713001943 PMC4100460 Weil T Insufficient dollars and qualified personnel to meet United States mental health needs J Nerv Ment Dis 2015 04 203 4 233 240 10.1097/NMD.0000000000000271 25816044 00005053-201504000-00001 Leong FTL Kalibatseva Z Cross-cultural barriers to mental health services in the United States Cerebrum 2011 03 2011 5 23447774 PMC3574791 Handley TE Kay-Lambkin FJ Inder KJ Lewin TJ Attia JR Fuller J Perkins D Coleman C Weaver N Kelly BJ Self-reported contacts for mental health problems by rural residents: Predicted service needs, facilitators and barriers BMC Psychiatry 2014 09 06 14 249 10.1186/s12888-014-0249-0 25193400 s12888-014-0249-0 PMC4172961 Comer JS Barlow DH The occasional case against broad dissemination and implementation: Retaining a role for specialty care in the delivery of psychological treatments Am Psychol 2014 01 69 1 1 18 10.1037/a0033582 23915401 2013-27498-001 PMC4260460 Taylor CB Fitzsimmons-Craft EE Graham AK Digital technology can revolutionize mental health services delivery: The COVID-19 crisis as a catalyst for change Int J Eat Disord 2020 07 53 7 1155 1157 10.1002/eat.23300 32449523 PMC7280562 Atherly A Van Den Broek-Altenburg E Hart V Gleason K Carney J Consumer reported care deferrals due to the COVID-19 pandemic, and the role and potential of telemedicine: Cross-sectional analysis JMIR Public Health Surveill 2020 09 14 6 3 e21607 10.2196/21607 32833661 v6i3e21607 PMC7498465 Huckvale K Nicholas J Torous J Larsen ME Smartphone apps for the treatment of mental health conditions: Status and considerations Curr Opin Psychol 2020 12 36 65 70 10.1016/j.copsyc.2020.04.008 32553848 S2352-250X(20)30052-X Eisenstadt M Liverpool S Infanti E Ciuvat RM Carlsson C Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: Systematic review and meta-analysis JMIR Ment Health 2021 11 08 8 11 e31170 10.2196/31170 34747713 v8i11e31170 PMC8663676 Hwang WJ Ha JS Kim MJ Research trends on mobile mental health application for general population: A scoping review Int J Environ Res Public Health 2021 03 02 18 5 2459 10.3390/ijerph18052459 33801537 ijerph18052459 PMC7967596 Hilty DM Chan S Hwang T Wong A Bauer AM Advances in mobile mental health: Opportunities and implications for the spectrum of e-mental health services Mhealth 2017 08 3 34 10.21037/mhealth.2017.06.02 28894744 mh-03-2017.06.02 PMC5583042 Price M Yuen EK Goetter EM Herbert JD Forman EM Acierno R Ruggiero KJ mHealth: A mechanism to deliver more accessible, more effective mental health care Clin Psychol Psychother 2014 08 21 5 427 436 10.1002/cpp.1855 23918764 PMC3926903 Rathbone AL Prescott J The use of mobile apps and SMS messaging as physical and mental health interventions: Systematic review J Med Internet Res 2017 08 24 19 8 e295 10.2196/jmir.7740 28838887 v19i8e295 Proctor E Silmere H Raghavan R Hovmand P Aarons G Bunger A Griffey R Hensley M Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda Adm Policy Ment Health 2011 03 38 2 65 76 10.1007/s10488-010-0319-7 20957426 PMC3068522 Marshall JM Dunstan DA Bartik W Apps with maps-Anxiety and depression mobile apps with evidence-based frameworks: Systematic search of major app stores JMIR Ment Health 2020 06 24 7 6 e16525 10.2196/16525 32579127 v7i6e16525 PMC7381031 Wasil AR Weisz JR DeRubeis RJ Three questions to consider before developing a mental health app World Psychiatry 2020 06 19 2 252 253 10.1002/wps.20757 32394568 PMC7215053 Baumel A Muench F Edan S Kane JM Objective user engagement with mental health apps: Systematic search and panel-based usage analysis J Med Internet Res 2019 09 25 21 9 e14567 10.2196/14567 31573916 v21i9e14567 PMC6785720 Bakker D Rickard N Engagement in mobile phone app for self-monitoring of emotional wellbeing predicts changes in mental health: MoodPrism J Affect Disord 2018 12 227 432 442 10.1016/j.jad.2017.11.016 29154165 S0165-0327(17)31678-6 Aizenstros A Bakker D Hofmann SG Curtiss J Kazantzis N Engagement with smartphone-delivered behavioural activation interventions: A study of the MoodMission smartphone application Behav Cogn Psychother 2021 09 49 5 569 581 10.1017/S1352465820000922 34396942 S1352465820000922 Bakker D Rickard N Engagement with a cognitive behavioural therapy mobile phone app predicts changes in mental health and wellbeing: MoodMission Aust Psychol 2020 11 12 54 4 245 260 10.1111/ap.12383 Marshall JM Dunstan DA Bartik W Clinical or gimmickal: The use and effectiveness of mobile mental health apps for treating anxiety and depression Aust N Z J Psychiatry 2020 01 54 1 20 28 10.1177/0004867419876700 31552747 Lee RA Jung ME Evaluation of an mHealth app (DeStressify) on university students' mental health: Pilot trial JMIR Ment Health 2018 01 23 5 1 e2 10.2196/mental.8324 29362209 v5i1e2 PMC5801522 Lecomte T Potvin S Corbière M Guay S Samson C Cloutier B Francoeur A Pennou A Khazaal Y Mobile apps for mental health issues: Meta-review of meta-analyses JMIR Mhealth Uhealth 2020 05 29 8 5 e17458 10.2196/17458 32348289 v8i5e17458 PMC7293054 Lau N O'Daffer A Colt S Yi-Frazier JP Palermo TM McCauley E Rosenberg AR Android and iPhone mobile apps for psychosocial wellness and stress management: Systematic search in app stores and literature review JMIR Mhealth Uhealth 2020 05 22 8 5 e17798 10.2196/17798 32357125 v8i5e17798 PMC7275252 Huberty J Green J Glissmann C Larkey L Puzia M Lee C Efficacy of the mindfulness meditation mobile app "Calm" to reduce stress among college students: Randomized controlled trial JMIR Mhealth Uhealth 2019 06 25 7 6 e14273 10.2196/14273 31237569 v7i6e14273 PMC6614998 Newton A Bagnell A Rosychuk R Duguay J Wozney L Huguet A Henderson J Curran J A mobile phone-based app for use during cognitive behavioral therapy for adolescents with anxiety (MindClimb): User-centered design and usability study JMIR Mhealth Uhealth 2020 12 08 8 12 e18439 10.2196/18439 33289671 v8i12e18439 PMC7755529 Anton MT Greenberger HM Andreopoulos E Pande RL Evaluation of a commercial mobile health app for depression and anxiety (AbleTo Digital+): Retrospective cohort study JMIR Form Res 2021 09 21 5 9 e27570 10.2196/27570 34546170 v5i9e27570 PMC8493460 Solness CL Kroska EB Holdefer PJ O'Hara MW Treating postpartum depression in rural veterans using internet delivered CBT: Program evaluation of MomMoodBooster J Behav Med 2021 08 44 4 454 466 10.1007/s10865-020-00188-5 33052526 10.1007/s10865-020-00188-5 PMC7556583 Hofmann SG Asnaani A Vonk IJJ Sawyer AT Fang A The efficacy of cognitive behavioral therapy: A review of meta-analyses Cognit Ther Res 2012 10 1 36 5 427 440 10.1007/s10608-012-9476-1 23459093 PMC3584580 Grist R Cavanagh K Computerised cognitive behavioural therapy for common mental health disorders, what works, for whom under what circumstances? A systematic review and meta-analysis J Contemp Psychother 2013 9 4 43 4 243 251 10.1007/s10879-013-9243-y Thompson EM Destree L Albertella L Fontenelle LF Internet-based acceptance and commitment therapy: A transdiagnostic systematic review and meta-analysis for mental health outcomes Behav Ther 2021 03 52 2 492 507 10.1016/j.beth.2020.07.002 33622516 S0005-7894(20)30100-3 French K Golijani-Moghaddam N Schröder T What is the evidence for the efficacy of self-help acceptance and commitment therapy? A systematic review and meta-analysis J Contextual Behav Sci 2017 10 6 4 360 374 10.1016/j.jcbs.2017.08.002 25904163 Valentine SE Bankoff SM Poulin RM Reidler EB Pantalone DW The use of dialectical behavior therapy skills training as stand-alone treatment: A systematic review of the treatment outcome literature J Clin Psychol 2015 01 71 1 1 20 10.1002/jclp.22114 25042066 Yang E Schamber E Meyer RML Gold JI Happier healers: Randomized controlled trial of mobile mindfulness for stress management J Altern Complement Med 2018 05 24 5 505 513 10.1089/acm.2015.0301 29420050 Caldeira C Chen Y Chan L Pham V Chen Y Zheng K Mobile apps for mood tracking: An analysis of features and user reviews AMIA Annu Symp Proc 2017 2017 495 504 29854114 PMC5977660 Ahtinen A Mattila E Välkkynen P Kaipainen K Vanhala T Ermes M Sairanen E Myllymäki T Lappalainen R Mobile mental wellness training for stress management: Feasibility and design implications based on a one-month field study JMIR Mhealth Uhealth 2013 07 1 2 e11 10.2196/mhealth.2596 25100683 v1i2e11 PMC4114468 Levin ME Haeger J Pierce B Cruz RA Evaluating an adjunctive mobile app to enhance psychological flexibility in acceptance and commitment therapy Behav Modif 2017 11 41 6 846 867 10.1177/0145445517719661 28689449 Kim H Tietsort C Posteher K Michaelides A Toro-Ramos T Enabling self-management of a chronic condition through patient-centered coaching: A case of an mHealth diabetes prevention program for older adults Health Commun 2020 12 35 14 1791 1799 10.1080/10410236.2019.1663583 31514532 Weisel KK Fuhrmann LM Berking M Baumeister H Cuijpers P Ebert DD Standalone smartphone apps for mental health-A systematic review and meta-analysis NPJ Digit Med 2019 2 118 10.1038/s41746-019-0188-8 31815193 188 PMC6889400 Mohr DC Weingardt KR Reddy M Schueller SM Three problems with current digital mental health research...and three things we can do about them Psychiatr Serv 2017 05 01 68 5 427 429 10.1176/appi.ps.201600541 28412890 PMC6903906 Andersson G Carlbring P Berger T Almlöv J Cuijpers P What makes internet therapy work? Cogn Behav Ther 2009 01 38 Suppl 1 55 60 10.1080/16506070902916400 19675956 913870867 Andersson G Cuijpers P Internet-based and other computerized psychological treatments for adult depression: A meta-analysis Cogn Behav Ther 2009 12 38 4 196 205 10.1080/16506070903318960 20183695 917833314 Mohr DC Duffecy J Ho J Kwasny M Cai X Burns MN Begale M A randomized controlled trial evaluating a manualized TeleCoaching protocol for improving adherence to a web-based intervention for the treatment of depression PLoS One 2013 8 8 8 e70086 10.1371/journal.pone.0070086 23990896 PONE-D-13-05935 PMC3749146 Weiner BJ Lewis CC Stanick C Powell BJ Dorsey CN Clary AS Boynton MH Halko H Psychometric assessment of three newly developed implementation outcome measures Implement Sci 2017 12 29 12 1 108 10.1186/s13012-017-0635-3 28851459 10.1186/s13012-017-0635-3 PMC5576104 Brooke J Thomas B Weerdmeester BA McClleland IL Jordan PW SUS: A 'quick and dirty' usability scale Usability Evaluation in Industry 1996 London, UK Taylor & Francis Ltd 189 194 Schlosser DA Campellone TR Truong B Anguera JA Vergani S Vinogradov S Arean P The feasibility, acceptability, and outcomes of PRIME-D: A novel mobile intervention treatment for depression Depress Anxiety 2017 06 34 6 546 554 10.1002/da.22624 28419621 PMC5634707 Torous J Kiang MV Lorme J Onnela J New tools for new research in psychiatry: A scalable and customizable platform to empower data driven smartphone research JMIR Ment Health 2016 05 05 3 2 e16 10.2196/mental.5165 27150677 v3i2e16 PMC4873624 Devilly GJ Borkovec TD Psychometric properties of the credibility/expectancy questionnaire J Behav Ther Exp Psychiatry 2000 06 31 2 73 86 10.1016/s0005-7916(00)00012-4 Thompson-Hollands J Bentley KH Gallagher MW Boswell JF Barlow DH Credibility and outcome expectancy in the unified protocol: Relationship to outcomes J Exp Psychopathol 2014 03 30 5 1 72 82 10.5127/jep.033712 Blei DM Ng AY Jordan MI Latent Dirichlet allocation J Mach Learn Res 2003 3 993 1022 Landis JR Koch GG The measurement of observer agreement for categorical data Biometrics 1977 03 33 1 159 10.2307/2529310 Spitzer RL Kroenke K Williams JBW Löwe B A brief measure for assessing generalized anxiety disorder: The GAD-7 Arch Intern Med 2006 05 22 166 10 1092 1097 10.1001/archinte.166.10.1092 16717171 166/10/1092 Cohen S Kamarck T Mermelstein R A global measure of perceived stress J Health Soc Behav 1983 385 396 10.2307/2136404 6668417 Kroenke K Strine TW Spitzer RL Williams JBW Berry JT Mokdad AH The PHQ-8 as a measure of current depression in the general population J Affect Disord 2009 04 114 1-3 163 173 10.1016/j.jad.2008.06.026 18752852 S0165-0327(08)00282-6 Gratz K Roemer L Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the Difficulties in Emotion Regulation Scale J Psychopathol Behav Assess 2004 03 26 1 41 54 10.1023/b:joba.0000007455.08539.94 Victor S Klonsky E Validation of a brief version of the Difficulties in Emotion Regulation Scale (DERS-18) in five samples J Psychopathol Behav Assess 2016 5 13 38 4 582 589 10.1007/s10862-016-9547-9 Scheier MF Carver CS Bridges MW Distinguishing optimism from neuroticism (and trait anxiety, self-mastery, and self-esteem): A reevaluation of the Life Orientation Test J Pers Soc Psychol 1994 12 67 6 1063 1078 10.1037//0022-3514.67.6.1063 7815302 Cohen J Statistical Power Analysis for the Behavioral Sciences. 2nd edition 1988 New York, NY Routledge Brooke J SUS: A retrospective J Usability Stud 2013 02 8 2 29 40 Deady M Johnston D Milne D Glozier N Peters D Calvo R Harvey S Preliminary effectiveness of a smartphone app to reduce depressive symptoms in the workplace: Feasibility and acceptability study JMIR Mhealth Uhealth 2018 12 04 6 12 e11661 10.2196/11661 30514694 v6i12e11661 PMC6299234 Mata-Greve F Johnson M Pullmann MD Friedman EC Griffith Fillipo I Comtois KA Arean P Mental health and the perceived usability of digital mental health tools among essential workers and people unemployed due to COVID-19: Cross-sectional survey study JMIR Ment Health 2021 08 05 8 8 e28360 10.2196/28360 34081592 v8i8e28360 PMC8354319 Gordon JS Sbarra D Armin J Pace TWW Gniady C Barraza Y Use of a guided imagery mobile app (See Me Serene) to reduce COVID-19-related stress: Pilot feasibility study JMIR Form Res 2021 10 04 5 10 e32353 10.2196/32353 34546941 v5i10e32353 PMC8491644 Rung AL Oral E Berghammer L Peters ES Feasibility and acceptability of a mobile mindfulness meditation intervention among women: Intervention study JMIR Mhealth Uhealth 2020 06 02 8 6 e15943 10.2196/15943 32442147 v8i6e15943 PMC7298633 Lattie E Cohen KA Winquist N Mohr DC Examining an app-based mental health self-care program, IntelliCare for college students: Single-arm pilot study JMIR Ment Health 2020 10 10 7 10 e21075 10.2196/21075 33037874 v7i10e21075 PMC7585772 Bakker D Kazantzis N Rickwood D Rickard N A randomized controlled trial of three smartphone apps for enhancing public mental health Behav Res Ther 2018 10 109 75 83 10.1016/j.brat.2018.08.003 30125790 S0005-7967(18)30124-4 Mohr DC Tomasino KN Lattie EG Palac HL Kwasny MJ Weingardt K Karr CJ Kaiser SM Rossom RC Bardsley LR Caccamo L Stiles-Shields C Schueller SM IntelliCare: An eclectic, skills-based app suite for the treatment of depression and anxiety J Med Internet Res 2017 01 05 19 1 e10 10.2196/jmir.6645 28057609 v19i1e10 PMC5247623 Kawadler JM Hemmings NR Ponzo S Morelli D Bird G Plans D Effectiveness of a smartphone app (BioBase) for reducing anxiety and increasing mental well-being: Pilot feasibility and acceptability study JMIR Form Res 2020 11 10 4 11 e18067 10.2196/18067 32969341 v4i11e18067 PMC7685925 Oser M Wallace ML Solano F Szigethy EM Guided digital cognitive behavioral program for anxiety in primary care: Propensity-matched controlled trial JMIR Ment Health 2019 04 04 6 4 e11981 10.2196/11981 30946022 v6i4e11981 PMC6470461 Flett JAM Hayne H Riordan BC Thompson LM Conner TS Mobile mindfulness meditation: A randomised controlled trial of the effect of two popular apps on mental health Mindfulness 2018 10 31 10 5 863 876 10.1007/s12671-018-1050-9 Hwang WJ Jo HH Evaluation of the effectiveness of mobile app-based stress-management program: A randomized controlled trial Int J Environ Res Public Health 2019 11 03 16 21 4270 10.3390/ijerph16214270 31684169 ijerph16214270 PMC6862035 Gross JJ Muñoz RF Emotion regulation and mental health Clin Psychol 1995 2 2 151 164 10.1111/j.1468-2850.1995.tb00036.x Barlow DH Harris BA Eustis EH Farchione TJ The unified protocol for transdiagnostic treatment of emotional disorders World Psychiatry 2020 06 19 2 245 246 10.1002/wps.20748 32394551 PMC7215073 Dubad M Elahi F Marwaha S The clinical impacts of mobile mood-monitoring in young people with mental health problems: The MeMO Study Front Psychiatry 2021 12 687270 10.3389/fpsyt.2021.687270 34393850 PMC8363129 Schueller SM Glover AC Rufa AK Dowdle CL Gross GD Karnik NS Zalta AK A mobile phone-based intervention to improve mental health among homeless young adults: Pilot feasibility trial JMIR Mhealth Uhealth 2019 07 02 7 7 e12347 10.2196/12347 31267980 v7i7e12347 Conversano C Rotondo A Lensi E Della Vista O Arpone F Reda MA Optimism and its impact on mental and physical well-being Clin Pract Epidemiol Ment Health 2010 05 14 6 25 29 10.2174/1745017901006010025 20592964 PMC2894461 Mak WW Tong AC Yip SY Lui WW Chio FH Chan AT Wong CC Efficacy and moderation of mobile app-based programs for mindfulness-based training, self-compassion training, and cognitive behavioral psychoeducation on mental health: Randomized controlled noninferiority trial JMIR Ment Health 2018 10 11 5 4 e60 10.2196/mental.8597 30309837 v5i4e60 PMC6231823 Pham Q Khatib Y Stansfeld S Fox S Green T Feasibility and efficacy of an mHealth game for managing anxiety: "Flowy" randomized controlled pilot trial and design evaluation Games Health J 2016 02 5 1 50 67 10.1089/g4h.2015.0033 26536488 Weber S Lorenz C Hemmings N Improving stress and positive mental health at work via an app-based intervention: A large-scale multi-center randomized control trial Front Psychol 2019 10 2745 10.3389/fpsyg.2019.02745 31866915 PMC6908507 Andersson G The benefits of optimism: A meta-analytic review of the life orientation test Pers Individ Dif 1996 11 21 5 719 725 10.1016/0191-8869(96)00118-3 Forgeard M Seligman M Seeing the glass half full: A review of the causes and consequences of optimism Prat Psychol 2012 6 18 2 107 120 10.1016/j.prps.2012.02.002