This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2226) mHealth for Wellness, Behavior Change and Prevention (2754) Anxiety and Stress Disorders (1013) Users' and Patients' Needs for Mental Health Services (382)

Published on in Vol 6, No 4 (2022): April

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/36794, first published .
Feasibility, Acceptability, and Preliminary Outcomes of a Cognitive Behavioral Therapy–Based Mobile Mental Well-being Program (Noom Mood): Single-Arm Prospective Cohort Study

Feasibility, Acceptability, and Preliminary Outcomes of a Cognitive Behavioral Therapy–Based Mobile Mental Well-being Program (Noom Mood): Single-Arm Prospective Cohort Study

Feasibility, Acceptability, and Preliminary Outcomes of a Cognitive Behavioral Therapy–Based Mobile Mental Well-being Program (Noom Mood): Single-Arm Prospective Cohort Study

Authors of this article:

Meaghan McCallum1 Author Orcid Image ;   Annabell Suh Ho1 Author Orcid Image ;   Ellen Siobhan Mitchell1 Author Orcid Image ;   Christine N May1 Author Orcid Image ;   Heather Behr1, 2 Author Orcid Image ;   Lorie Ritschel3, 4 Author Orcid Image ;   Kirk Mochrie3 Author Orcid Image ;   Andreas Michaelides1 Author Orcid Image
Article Authors Cited by (3) Tweetations (8) Metrics

    Original Paper

    1Academic Research, Noom Inc, New York, NY, United States

    2Department of Integrative Health, Saybrook University, Pasadena, CA, United States

    3Triangle Area Psychology Clinic, Durham, NC, United States

    4School of Medicine, University of North Carolina, Chapel Hill, NC, United States

    Corresponding Author:

    Ellen Siobhan Mitchell, PhD

    Academic Research

    Noom Inc

    229 W 28th St, Fl 9

    New York, NY, 10001

    United States

    Phone: 1 631 938 1248

    Email: siobhan@noom.com


    Background: The prevalence of anxiety, depression, and general distress has risen in recent years. Mobile mental health programs have been found to provide support to nonclinical populations and may overcome some of the barriers associated with traditional in-person treatment; however, researchers have voiced concerns that many publicly available mobile mental health programs lack evidence-based theoretical foundations, peer-reviewed research, and sufficient engagement from the public.

    Objective: This study aimed to evaluate the feasibility, acceptability, and preliminary outcomes of Noom Mood, a commercial mobile cognitive behavioral therapy– and mindfulness-based program.

    Methods: In this single-arm prospective cohort study, individuals who joined Noom Mood between August and October 2021 completed surveys at baseline and 4-week follow-up. Per-protocol analyses included those who completed both surveys (n=113), and intention-to-treat analyses included all participants (N=185).

    Results: A majority of the sample reported that the program is easy to use, they felt confident recommending the program to a friend, and they perceived the program to be effective at improving stress and anxiety. There were significant improvements in anxiety symptoms, perceived stress, depressive feelings, emotion regulation, and optimism in both the per-protocol and intention-to-treat analyses (all P<.001). Participants reported benefiting most from learning skills (eg, breathing and cognitive reframing techniques), interacting with the program features, and gaining awareness of their emotions and thought patterns. Participants also made a number of suggestions to improve product functionality and usability.

    Conclusions: Results suggest that Noom Mood is feasible and acceptable to participants, with promising preliminary outcomes. Future studies should build on these results to evaluate the effects of Noom Mood using more rigorous designs.

    JMIR Form Res 2022;6(4):e36794

    doi:10.2196/36794

    Keywords

    mHealth; mobile mental health; mental health; stress; anxiety 


    The World Health Organization stresses the importance of mental health, which they broadly define as a state of “well-being in which an individual realizes his or her own abilities and can cope with the normal stresses of life” [1]. Many individuals are affected by difficulties with mental health [2]; for example, anxiety disorders are highly prevalent worldwide and are estimated to affect 18% of individuals in the United States alone [3,4]. Lifetime prevalence for depression is approximately 17% [2]. Furthermore, it is increasingly recognized that the general population can benefit from mental health support, regardless of whether clinical thresholds for mental illness are met [5,6]: as many as 57% to 84% of US adults have reported subclinical but substantial amounts of stress or worry in recent years [7,8]. Estimates suggest that anxiety, depression, and stress are associated with greater risk of mortality and hundreds of billions of dollars in economic burden per year [9,10]. 

    Although a number of empirically supported treatments for mental health difficulties are available, myriad barriers exist that make it difficult for many people to access traditional in-person support, including cost, long waiting times to see providers, and limited provider availability, especially for individuals living in remote areas [11-15]. The COVID-19 pandemic has also increased barriers to accessing in-person support, potentially increasing willingness to seek digital support [16,17]. In addition, many individuals avoid seeking treatment due to stigma or to mistrust of the mental health system more generally [11,13].

    In recent years, there has been a proliferation of interest in and development of mobile mental health programs. Use of these programs has tripled in recent years [18], and multiple reviews suggest that mobile mental health apps have the capacity to improve mental health and emotion regulation in the general population [19,20]. Mobile mental health has the potential to address many of the aforementioned barriers to treatment [21,22]; perhaps most importantly, mobile mental health allows for support or psychoeducation that is not restricted by time, location, or provider availability. In addition, digital (ie, via smartphone) delivery increases accessibility and autonomy in allowing for largely self-directed care [5,23]. Such programs facilitate self-monitoring of mood or activity, a well-known strategy to change undesired behaviors [5]. Lastly, mobile platforms allow for objective measurement of behavioral indicators, such as the number of articles read, and, therefore, allow individuals to track which strategies are most effective in helping them achieve behavioral change.

    Despite this proliferation of readily accessible mobile mental health programs, researchers have raised several concerns that merit attention and that can be viewed through the lens of implementation science (see Proctor et al [24] for an in-depth discussion of implementation science variables as they apply to outcome studies). First, many mobile health (mHealth) programs available to the public are not based on evidence-based theoretical frameworks [25]. Moreover, users are self-selected, meaning that the problems they are experiencing may or may not map onto the content including the mobile app (ie, problems with appropriateness) [25]. Second, whether evidence based or not, many programs are used briefly and then discarded (ie, problems with adoption) or do not reach a broad enough segment of the population to be useful (ie, problems with penetration) [18,26]. Research has found that thousands of programs have been released on app stores that retain a very limited number of active users over time; for example, studies have shown that 97% of users do not use these mental health apps at day 15 [26,27]. This represents an obvious challenge for mental health programs, as intervention engagement has been associated with better outcomes in a multitude of studies [28-30]. Lastly, few of these mobile mental health programs include a research component to evaluate feasibility, acceptability, or outcomes of any sort; of programs based on theoretical frameworks, only approximately 6.2% have associated peer-reviewed research [25,31,32].

    As such, this study was designed to address these gaps in the literature by examining the feasibility, acceptability, and preliminary outcomes of Noom Mood, a widely available commercial mHealth program that incorporates evidence-based recommendations for mobile mental health programs [5]. In particular, this study aims to contribute to the substantial gap in the evidence base, identified by implementation science researchers and review papers on mobile mental health, in data from commercial programs [25,31,33,34]. Another contribution of this study stems from Noom Mood’s inclusion of personal coaching for guidance and implementation of cognitive behavioral therapy (CBT) techniques, but not clinical therapy. Few studies have examined widely available mental health programs guided by personal coaching; many existing studies examine mental health programs that are entirely self-guided (ie, without individualized coaching support), are designed to provide clinical therapy or serve as an adjunct to therapy, or provide personalized coaching in other contexts (eg, employer-provided coaching or for specific conditions) [35-38]. 

    Noom Mood is a structured, skills-based approach to stress and anxiety management. Noom Mood uses strategies from empirically supported treatments that have been shown to improve mental health outcomes, such as anxiety, depression, and stress (eg, CBT, dialectical behavior therapy [DBT], acceptance and commitment therapy [ACT], and mindfulness-based stress reduction [MBSR]) [39-43]. Importantly, preliminary evidence has shown that CBT and MBSR can be deployed on a mobile platform and that these programs are associated with improvements in mental well-being in nonclinical and clinical populations [23,44]; however, as described previously, more empirical evaluation is needed of evidence-based, commercial programs. Program components include the following: (1) a daily curriculum consisting of psychoeducational articles for users to read, (2) individualized coaching offered through in-app messaging, (3) weekly skills-based activities, and (4) a mood-logging feature. All four components are expected to improve mental well-being (eg, reduce perceived anxiety and depressive symptoms and perceived stress). The curriculum, activities, and coaching were derived from evidence-based frameworks (ie, CBT, DBT, ACT, and MBSR) that have been shown to be effective in improving these outcomes, so these three components would be expected to be most directly related to outcomes. The fourth component of mood logging is based on behavior change techniques of self-monitoring, helping users to build self-awareness of their mood and associated behaviors [45]. More specifically, the daily curriculum was developed in collaboration with clinical psychologists and was designed to translate evidence-based treatments and psychoeducation into a format that is useful for individuals within a self-help framework. For example, each day, participants are presented with a short article that explains conceptual terms and principles (eg, cognitive defusion from ACT), provides practical tips and quizzes to build knowledge, and guides users through a relevant practical activity (eg, how to practice cognitive defusion over the next week; Figure 1). Because of the utility of skills-training activities that help to apply evidence-based principles into daily life [5,46,47], Noom Mood introduces individuals to a short 10- to 15-minute practical activity based on evidence-based frameworks, such as breathing techniques and cognitive reframing at the beginning of each week. The activity is implemented for 1 week, with a practice on day 7 in which individuals reflect on the skill learned and how well it worked for them (Figure 1). Lastly, Noom Mood includes a messaging feature that allows participants to communicate directly with health coaches (Figure 1). Coaches help users to understand and engage in activities, encourage reflection and awareness of patterns, and provide validation for emotional experiences based on CBT techniques. Coaching protocols were adapted to this mental well-being context from the Noom weight management program, for which coaching has been refined and tested and shown to provide guidance on activities, emotional self-awareness, and emotional validation [48]. Noom Mood coaches are trained in CBT techniques but are not licensed clinicians, as Noom Mood does not provide clinical assessment, diagnoses, or treatment and is not a replacement for therapy. The coaching feature was included to address concerns that have been cited in previous studies of evidence-based programs [48-50]. Specifically, human contact from remote coaches within otherwise self-guided digital programs may encourage engagement and improve outcomes [48,51,52]. One randomized controlled trial (RCT) found that engagement check-ins from coaches improved engagement in a web-based depression program [53].

    ‎
    Figure 1. Screenshots of the Noom Mood program.
    View this figure

    The first step in evaluating any new mHealth platform is to investigate stakeholders’ views on the feasibility and acceptability of the proposed product [24,50]. Feasibility is defined as the extent to which end users feel that they could and would use the product in their lives for the purposes for which it was designed [54]. Acceptability is defined as the extent to which stakeholders find the product satisfactory with regard to its content and perceived credibility [54]. Results from feasibility and acceptability testing are then used to refine and update the platform to align with stakeholders’ suggestions more closely. 

    The primary goal of this study was to evaluate the feasibility and acceptability of Noom Mood, as well as to gather preliminary data on whether the program might be associated with improved well-being. We hypothesized that users would find the platform to be feasible and acceptable. Furthermore, we hypothesized that participants who used the program would report some benefit in terms of improved anxiety symptoms, stress, depressive feelings, emotion regulation, and optimism by the end of the 4-week study.


    A single-arm prospective cohort design was used to test feasibility and acceptability of Noom Mood, as well as initial symptom and well-being outcomes.

    Ethics Approval

    The study was approved by the Advarra Institutional Review Board (protocol No. 00055306).

    Procedure and Participants

    Participants were recruited from the pool of individuals who had voluntarily signed up for the Noom Mood program. A randomly selected subset of adults who voluntarily enrolled in the Noom Mood program between August and October 2021 were invited to participate. All participants provided informed consent prior to participation. Inclusion criteria for participants were as follows: located within the United States, English speaking, and aged 18 years or older. Participants were invited to complete the baseline questionnaire within 1 business day of signing up for the Noom Mood program. Those who completed the baseline questionnaire were invited to complete the follow-up survey 4 weeks later. Study completers were compensated with a US $20 gift card for their participation. Participants did not receive the program for free during or after the study. The entire study occurred remotely, including online administration of surveys via email.

    Noom Mood Program

    The Noom Mood program was deployed as described above. At the time of this study, approximately 15 psychoeducational articles were presented to participants each week. In addition to the curriculum, participants had access to mood-logging features, and they were encouraged by coaches to engage in the curriculum and to log their mood once per day.

    Measures

    Feasibility

    Feasibility was assessed at 4-week follow-up.

    System Usability Scale

    The System Usability Scale (SUS) [55] is a 10-item scale assessing stakeholders’ views of ease of use. Items were modified to substitute “program” for “system.” Participants were asked to rate their agreement with each usability statement (eg, “I thought the program was easy to use”) on a scale of 1 (“strongly disagree”) to 5 (“strongly agree”). After reverse-scoring relevant items, sum scores were multiplied by 2.5 to create a final score ranging from 0 to 100. Research indicates that SUS scores above 68 are considered above average and scores below 68 are below average. Internal reliability for the SUS was excellent (α=.90).

    Program Engagement Data

    As in past work [56], feasibility was also evaluated via the amount of time participants spent engaging with the program. Engagement data consisted of usage and self-report data recorded by the program for 4 weeks. Self-report and usage data were collected by the mobile program and stored on a secured cloud server from Amazon Web Services [57]. Data were deidentified prior to extraction from the database. Engagement measures included the frequency with which participants completed mood logs, number of times the app was opened, number of articles read, number of messages sent to the coach, and number of activities completed. Data were also extracted to evaluate the number of days the user was active, which was defined as the number of days with at least one in-app action. In order to measure real-world engagement, participants were not given specific minimum engagement requirements to remain in the study.

    Acceptability

    Acceptability was assessed at 4-week follow-up. 

    Credibility and Expectancy Questionnaire

    The Credibility and Expectancy Questionnaire (CEQ) [58] is a 6-item scale that was originally designed to assess perceptions of treatment credibility and expectancy for improvement in psychotherapy. To render the scale more appropriate for use in this study, questionnaire items were modified slightly (ie, “program” was substituted for “therapy” and “stress and anxiety” was substituted for “symptoms”). Items in the CEQ range either from 1 to 9 or from 0 to 100, depending on the item. In line with the CEQ’s factor structure and following previous work [59], we computed average credibility and expectancy scores reflected by the first three and last three items of the scale, respectively. Internal reliability was excellent (credibility subscale: α=.90; expectancy subscale: α=.93).

    Program Satisfaction Questionnaire

    We asked the following open-ended questions: (1) What is the main benefit you received from Noom’s stress and anxiety management program? (2) How can we improve Noom’s stress and anxiety management program for you? (3) What was the most helpful part of the program? and (4) What was the least helpful part of the program? Because of the variety of answers possible, content analysis was used to code each response into categories and calculate the percentage of responses allocated to each category. The categories were created using latent Dirichlet allocation (LDA), a machine learning approach for automatic clustering of text data [60]. LDA is an unsupervised approach that automatically identifies latent clusters of words (ie, categories) that cluster within unclassified data. Each word cluster was assigned a label, or category name, by a master coder with experience with the program. For each question, each participant response was given a score (0 or 1) for each category since one response could apply to multiple categories. Interrater reliability between the master coder and another coder blind to the study’s hypotheses and design ranged from 0.72 to 1.0 for all categories, suggesting good to excellent reliability [61].

    Symptom and Well-being Outcomes

    Symptom and well-being outcomes were assessed at baseline and 4-week follow-up.

    7-Item Generalized Anxiety Disorder Scale

    The 7-item Generalized Anxiety Disorder scale (GAD-7) [62] is a 7-item scale that assesses the extent to which individuals experience symptoms of anxiety (eg, “Feeling nervous, anxious, or on edge”) on a scale of 0 (“not at all”) to 3 (“nearly every day”). Internal reliability for the GAD-7 was good (α=.82 and α=.87 for baseline and follow-up, respectively).

    4-Item Perceived Stress Scale

    The 4-item Perceived Stress Scale (PSS-4) [63] is a 4-item scale assessing the frequency with which individuals experience various symptoms of stress (eg, “How often have you felt that you were unable to control the important things in your life?”) on a scale of 0 (“never”) to 4 (“very often”). Internal reliability for the PSS-4 was adequate (α=.68 and α=.69 for baseline and follow-up, respectively).

    8-Item Patient Health Questionnaire Depression Scale

    The 8-item Patient Health Questionnaire depression scale (PHQ-8) [64] is an 8-item scale that assesses the extent to which participants experience feelings of depression (eg, “feeling down, depressed, or hopeless” or “little interest or pleasure in doing things”) on a scale of 0 (“not at all”) to 3 (“nearly every day”). Internal reliability for the PHQ-8 was good (α=.84 and α=.85 for baseline and follow-up, respectively).

    Difficulties in Emotion Regulation Scale–Short Form

    The Difficulties in Emotion Regulation Scale–Short Form (DERS-SF) [65,66] is an 18-item scale assessing emotion dysregulation. It comprises six subscales: emotional awareness, clarity about the nature of one’s emotions, acceptance of one’s emotions, access to effective emotion regulation strategies, ability to engage in goal-directed activities while experiencing negative emotions, and ability to manage one’s impulses during negative emotions. These subscales (α=.74-.91 and α=.76-.91) and the DERS-SF total score (α=.89 at both time points) demonstrated good internal consistency at baseline and follow-up, respectively.

    Life Orientation Test–Revised

    The Life Orientation Test–Revised (LOT-R) [67] is a 10-item scale that assesses trait optimism. Individuals are asked to rate their agreement with each statement (eg, “In uncertain times, I usually expect the best.”) on a scale of 0 (“strongly disagree”) to 4 (“strongly agree”). Internal reliability for the LOT-R was good (α=.86 and α=.85 at baseline and follow-up, respectively).

    Statistical Analysis

    Analyses were conducted in SPSS software (version 27; IBM Corp). For acceptability and feasibility, survey responses were descriptively analyzed with mean scores and percentages of participants that chose each response. For open-ended acceptability responses, content-analyzed categories are presented descriptively with the percentage of responses that fall into each category. Descriptive statistics were also conducted for engagement measures to evaluate feasibility. For preliminary outcomes, paired 2-tailed t tests were conducted to evaluate changes on all quantitative variables from baseline to week 4. Both per-protocol and intention-to-treat analyses were conducted. The per-protocol sample consisted of participants who completed both assessments (n=113) and included those who started the program but stopped using it. Intention-to-treat analyses included data from all participants who began the study (N=185); baseline scores were carried forward for participants who did not complete the week-4 assessment. Effect sizes were calculated using Cohen d [68].


    Participant Characteristics

    Participants’ demographic characteristics are presented in Table 1. A total of 185 unique Noom Mood users enrolled in the study and completed the baseline survey. Of these, 113 (62.1%) participants completed the follow-up survey. Participants who completed both baseline and follow-up surveys did not differ significantly from those who completed only the baseline survey in terms of any demographic variables or baseline survey values.

    Table 1. Participant characteristics.
    DemographicsPer-protocol sample (n=113)Intention-to-treat sample (N=185)
    Age (years), mean (SD)36.8 (9.8)37.3 (10.4)
    Gender, n (%)


    		Male15 (13.3)32 (17.3)

    		Female94 (83.2)141 (76.2)

    		Other2 (1.8)3 (1.6)

    		Prefer not to say or N/Aa2 (1.8)9 (4.9)
    Ethnicity, n (%)


    		Hispanic13 (11.5)20 (10.8)

    		Not Hispanic97 (85.8)153 (82.7)

    		Prefer not to say or N/A3 (2.7)12 (6.5)
    Race, n (%)


    		White99 (87.6)153 (82.7)

    		Black or African American5 (4.4)7 (3.8)

    		Asian or Pacific Islander3 (2.7)11 (5.9)

    		Other0 (0)1 (0.5)

    		Prefer not to say or N/A6 (5.3)13 (7.0)
    Employment status, n (%)


    		Employed 88 (77.8)144 (77.8)

    		Not employed12 (10.6)18 (9.7)

    		Retired1 (0.9)2 (1.1)

    		Disabled5 (4.4)6 (3.2)

    		Student5 (4.4)6 (3.2)

    		Prefer not to say or N/A2 (1.8)9 (4.9)
    Education, n (%)


    		High school, GEDb, or less education7 (6.2)10 (5.4)

    		Some college or associate degree24 (21.2)37 (20.0)

    		College graduate45 (39.8)67 (36.2)

    		Graduate degree 35 (31.0)62 (33.5)

    		Prefer not to say or N/A2 (1.8)9 (4.9)

    aN/A: not applicable.

    bGED: General Education Development.

    Feasibility

    Responses to the SUS are presented in Table 2. As noted above, scores of 68 or higher on the SUS indicate above-average ratings of system usability. A majority (79/109, 72.5%) of participants had overall system usability scores of 68 or higher (mean 77.40, SD 19.45), which is considered an indication of good usability [59]. Most participants reported that the program was easy to use (85/110, 77.3%), and they thought that other people would be able to learn to use the program very quickly (93/109, 85.3%).

    Program engagement data are presented in Table 3. Engagement data are presented as weekly averages (ie, the number of times the participant engaged in the behavior over the course of the study divided by the total number of weeks). Participants engaged within the app several times per week on average. Over 4 weeks, the per-protocol sample averaged 14.1 (SD 9.02) app opens, with 2 mean app opens per week. They had an average of 12.1 days with an in-app action, amounting to 1.7 active days per week. The intention-to-treat sample opened the app, on average, 13.7 (SD 8.6) times over 4 weeks, with an average of 1.96 app opens per week. They completed at least one in-app action on an average of 11.2 (SD 8.7) days, which amounted to 1.6 active days per week.

    Table 2. Participants reporting good feasibility and acceptability.
    Survey measureaValue
    System Usability Scale item, n (%)

    		I would like to use this program frequently. (n=109)62 (56.9)

    		I found the program unnecessarily complex.b (n=110)81 (73.6)

    		I thought the program was easy to use. (n=110)85 (77.3)

    		I would need the support of a technical person to be able to use this program.b (n=108)95 (88.0)

    		I found the various functions in this program were well integrated. (n=109)76 (69.7)

    		I thought there was too much inconsistency in the program.b (n=109)87 (79.8)

    		I would imagine that most people would learn to use this program very quickly. (n=109)93 (85.3)

    		I found the program very cumbersome to use.b (n=109)77 (70.6)

    		I felt very confident using the program. (n=108)78 (72.2)

    		I needed to learn a lot of things before I could get going with the program.b (n=107)89 (83.2)
    System Usability Scale score of 68 or higher (n=109), n (%)79 (72.5)
    System Usability Scale overall score, mean (SD)77.4 (19.4)
    Credibility and Expectancy Questionnaire item, n (%)

    		At this point, how logical does the program seem to you? (n=110)101 (91.8)

    		How successful do you think this program was in reducing stress and anxiety? (n=109)83 (76.1)

    		How confident would you be in recommending this program to a friend who experiences stress and anxiety? (n=108)87 (80.6)

    		By the end of the program, how much improvement in stress and anxiety do you think will occur?c (n=108)63 (58.3)

    		At this point, how much do you really feel that the program will help to reduce stress and anxiety? (n=108)85 (78.7)

    		By the end of this program, how much improvement in stress and anxiety do you feel will occur?c (n=107)62 (57.9)

    aThe table includes participants who chose 4 or greater (out of 5) on the System Usability Scale or 5 or greater (out of 9) on the Credibility and Expectancy Questionnaire, except where indicated.

    bThese participants chose 2 or less (out of 5) on the System Usability Scale.

    cThese participants chose at least 50% out of 100%.

    Table 3. Average total engagement over 4 weeks.
    Type of engagementPer-protocol sample (n=110)a, mean (SD)Intention-to-treat sample (n=181)a, mean (SD)
    App opens14.11 (9.02)13.72 (8.60)
    Articles read37.12 (28.87)34.39 (27.44)
    Mood logs10.59 (9.54)10.00 (10.79)
    Days with one in-app action12.14 (9.03)11.24 (8.68)
    Messages sent to coach9.71 (10.33)8.24 (9.09)
    Activitiesb1.33 (2.68) 1.30 (2.54)

    aSample sizes represent all participants for whom matching data from the database could be identified.

    bActivities were calculated over 3 weeks because one offline activity was not tracked by the program.

    Acceptability

    Responses to the CEQ are presented in Table 2. Of note, the table displays the frequency and percentage of participants who chose at least a 5 (“somewhat”) out of 9 (“very much”) on the CEQ. The vast majority of participants (101/110, 91.8%) rated the program as at least somewhat logical (mean 7.1, SD 1.9, range 1-9). Most (83/109, 76.1%) thought the program was at least somewhat successful at reducing stress and anxiety (mean 5.6, SD 2.2, range 1-9). Many participants (87/108, 80.6%) also felt at least somewhat confident in recommending the program to a friend (mean 6.1, SD 2.3, range 1-9). Most participants (85/108, 78.7%) felt the program would help to reduce stress and anxiety at least somewhat (mean 5.7, SD 2.3), with more than half (63/108, 58.3%) expecting it to reduce their stress or anxiety by 50% or more (mean 4.9, SD 2.5, with 0 referring to 0% and 10 referring to 100%).

    Responses to the Program Satisfaction Questionnaire are presented in Table 4. Participants reported benefiting most from the skills and techniques they learned or practiced (eg, breathing techniques and thought reframing; 38/106, 35.8%). Participants also reported benefiting from the program’s features or capabilities (eg, mood tracking and articles; 31/106, 29.2%) and greater awareness (eg, learning and reflection) encouraged by the program (30/106, 28.3%). Specifically, participants found the articles (18/106, 17.0%), coaching (18/106, 17.0%), and qualities of the program (eg, manageable, convenient, and “great” attitude; 16/106, 15.1%) to be the most helpful parts of Noom Mood.

    For potential areas of improvement, most participants did not provide a response or indicated that they had no suggested improvements (37/106, 34.9%). The next most common response was “other” (21/106, 19.8%), or participants requested a new feature or program idea (19/106, 17.9%). “Other” responses included increasing the frequency of reminders, expanding areas of content (eg, support for procrastination), and slowing the pace of tasks. Participants also preferred a lower cost (16/106, 15.1%), with some mentioning the potential to be reimbursed, as well as a more personalized experience (9/106, 8.5%) and greater flexibility (9/106, 8.5%), such as the ability to progress while skipping articles, accessing future articles, or repeating an activity for another week.

    When asked to describe the least helpful parts of Noom Mood, most participants did not provide a response (40/106, 37.7%). The next most common response was “other” (21/106, 19.8%); responses noted that the program contained too much repetition and that the pacing of the program needed improvement. Lastly, some participants (17/106, 16.0%) described coaching as the least helpful aspect of the program, noting that they would prefer to interact with a coach with specialized expertise or to receive more personalized responses.

    Table 4. Response frequencies in each category for the Program Satisfaction Questionnaire.
    CategoryParticipant responses (n=106), n (%)a
    Main benefit of Noom Mood

    		Skills and techniques38 (35.8)

    		Program features or capabilities31 (29.2)

    		Awareness (ie, learning and reflection)30 (28.3)

    		Emotional experience and management27 (25.5)

    		Other18 (17.0)

    		None or no response15 (14.2)
    Areas to improve

    		None or no response37 (34.9)

    		Other21 (19.8)

    		New feature or program idea19 (17.9)

    		Cost16 (15.1)

    		Coaching15 (14.2)

    		Personalization9 (8.5)

    		Flexibility9 (8.5)

    		Articles6 (5.7)

    		Activities3 (2.8)
    Most helpful part of Noom Mood

    		Coaching18 (17.0)

    		Articles18 (17.0)

    		None or no response17 (16.0)

    		Qualities of the program16 (15.1)

    		Skills and techniques15 (14.2)

    		Activities14 (13.2)

    		Awareness (ie, learning and reflection)9 (8.5)

    		Other9 (8.5)

    		Mood tracking5 (4.7)

    		Everything3 (2.8)
    Least helpful part of Noom Mood

    		None or no response40 (37.7)

    		Other21 (19.8)

    		Coaching17 (16.0)

    		Activities9 (8.5)

    		Mood tracking7 (6.6)

    		Articles6 (5.7)

    		Personalization and interactivity5 (4.7)

    		Cost4 (3.8)

    		Everything2 (1.9)

    aEach response could be placed in more than one category. Categories were derived from individuals’ open-ended responses.

    Symptom and Well-being Outcomes

    From baseline to 4 weeks, there was a significant reduction in anxiety symptoms for both per-protocol samples (Table 5; t112=10.92, P<.001, d=1.03) and intention-to-treat samples (t184=9.48, P<.001, d=0.70) with large and medium effect sizes, respectively. There was also a significant improvement in perceived stress (per-protocol sample: t112=7.69, P<.001, d=0.72; intention-to-treat sample: t184=7.09, P<.001, d=0.52) and depressive feelings (per-protocol sample: t110=7.88, P<.001, d=0.75; intention-to-treat sample: t181=7.40, P<.001, d=0.55) with medium effect sizes. Finally, there were significant improvements in emotion regulation (per-protocol sample: t105=5.93, P<.001, d=0.58; intention-to-treat sample: t178=5.79, P<.001, d=0.43) and optimism (per-protocol sample: t104=–5.04, P<.001, d=–0.49; intention-to-treat sample: t175=–5.15, P<.001, d=–0.39) with small to medium effect sizes. 

    Table 5. Symptom and well-being outcomes from baseline to 4 weeks.
    OutcomePer-protocol sample (n=113)aIntention-to-treat sample (N=185)b

    		Baseline,  mean (SD)4 weeks, mean (SD)ΔMean (% change)cP valueEffect sizedBaseline, mean (SD)4 weeks, mean (SD)ΔMean (% change)cP valueEffect sized
    Anxiety symptoms (GAD-7e)13.30 (4.31)8.54 (4.61)–4.76 (–35.81)<.0011.0313.28 (4.39)10.18 (5.14)–3.10 (–23.32)<.0010.70
    Perceived stress (PSS-4f)8.96 (2.39)7.08 (2.29)–1.88 (–21.03)<.0010.728.89 (2.41)7.73 (2.48)–1.16 (–13.07)<.0010.52
    Depressive feelings (PHQ-8g)11.67 (5.47)7.77 (4.98)–3.90 (–33.39)<.0010.7511.99 (5.55)9.40 (5.66)–2.59 (–21.61)<.0010.55
    Emotion regulation (DERS-SFh)45.97 (11.86)39.39 (11.30)–6.57 (–14.30) <.0010.5847.01 (13.09)42.95 (13.49)–4.05 (–8.63) <.0010.43
    Optimism (LOT-Ri)7.05 (3.49)8.16 (3.15)1.11 (15.75)<.0010.497.33 (3.57)8.09 (3.39)0.75 (10.30) <.0010.39

    aPer-protocol analyses only included participants who completed both survey assessments.

    bFor intention-to-treat analyses, baseline responses were carried forward for nonresponders.

    cNegative values indicate decreases compared to baseline.

    dEffect sizes constitute Cohen d.

    eGAD-7: 7-item Generalized Anxiety Disorder scale.

    fPSS-4: 4-item Perceived Stress Scale.

    gPHQ-8: 8-item Patient Health Questionnaire depression scale.

    hDERS-SF: Difficulties in Emotion Regulation Scale–Short Form; negative values on the DERS-SF indicate better emotional regulation (ie, fewer difficulties with emotional regulation).

    iLOT-R: Life Orientation Test–Revised; positive values on the LOT-R indicate more optimism.


    Principal Findings

    In reviews of mental health programs, researchers have voiced concerns about limited published research on commercial programs, and that programs either have limited public engagement or are not based on evidence-based theory [18,25-27,31,32]. Given the identified need for evidence from this type of commercial program [25,31], this pilot study evaluated the feasibility, acceptability, and preliminary outcomes of Noom Mood, which is widely publicly available, based on CBT and MBSR techniques, designed to encourage engagement among the general public, and includes personal coaching. Our results suggest that the program was usable, feasible, and acceptable to participants. In addition, self-reported anxiety symptoms, stress, depressive feelings, emotion regulation, and optimism improved from baseline to 4 weeks.

    Feasibility and Acceptability

    Feasibility

    Overall, participants rated the program as feasible. The average system usability score was 77.4, which surpasses the threshold for good usability [69], and more than 75% of participants reported that the program was easy to use. These scores are in line with feasibility and usability scores from other mobile programs [70-73]. Similar to levels of engagement reported in studies of comparable mobile mental health programs [46,70,74], participants in this study engaged with Noom Mood regularly, opening the program approximately two times per week and performing an action within the app once every 2 to 3 days (11 of 28 days). Participants engaged most with the articles and least with activities. Of note, it is possible that participants completed activities offline throughout the week, which is how they were designed, but did not mark them as complete in the app. As such, it is likely that the data collected on activities underestimate participant engagement in this aspect of Noom Mood, given that many activities focus on offline experiences (eg, practicing breathing exercises or grounding techniques). Future studies will aim to assess actions completed offline in relationship to symptom outcomes.

    Acceptability

    The vast majority of participants found the program to be logical (92%) and effective at reducing stress and anxiety (76%). Importantly, 81% of participants felt confident in recommending the program to a friend. These findings are similar to other studies of mobile mental health programs and suggest that the program was perceived to be acceptable to users [35,70,73]. Additionally, at the follow-up assessment, more than half of the participants reported that they expected that the program would eventually reduce their stress or anxiety by an additional 50% or more. Future work should investigate long-term outcomes and whether these participant expectations are borne out.

    Participants reported benefiting most from skills training; program features such as articles, activities, and coaching; learning to better manage their emotions; and reflective processes such as learning, reflecting, and increasing their awareness. Participants reported benefiting from taking the time to reflect on how they were feeling and increasingly becoming aware of their emotions and thought patterns. Many participants also mentioned benefiting from the structure and accountability of a designated program. Participants appreciated the overall tenor of the program; one participant reflected that “the attitude it strikes is a great balance of cheeky humor but realistic so it’s not overly strict nor overly cheesy. Makes me connect with it well and stick with it.” Other participants, however, reported that they hoped for a more serious tone to the articles. At the time of the study, the program incorporated jokes and hashtags for the sake of relatability, and has since been modified in response to participant feedback. 

    Participants also indicated that the program could be improved to better help individuals progress in a way that best suits an individual’s idiosyncratic wants or needs. For example, some participants wanted a slower pace, whereas others requested more daily reminders. Additionally, some participants provided feedback that they wanted more specialized interactions with coaches. While individuals were informed that Noom Mood is not a replacement for therapy and does not provide clinical assessment or treatment, it is possible that participants were expecting the coaching feature to function more similarly to therapy. However, some participants provided feedback stating that responses given by coaches did not feel personalized and felt too generic. It is also possible that some participants may not have been good candidates for a self-help approach. As mentioned previously, in the literature, there is limited understanding of how participants would experience a commercial mobile mental health program with personal coaching, rather than therapy. This study contributes initial understanding that, in this context, coaching can be helpful, but it can also raise confusion about the role of a coach when providing guidance and support rather than therapy. Future iterations of the program should, thus, be sure to set expectations for this feature clearly. 

    Participants also relayed some suggestions for program improvements that would provide support in varying environments or situations, such as support for moms with young children, skills to reduce procrastination, video and audio recordings, and easily accessible summaries of activities or articles, all of which should be considered in future programs. Since the time of the study, audio recordings have been added to the program. Some participants reported that they would prefer that the program be offered at a lower cost, and some mentioned they would like the program to be covered by health insurance plans. In order to increase accessibility, future initiatives and programs should consider efforts to provide reimbursable experiences (eg, through employee wellness initiatives).

    Preliminary Outcomes

    Anxiety Symptoms, Perceived Stress, and Depressive Feelings

    From baseline to 4 weeks, anxiety symptoms improved by 36% (d=1.03) in per-protocol analyses and 23% (d=0.70) in intention-to-treat analyses. In addition, stress reductions were 21% (d=0.72, per-protocol analysis) and 13% (d=0.52, intention-to-treat analysis), and depressive feelings decreased by 33% (d=0.75, per-protocol analysis) and 22% (d=0.55, intention-to-treat analysis). These effect sizes are comparable to those reported in studies of other mobile mental health programs with the same study length and outcome measures [44,75-80]. Specifically, anxiety and stress decreased in ways that were comparable to or greater than anxiety reductions shown in previous studies, whereas depression showed comparable, though smaller, effect sizes [44,75,78,79]. Of course, this may reflect the fact that the program focuses more on stress and anxiety management than on depression. Of all our outcome measures, anxiety showed the biggest effect sizes, which contrasts with some studies that have found that anxiety scores did not improve as much as other symptom measures, such as depression [75,80].

    Emotion Regulation and Optimism 

    In this study, we found that emotion regulation improved by 14% (d=0.58, per-protocol analysis) and 8.6% (d=0.43, intention-to-treat analysis). Emotion dysregulation is hypothesized to underpin a wide range of psychological difficulties [81]; in fact, transdiagnostic interventions, such as DBT or the Unified Protocol [82], focus on emotion dysregulation as the primary treatment target. Notably, however, emotion regulation is rarely included as an outcome variable in mobile mental health programs, despite its empirical and theoretical relevance to mental health and well-being [19]. In two studies of mHealth programs conducted with young adults [83] and homeless youth [84] that measured emotion regulation as an outcome variable, results showed no significant improvements in emotion regulation capacity. 

    We found significantly higher optimism at 4 weeks compared to baseline (15.7% or d=0.49, per-protocol analysis; 10% or d=0.39, intention-to-treat analysis). To our knowledge, this is the first mobile mental health study to measure changes in optimism, though some studies of mobile mental health programs have found improvements in other positive psychological constructs, such as life satisfaction, general mental well-being, or quality of life [46,85-88]. A robust literature base demonstrates that optimism is inversely correlated with depression and anxiety and positively correlated with measures of life satisfaction and self-reported health variables [89,90]. Importantly, optimism may influence physical and mental health by encouraging adaptive coping [85]. Consistent with previous findings, both baseline and 4-week optimism scores were significantly negatively correlated with time-matched anxiety symptoms, stress, and depressive feelings, and optimism scores were positively correlated with emotion regulation (ie, higher optimism is correlated with greater capacity to regulate one’s emotions). Future studies should evaluate optimism and its associations with other mental health outcomes.

    Limitations

    This pilot study had several limitations. First, without a control group, it was not possible to separate the effects of the program itself from improvement over time (ie, regression to the mean and maturation). In addition, other interventions were uncontrolled; that is, program participants may have been participating in active therapy or may have been taking psychotropic medications while they were participating in this study. Nevertheless, it is unlikely that these findings are purely spurious, as the effect sizes are similar to those found in active treatment groups in RCTs, and they are much larger than those found in control groups (eg, see Bakker et al [75]). Now that preliminary feasibility and acceptability have been established, future studies should use randomized designs to confirm that these results were due to the program itself. Also, the study was conducted over 4 weeks, and it is unclear whether results would change over longer periods of time. Further, the study examined the program as a whole, making it difficult to isolate which specific program components led to changes in outcomes. Future studies should use causal methods to explore this further. In addition, the sample was primarily female, White, and highly educated, which is typical of studies of mobile mental health programs [19]. Future research should evaluate to what extent these results would generalize to other populations and actively recruit from hard-to-reach populations. Lastly, this study did not assess other variables that may have caused improvement in symptoms, such as psychiatric services, individual or group therapy, and participants’ use of other self-help materials.

    Conclusions

    In this study, we explored the usability, feasibility, acceptability, and preliminary effectiveness of Noom Mood, a publicly available, mobile mental well-being program based on CBT and MBSR with personal coaching. The program follows 11 of Bakker et al’s [5] evidence-based recommendations for mobile mental health programs: it is based on CBT; addresses both anxiety and low mood; is designed for use by nonclinical populations; includes reporting of thoughts, feelings, and behaviors; recommends activities; provides mental health information; encourages non–technology-based activities; includes gamification or intrinsic motivation to engage; shows logs of past app use (eg, patterns of logged mood); uses reminders to engage (eg, messages from the coach); and provides a simple and intuitive interface and interactions. Our results suggest that Noom Mood was usable, feasible, and acceptable to participants, with promising preliminary improvements in anxiety symptoms, stress, depressive feelings, emotion regulation, and optimism. Future directions should include (1) the incorporation of changes suggested by participants in this study and (2) more rigorous testing of outcome variables, such as through randomized designs.

    Conflicts of Interest

    Authors MM, ASH, ESM, CNM, HB, and AM are employees at Noom, Inc, and have received salary and stock options for their employment. LR received payment from Noom, Inc, for their role as a consultant on this project and for their contribution as an author on this paper. LR is also a co-owner of the Triangle Area Psychology Clinic; a consultant to, and a DBT trainer for, Behavioral Tech, LLC; and an employee of University of North Carolina School of Medicine. There are no specific conflicts to report with those entities, and none of those entities were involved in their contribution to this project.

    1. Mental health: Strengthening our response. World Health Organization. 2018 Mar 30.   URL: https://www.who.int/news-room/fact-sheets/detail/mental-health-strengthening-our-response [accessed 2021-12-01]
    2. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005 Jun;62(6):593-602. [CrossRef] [Medline]
    3. Remes O, Brayne C, van der Linde R, Lafortune L. A systematic review of reviews on the prevalence of anxiety disorders in adult populations. Brain Behav 2016 Jul;6(7):e00497 [FREE Full text] [CrossRef] [Medline]
    4. Demyttenaere K, Bruffaerts R, Posada-Villa J, Gasquet I, Kovess V, Lepine JP, WHO World Mental Health Survey Consortium. Prevalence, severity, and unmet need for treatment of mental disorders in the World Health Organization World Mental Health Surveys. JAMA 2004 Jun 02;291(21):2581-2590. [CrossRef] [Medline]
    5. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: Review and evidence-based recommendations for future developments. JMIR Ment Health 2016 Mar;3(1):e7 [FREE Full text] [CrossRef] [Medline]
    6. Arango C, Díaz-Caneja CM, McGorry PD, Rapoport J, Sommer IE, Vorstman JA, et al. Preventive strategies for mental health. Lancet Psychiatry 2018 Jul;5(7):591-604. [CrossRef] [Medline]
    7. Canady VA. APA stress report amid COVID-19 points to parental challenges. Mental Health Weekly. 2020 May 29.   URL: https://onlinelibrary.wiley.com/doi/10.1002/mhw.32385 [accessed 2022-04-09]
    8. Witters D, Harter J. Worry and stress fuel record drop in US life satisfaction. Gallup. 2020 May 08.   URL: https://news.gallup.com/poll/310250/worry-stress-fuel-record-drop-life-satisfaction.aspx [accessed 2021-12-01]
    9. Walker ER, McGee RE, Druss BG. Mortality in mental disorders and global disease burden implications: A systematic review and meta-analysis. JAMA Psychiatry 2015 Apr;72(4):334-341 [FREE Full text] [CrossRef] [Medline]
    10. Whiteford A, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: Findings from the Global Burden of Disease Study 2010. Lancet 2013 Nov 09;382(9904):1575-1586. [CrossRef] [Medline]
    11. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, et al. Barriers to mental health treatment: Results from the WHO World Mental Health surveys. Psychol Med 2014 Apr;44(6):1303-1317 [FREE Full text] [CrossRef] [Medline]
    12. Weil T. Insufficient dollars and qualified personnel to meet United States mental health needs. J Nerv Ment Dis 2015 Apr;203(4):233-240. [CrossRef] [Medline]
    13. Leong FTL, Kalibatseva Z. Cross-cultural barriers to mental health services in the United States. Cerebrum 2011 Mar;2011:5 [FREE Full text] [Medline]
    14. Handley TE, Kay-Lambkin FJ, Inder KJ, Lewin TJ, Attia JR, Fuller J, et al. Self-reported contacts for mental health problems by rural residents: Predicted service needs, facilitators and barriers. BMC Psychiatry 2014 Sep 06;14:249 [FREE Full text] [CrossRef] [Medline]
    15. Comer JS, Barlow DH. The occasional case against broad dissemination and implementation: Retaining a role for specialty care in the delivery of psychological treatments. Am Psychol 2014 Jan;69(1):1-18 [FREE Full text] [CrossRef] [Medline]
    16. Taylor CB, Fitzsimmons-Craft EE, Graham AK. Digital technology can revolutionize mental health services delivery: The COVID-19 crisis as a catalyst for change. Int J Eat Disord 2020 Jul;53(7):1155-1157 [FREE Full text] [CrossRef] [Medline]
    17. Atherly A, Van Den Broek-Altenburg E, Hart V, Gleason K, Carney J. Consumer reported care deferrals due to the COVID-19 pandemic, and the role and potential of telemedicine: Cross-sectional analysis. JMIR Public Health Surveill 2020 Sep 14;6(3):e21607 [FREE Full text] [CrossRef] [Medline]
    18. Huckvale K, Nicholas J, Torous J, Larsen ME. Smartphone apps for the treatment of mental health conditions: Status and considerations. Curr Opin Psychol 2020 Dec;36:65-70 [FREE Full text] [CrossRef] [Medline]
    19. Eisenstadt M, Liverpool S, Infanti E, Ciuvat RM, Carlsson C. Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: Systematic review and meta-analysis. JMIR Ment Health 2021 Nov 08;8(11):e31170 [FREE Full text] [CrossRef] [Medline]
    20. Hwang WJ, Ha JS, Kim MJ. Research trends on mobile mental health application for general population: A scoping review. Int J Environ Res Public Health 2021 Mar 02;18(5):2459 [FREE Full text] [CrossRef] [Medline]
    21. Hilty DM, Chan S, Hwang T, Wong A, Bauer AM. Advances in mobile mental health: Opportunities and implications for the spectrum of e-mental health services. Mhealth 2017 Aug;3:34 [FREE Full text] [CrossRef] [Medline]
    22. Price M, Yuen EK, Goetter EM, Herbert JD, Forman EM, Acierno R, et al. mHealth: A mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother 2014 Aug;21(5):427-436 [FREE Full text] [CrossRef] [Medline]
    23. Rathbone AL, Prescott J. The use of mobile apps and SMS messaging as physical and mental health interventions: Systematic review. J Med Internet Res 2017 Aug 24;19(8):e295 [FREE Full text] [CrossRef] [Medline]
    24. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
    25. Marshall JM, Dunstan DA, Bartik W. Apps with maps-Anxiety and depression mobile apps with evidence-based frameworks: Systematic search of major app stores. JMIR Ment Health 2020 Jun 24;7(6):e16525 [FREE Full text] [CrossRef] [Medline]
    26. Wasil AR, Weisz JR, DeRubeis RJ. Three questions to consider before developing a mental health app. World Psychiatry 2020 Jun;19(2):252-253 [FREE Full text] [CrossRef] [Medline]
    27. Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: Systematic search and panel-based usage analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [FREE Full text] [CrossRef] [Medline]
    28. Bakker D, Rickard N. Engagement in mobile phone app for self-monitoring of emotional wellbeing predicts changes in mental health: MoodPrism. J Affect Disord 2018 Dec;227:432-442. [CrossRef] [Medline]
    29. Aizenstros A, Bakker D, Hofmann SG, Curtiss J, Kazantzis N. Engagement with smartphone-delivered behavioural activation interventions: A study of the MoodMission smartphone application. Behav Cogn Psychother 2021 Sep;49(5):569-581. [CrossRef] [Medline]
    30. Bakker D, Rickard N. Engagement with a cognitive behavioural therapy mobile phone app predicts changes in mental health and wellbeing: MoodMission. Aust Psychol 2020 Nov 12;54(4):245-260. [CrossRef]
    31. Marshall JM, Dunstan DA, Bartik W. Clinical or gimmickal: The use and effectiveness of mobile mental health apps for treating anxiety and depression. Aust N Z J Psychiatry 2020 Jan;54(1):20-28. [CrossRef] [Medline]
    32. Lee RA, Jung ME. Evaluation of an mHealth app (DeStressify) on university students' mental health: Pilot trial. JMIR Ment Health 2018 Jan 23;5(1):e2 [FREE Full text] [CrossRef] [Medline]
    33. Lecomte T, Potvin S, Corbière M, Guay S, Samson C, Cloutier B, et al. Mobile apps for mental health issues: Meta-review of meta-analyses. JMIR Mhealth Uhealth 2020 May 29;8(5):e17458 [FREE Full text] [CrossRef] [Medline]
    34. Lau N, O'Daffer A, Colt S, Yi-Frazier JP, Palermo TM, McCauley E, et al. Android and iPhone mobile apps for psychosocial wellness and stress management: Systematic search in app stores and literature review. JMIR Mhealth Uhealth 2020 May 22;8(5):e17798 [FREE Full text] [CrossRef] [Medline]
    35. Huberty J, Green J, Glissmann C, Larkey L, Puzia M, Lee C. Efficacy of the mindfulness meditation mobile app "Calm" to reduce stress among college students: Randomized controlled trial. JMIR Mhealth Uhealth 2019 Jun 25;7(6):e14273 [FREE Full text] [CrossRef] [Medline]
    36. Newton A, Bagnell A, Rosychuk R, Duguay J, Wozney L, Huguet A, et al. A mobile phone-based app for use during cognitive behavioral therapy for adolescents with anxiety (MindClimb): User-centered design and usability study. JMIR Mhealth Uhealth 2020 Dec 08;8(12):e18439 [FREE Full text] [CrossRef] [Medline]
    37. Anton MT, Greenberger HM, Andreopoulos E, Pande RL. Evaluation of a commercial mobile health app for depression and anxiety (AbleTo Digital+): Retrospective cohort study. JMIR Form Res 2021 Sep 21;5(9):e27570 [FREE Full text] [CrossRef] [Medline]
    38. Solness CL, Kroska EB, Holdefer PJ, O'Hara MW. Treating postpartum depression in rural veterans using internet delivered CBT: Program evaluation of MomMoodBooster. J Behav Med 2021 Aug;44(4):454-466 [FREE Full text] [CrossRef] [Medline]
    39. Hofmann SG, Asnaani A, Vonk IJJ, Sawyer AT, Fang A. The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognit Ther Res 2012 Oct 1;36(5):427-440 [FREE Full text] [CrossRef] [Medline]
    40. Grist R, Cavanagh K. Computerised cognitive behavioural therapy for common mental health disorders, what works, for whom under what circumstances? A systematic review and meta-analysis. J Contemp Psychother 2013 Sep 4;43(4):243-251. [CrossRef]
    41. Thompson EM, Destree L, Albertella L, Fontenelle LF. Internet-based acceptance and commitment therapy: A transdiagnostic systematic review and meta-analysis for mental health outcomes. Behav Ther 2021 Mar;52(2):492-507. [CrossRef] [Medline]
    42. French K, Golijani-Moghaddam N, Schröder T. What is the evidence for the efficacy of self-help acceptance and commitment therapy? A systematic review and meta-analysis. J Contextual Behav Sci 2017 Oct;6(4):360-374. [CrossRef] [Medline]
    43. Valentine SE, Bankoff SM, Poulin RM, Reidler EB, Pantalone DW. The use of dialectical behavior therapy skills training as stand-alone treatment: A systematic review of the treatment outcome literature. J Clin Psychol 2015 Jan;71(1):1-20. [CrossRef] [Medline]
    44. Yang E, Schamber E, Meyer RML, Gold JI. Happier healers: Randomized controlled trial of mobile mindfulness for stress management. J Altern Complement Med 2018 May;24(5):505-513. [CrossRef] [Medline]
    45. Caldeira C, Chen Y, Chan L, Pham V, Chen Y, Zheng K. Mobile apps for mood tracking: An analysis of features and user reviews. AMIA Annu Symp Proc 2017;2017:495-504 [FREE Full text] [Medline]
    46. Ahtinen A, Mattila E, Välkkynen P, Kaipainen K, Vanhala T, Ermes M, et al. Mobile mental wellness training for stress management: Feasibility and design implications based on a one-month field study. JMIR Mhealth Uhealth 2013 Jul;1(2):e11 [FREE Full text] [CrossRef] [Medline]
    47. Levin ME, Haeger J, Pierce B, Cruz RA. Evaluating an adjunctive mobile app to enhance psychological flexibility in acceptance and commitment therapy. Behav Modif 2017 Nov;41(6):846-867. [CrossRef] [Medline]
    48. Kim H, Tietsort C, Posteher K, Michaelides A, Toro-Ramos T. Enabling self-management of a chronic condition through patient-centered coaching: A case of an mHealth diabetes prevention program for older adults. Health Commun 2020 Dec;35(14):1791-1799. [CrossRef] [Medline]
    49. Weisel KK, Fuhrmann LM, Berking M, Baumeister H, Cuijpers P, Ebert DD. Standalone smartphone apps for mental health-A systematic review and meta-analysis. NPJ Digit Med 2019;2:118 [FREE Full text] [CrossRef] [Medline]
    50. Mohr DC, Weingardt KR, Reddy M, Schueller SM. Three problems with current digital mental health research...and three things we can do about them. Psychiatr Serv 2017 May 01;68(5):427-429 [FREE Full text] [CrossRef] [Medline]
    51. Andersson G, Carlbring P, Berger T, Almlöv J, Cuijpers P. What makes internet therapy work? Cogn Behav Ther 2009 Jan;38 Suppl 1:55-60. [CrossRef] [Medline]
    52. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: A meta-analysis. Cogn Behav Ther 2009 Dec;38(4):196-205. [CrossRef] [Medline]
    53. Mohr DC, Duffecy J, Ho J, Kwasny M, Cai X, Burns MN, et al. A randomized controlled trial evaluating a manualized TeleCoaching protocol for improving adherence to a web-based intervention for the treatment of depression. PLoS One 2013 Aug;8(8):e70086 [FREE Full text] [CrossRef] [Medline]
    54. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017 Dec 29;12(1):108 [FREE Full text] [CrossRef] [Medline]
    55. Brooke J. SUS: A 'quick and dirty' usability scale. In: Thomas B, Weerdmeester BA, McClleland IL, Jordan PW, editors. Usability Evaluation in Industry. London, UK: Taylor & Francis Ltd; 1996:189-194.
    56. Schlosser DA, Campellone TR, Truong B, Anguera JA, Vergani S, Vinogradov S, et al. The feasibility, acceptability, and outcomes of PRIME-D: A novel mobile intervention treatment for depression. Depress Anxiety 2017 Jun;34(6):546-554 [FREE Full text] [CrossRef] [Medline]
    57. Torous J, Kiang MV, Lorme J, Onnela J. New tools for new research in psychiatry: A scalable and customizable platform to empower data driven smartphone research. JMIR Ment Health 2016 May 05;3(2):e16 [FREE Full text] [CrossRef] [Medline]
    58. Devilly GJ, Borkovec TD. Psychometric properties of the credibility/expectancy questionnaire. J Behav Ther Exp Psychiatry 2000 Jun;31(2):73-86. [CrossRef]
    59. Thompson-Hollands J, Bentley KH, Gallagher MW, Boswell JF, Barlow DH. Credibility and outcome expectancy in the unified protocol: Relationship to outcomes. J Exp Psychopathol 2014 Mar 30;5(1):72-82. [CrossRef]
    60. Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res 2003;3:993-1022 [FREE Full text]
    61. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977 Mar;33(1):159. [CrossRef]
    62. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    63. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav 1983:385-396. [CrossRef] [Medline]
    64. Kroenke K, Strine TW, Spitzer RL, Williams JBW, Berry JT, Mokdad AH. The PHQ-8 as a measure of current depression in the general population. J Affect Disord 2009 Apr;114(1-3):163-173. [CrossRef] [Medline]
    65. Gratz K, Roemer L. Multidimensional assessment of emotion regulation and dysregulation: Development, factor structure, and initial validation of the Difficulties in Emotion Regulation Scale. J Psychopathol Behav Assess 2004 Mar;26(1):41-54. [CrossRef]
    66. Victor S, Klonsky E. Validation of a brief version of the Difficulties in Emotion Regulation Scale (DERS-18) in five samples. J Psychopathol Behav Assess 2016 May 13;38(4):582-589. [CrossRef]
    67. Scheier MF, Carver CS, Bridges MW. Distinguishing optimism from neuroticism (and trait anxiety, self-mastery, and self-esteem): A reevaluation of the Life Orientation Test. J Pers Soc Psychol 1994 Dec;67(6):1063-1078. [CrossRef] [Medline]
    68. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd edition. New York, NY: Routledge; 1988.
    69. Brooke J. SUS: A retrospective. J Usability Stud 2013 Feb;8(2):29-40 [FREE Full text]
    70. Deady M, Johnston D, Milne D, Glozier N, Peters D, Calvo R, et al. Preliminary effectiveness of a smartphone app to reduce depressive symptoms in the workplace: Feasibility and acceptability study. JMIR Mhealth Uhealth 2018 Dec 04;6(12):e11661 [FREE Full text] [CrossRef] [Medline]
    71. Mata-Greve F, Johnson M, Pullmann MD, Friedman EC, Griffith Fillipo I, Comtois KA, et al. Mental health and the perceived usability of digital mental health tools among essential workers and people unemployed due to COVID-19: Cross-sectional survey study. JMIR Ment Health 2021 Aug 05;8(8):e28360 [FREE Full text] [CrossRef] [Medline]
    72. Gordon JS, Sbarra D, Armin J, Pace TWW, Gniady C, Barraza Y. Use of a guided imagery mobile app (See Me Serene) to reduce COVID-19-related stress: Pilot feasibility study. JMIR Form Res 2021 Oct 04;5(10):e32353 [FREE Full text] [CrossRef] [Medline]
    73. Rung AL, Oral E, Berghammer L, Peters ES. Feasibility and acceptability of a mobile mindfulness meditation intervention among women: Intervention study. JMIR Mhealth Uhealth 2020 Jun 02;8(6):e15943 [FREE Full text] [CrossRef] [Medline]
    74. Lattie E, Cohen KA, Winquist N, Mohr DC. Examining an app-based mental health self-care program, IntelliCare for college students: Single-arm pilot study. JMIR Ment Health 2020 Oct 10;7(10):e21075 [FREE Full text] [CrossRef] [Medline]
    75. Bakker D, Kazantzis N, Rickwood D, Rickard N. A randomized controlled trial of three smartphone apps for enhancing public mental health. Behav Res Ther 2018 Oct;109:75-83. [CrossRef] [Medline]
    76. Mohr DC, Tomasino KN, Lattie EG, Palac HL, Kwasny MJ, Weingardt K, et al. IntelliCare: An eclectic, skills-based app suite for the treatment of depression and anxiety. J Med Internet Res 2017 Jan 05;19(1):e10 [FREE Full text] [CrossRef] [Medline]
    77. Kawadler JM, Hemmings NR, Ponzo S, Morelli D, Bird G, Plans D. Effectiveness of a smartphone app (BioBase) for reducing anxiety and increasing mental well-being: Pilot feasibility and acceptability study. JMIR Form Res 2020 Nov 10;4(11):e18067 [FREE Full text] [CrossRef] [Medline]
    78. Oser M, Wallace ML, Solano F, Szigethy EM. Guided digital cognitive behavioral program for anxiety in primary care: Propensity-matched controlled trial. JMIR Ment Health 2019 Apr 04;6(4):e11981 [FREE Full text] [CrossRef] [Medline]
    79. Flett JAM, Hayne H, Riordan BC, Thompson LM, Conner TS. Mobile mindfulness meditation: A randomised controlled trial of the effect of two popular apps on mental health. Mindfulness 2018 Oct 31;10(5):863-876. [CrossRef]
    80. Hwang WJ, Jo HH. Evaluation of the effectiveness of mobile app-based stress-management program: A randomized controlled trial. Int J Environ Res Public Health 2019 Nov 03;16(21):4270 [FREE Full text] [CrossRef] [Medline]
    81. Gross JJ, Muñoz RF. Emotion regulation and mental health. Clin Psychol 1995;2(2):151-164. [CrossRef]
    82. Barlow DH, Harris BA, Eustis EH, Farchione TJ. The unified protocol for transdiagnostic treatment of emotional disorders. World Psychiatry 2020 Jun;19(2):245-246 [FREE Full text] [CrossRef] [Medline]
    83. Dubad M, Elahi F, Marwaha S. The clinical impacts of mobile mood-monitoring in young people with mental health problems: The MeMO Study. Front Psychiatry 2021;12:687270 [FREE Full text] [CrossRef] [Medline]
    84. Schueller SM, Glover AC, Rufa AK, Dowdle CL, Gross GD, Karnik NS, et al. A mobile phone-based intervention to improve mental health among homeless young adults: Pilot feasibility trial. JMIR Mhealth Uhealth 2019 Jul 02;7(7):e12347 [FREE Full text] [CrossRef] [Medline]
    85. Conversano C, Rotondo A, Lensi E, Della Vista O, Arpone F, Reda MA. Optimism and its impact on mental and physical well-being. Clin Pract Epidemiol Ment Health 2010 May 14;6:25-29 [FREE Full text] [CrossRef] [Medline]
    86. Mak WW, Tong AC, Yip SY, Lui WW, Chio FH, Chan AT, et al. Efficacy and moderation of mobile app-based programs for mindfulness-based training, self-compassion training, and cognitive behavioral psychoeducation on mental health: Randomized controlled noninferiority trial. JMIR Ment Health 2018 Oct 11;5(4):e60 [FREE Full text] [CrossRef] [Medline]
    87. Pham Q, Khatib Y, Stansfeld S, Fox S, Green T. Feasibility and efficacy of an mHealth game for managing anxiety: "Flowy" randomized controlled pilot trial and design evaluation. Games Health J 2016 Feb;5(1):50-67. [CrossRef] [Medline]
    88. Weber S, Lorenz C, Hemmings N. Improving stress and positive mental health at work via an app-based intervention: A large-scale multi-center randomized control trial. Front Psychol 2019;10:2745 [FREE Full text] [CrossRef] [Medline]
    89. Andersson G. The benefits of optimism: A meta-analytic review of the life orientation test. Pers Individ Dif 1996 Nov;21(5):719-725. [CrossRef]
    90. Forgeard M, Seligman M. Seeing the glass half full: A review of the causes and consequences of optimism. Prat Psychol 2012 Jun;18(2):107-120. [CrossRef]

    ‎
    ACT: acceptance and commitment therapy
    CBT: cognitive behavioral therapy
    CEQ: Credibility and Expectancy Questionnaire
    DBT: dialectical behavior therapy
    DERS-SF: Difficulties in Emotion Regulation Scale–Short Form
    GAD-7: 7-item Generalized Anxiety Disorder scale
    LDA: latent Dirichlet allocation
    LOT-R: Life Orientation Test–Revised
    MBSR: mindfulness-based stress reduction
    mHealth: mobile health
    PHQ-8: 8-item Patient Health Questionnaire depression scale
    PSS-4: 4-item Perceived Stress Scale
    RCT: randomized controlled trial
    SUS: System Usability Scale

    Edited by A Mavragani; submitted 26.01.22; peer-reviewed by M EIsenstadt, A Teles; comments to author 25.02.22; revised version received 08.03.22; accepted 23.03.22; published 15.04.22

    Copyright

    ©Meaghan McCallum, Annabell Suh Ho, Ellen Siobhan Mitchell, Christine N May, Heather Behr, Lorie Ritschel, Kirk Mochrie, Andreas Michaelides. Originally published in JMIR Formative Research (https://formative.jmir.org), 15.04.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.