Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 03.07.20 in Vol 4, No 7 (2020): July

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/17722, first published Jan 10, 2020.

This paper is in the following e-collection/theme issue:

    Original Paper

    Use of a Mobile App to Augment Psychotherapy in a Community Psychiatric Clinic: Feasibility and Fidelity Trial

    1Rose: Smarter Mental Health, Washington, DC, United States

    2Key Point Health Services, Inc, Baltimore, MD, United States

    3Department of Psychiatry and Behavioral Sciences, Johns Hopkins University, Baltimore, MD, United States

    4Carey Business School, Johns Hopkins University, Baltimore, MD, United States

    Corresponding Author:

    Atif Adam, MPH, MD, PhD

    Rose: Smarter Mental Health

    3400 Prospect St NW

    Washington, DC, 20007

    United States

    Phone: 1 4103363626

    Email: dratif@askrose.com


    ABSTRACT

    Background: Even though 1 in 5 Americans experience some form of mental illness each year, 80% have been shown to discontinue psychotherapy prematurely. The traditional psychotherapy service delivery model, consisting of isolated clinical sessions, lacks the ability to keep patients engaged outside clinical sessions. Newer digital mental health platforms can address the clinical need for a robust tool that tracks mental well-being and improves engagement in patients with depressive symptoms.

    Objective: The primary goals of this feasibility study were to (1) assess compliance among providers and their patients with a digital mental health platform protocol, and (2) examine the usability and fidelity of a mobile app through structured participant feedback.

    Methods: A sample of 30 participants was recruited for a 5-week study from a community-based mental health clinic in Baltimore, Maryland, USA. Inclusion criteria were: aged 18 years or older, having access to a smartphone, and having at least mild-to-moderate depression and/or anxiety as measured by the Patient Health Questionnaire-9 (PHQ-9) and Generalized Anxiety Disorder-7 (GAD-7) scales, respectively. Eligible participants were randomized into one of two study arms: (1) the intervention arm or (2) the waitlist control arm. Participants in the intervention arm were asked to download the Rose app and were prompted to complete clinical assessments (PHQ-9 and GAD-7) every other week, daily mood and anxiety Likert scales, and daily journal entries. The participants in the waitlist arm served as controls for the study and completed the clinical assessments only. Both arms engaged in weekly psychotherapy sessions, with participant in-app input informing the psychotherapy process of the intervention arm, while those in the waitlist control arm continued their standard care. Outcomes of interest included adherence to completion of in-app assessments and usability of the Rose mobile app assessed through the modified Mobile Application Rating Scale.

    Results: Over the study period, a sample of 30 participants used the Rose app 2834 times to complete clinical assessments. On average, 70% (21; 95% CI 61.14%-77.41%) of participants completed mood and anxiety daily check-ins and journal entries 5 days per week. Nearly all participants (29/30, 97%) completed all PHQ-9 and GAD-7 in-app scales during the study. Subjective impressions showed that 73% (22/30) of participants found the mobile app to be engaging and in line with their needs, and approximately 70% (21/30) of participants reported the app functionality and quality of information to be excellent. Additionally, more than two-thirds of the participants felt that their knowledge and awareness of depression and anxiety management improved through using the app.

    Conclusions: Steady compliance and high app ratings showcase the utility of the Rose mobile mental health app in augmenting the psychotherapy process for patients with mood disorders and improving mental health knowledge. Future studies are needed to further examine the impact of Rose on treatment outcomes.

    Trial Registration: ClinicalTrials.gov NCT04200170; https://clinicaltrials.gov/ct2/show/NCT04200170

    JMIR Form Res 2020;4(7):e17722

    doi:10.2196/17722

    KEYWORDS



    Introduction

    Each year, 1 in 5 Americans (approximately 46.6 million people) experience some form of mental illness [1-3]. The National Institute of Mental Health estimated that 17.3 million adults in the United States had at least one major depressive episode in 2017, representing 7.1% of the population. However, studies show that up to 60% of adults suffering from mental illness report an inability to receive appropriate treatment and close to 80% seeking care prematurely discontinue psychotherapy [4-6]. Reasons for discontinuation include the stigma associated with psychotherapy, difficulties in connecting with therapists outside of clinical sessions, and poor fit with therapists [7,8].

    In recent years, feedback-informed care (FIC) models have been used to improve patient compliance and psychotherapy outcomes [9,10]. FIC models are designed to encourage patients to provide feedback on care progress, assess well-being outside clinical sessions, and utilize a data driven approach to bolster the therapeutic alliance and build individually tailored therapies [9]. A key gap in FIC models has been the availability of resources and tools that allow patients to stay connected to their providers and capturing mental well-being data that currently can only be collected during relatively infrequent in-person psychotherapy sessions [11,12].

    The widespread use and availability of smartphones have made successful adoption of mobile technology increasingly crucial in developing FIC models [12-14] and identifying best practices. Unfortunately, of the 10,000 mental health apps currently available, less than 5% are backed by evidenced-based research [15]. Novel health technologies looking to revolutionize mental health care delivery and empower patients need to have an evidence-base that captures engagement of patients in their daily life with their therapists and the utility of collecting wellbeing data to address specific health concerns. The Rose digital health platform was developed in response to the clinical need for a robust tool to track mental wellbeing outside of clinical care. Rose’s digital health platform integrates a patient-facing mobile phone app for real-time mood tracking, clinical surveys, daily journaling, and curated insights with a web-based clinical dashboard allowing clinicians to check on their patient’s mental health status between clinical visits (Figure 1).

    The primary goals of this feasibility study were to (1) assess the usage compliance with a mobile mental health platform among providers and their patients with at least mild-to-moderate depression and/or anxiety, and (2) examine the usability and fidelity of the mobile app through structured participant feedback. We hypothesized consistent usage adherence to the mobile platform over the study’s duration. A secondary exploratory aim evaluated the short-term impact of mobile app usage on mood and anxiety symptomatology.

    Figure 1. Rose mobile app screens.
    View this figure

    Methods

    Recruitment

    Participants were recruited from the patient population at Key Point Health Services, a sizeable outpatient community psychiatric clinic in Baltimore, Maryland, USA. Inclusion criteria consisted of age 18 years or older, regularly making on-going appointments with a psychotherapist at the clinic, having a score of >5 on the Patient Health Questionnaire-9 (PHQ-9) and/or >5 on Generalized Anxiety Disorders-7 (GAD-7), having access to a smartphone, and being fluent in English. Exclusion criteria included a current diagnosis of decompensated psychosis (eg, hallucinations, delusions, thought disorder) and/or current suicidal or homicidal ideation. The cut-off of 5 on the PHQ-9 and GAD-7 represent mild depression and mild anxiety from clinical severity categorization [16,17]. Eligible patients were informed of the study via brochures provided by their established psychotherapists. Interested patients were directed to a secure online screener, which collected data on inclusion and exclusion criteria, demographics, and the PHQ-9 and GAD-7. This pilot was designed as an open-label trial with randomized allocation into study arms. A study member who was not involved in the day-to-day study operations used the STATA 14 (StataCorp) sequence generator feature for randomization. These were stored in sealed envelopes. At study registration and the online screening, the study coordinator opened the randomization envelopes to allocate the eligible participants to either the intervention arm or the control arm. Eligible, consented participants were then randomized via computer algorithm into one of two study arms: (1) the intervention arm or (2) the waitlist control arm, in a 2:1 ratio. All nine study therapists had appointments with at least one participant from each arm. Of note, no patient medical records were accessed as part of the study.

    Intervention Arm Design

    Eligible participants continued meeting with their therapists for weekly psychotherapy sessions. Therapists were asked to register for the Rose web-based clinical dashboard, which generated a unique therapist ID. Participants in the intervention arm were asked to download the Rose app to their mobile device and were given their therapist’s unique ID for linking within the app. Participants were informed that the health metrics would be shared with their therapist to help the clinician better plan upcoming sessions. Participants were prompted to complete a prepilot battery of assessments (PHQ-9, GAD-7, mood and anxiety Likert scales) upon first logging into the app. This battery was also pushed to the users at 2- and 4-week (postpilot) time points. Those in the intervention arm were also asked to continue using the Rose app daily during the 4-week period. The therapist was asked to use the Rose clinical dashboard to review data in conjunction with each weekly in-clinic session for four sessions. All in-clinic sessions were scheduled in the same clinical space where participants previously received care. Each clinical session was approximately 45 minutes long, and therapist documentation was done using their standard of care electronic medical record system.

    Control Arm Design

    The participants in the waitlist control arm served as controls for the study. They completed the prepilot and postpilot assessments (PHQ-9, GAD-7, mood and anxiety Likert scales) only. These participants continued their weekly standard care with the psychotherapy process. Waitlist participants were offered entrance to the intervention arm at the end of the study period or earlier if participants in the intervention arm dropped out mid-study. During their time on the waitlist, participants could reach out to study personnel if they needed assistance with their psychiatric care.

    Ethics

    The study was reviewed and approved by the Advarra Commercial Institutional Review Board. For data security purposes, Rose utilizes cloud-based file-sharing and data storage services that are ISO 27001 certified and HIPAA (Health Insurance Portability and Accountability Act) compliant. If at any point during the study, participants denoted suicidal ideation or other acute psychiatric concerns, they were automatically provided with emergency contact numbers and the study clinician (AJ) reached out directly. Additionally, if a high-risk participant was identified during in-person psychotherapy sessions, the therapist contacted the study clinician immediately with any concerns. If it was deemed that a participant required a higher level of care (eg, inpatient hospitalization), they were removed from the study.

    Mobile Mental Health App (Rose)

    Rose is a mobile health platform that consists of two primary components: (1) a patient-facing mobile app and (2) a clinician-facing web-based dashboard. Patients and clinicians are linked within the Rose system using a unique identifier that the clinician provides to the patient. The Rose app allows patients to track their mood and anxiety levels in real-time, complete validated assessments, and keep an in-app daily journal. Rose utilizes machine learning algorithms that identify mental health status and provide curated, individualized in-app daily insights for self-care. The Rose clinician-facing dashboard is designed for use in conjunction with in-person appointments. It is meant to augment in-person care by summarizing the data entered by the patient into the mobile app. Providers are given a walkthrough on using the dashboard and can reach the Rose team with any questions directly through the dashboard.

    Measures

    The measures were as follows:

    • Patient Health Questionnaire-9 (PHQ-9) [17] and Generalized Anxiety Disorder-7 (GAD-7) [18] Scale. The PHQ-9 and GAD-7 are valid and reliable screening tools for depression and anxiety, respectively. These assessments parallel the diagnostic symptom criteria that define DSM-IV major depression and GAD. The Rose app prompted users to complete these assessments every two weeks. The app follows the format and temporal framework corresponding to DSM-IV criteria.
    • Modified Mobile Application Rating Scale (mMARS) (Multimedia Appendix 1) [19]. The MARS is a well-established framework for classifying and assessing the objective and subjective quality of apps, as well as their perceived impact. It is designed to score apps on the criteria of engagement, functionality, aesthetics, information quality, and subjective app quality. While the MARS framework is extensive, sections within the scale were not pertinent for this study and the overall length was deemed too tedious for patients to complete. The mMARS is a shortened version of the MARS, keeping the pertinent sections and modifying verbiage to reflect the current app usage.
    • Mood & Anxiety Likert Scales [20]. The Rose app sends timed notifications once a day to rate current mood and anxiety, each on a 5-point Likert scale.
    • In-app Journaling. The in-app journaling feature allows the user to enter a free text description of how they are doing, like in the case of a diary. A built-in sentiment analysis analyzes the entered sentences.

    Statistical Power Calculation

    The power calculation for this study is based on a meta-analysis of internet-delivered treatments for adult depression and anxiety. It was found that internet interventions used in tandem with professional psychotherapy support showed an average between-group effect size of d=0.61 (large mean effect size of d=1.0). Based on the study’s five-psychotherapy session design (first session for on-boarding and four follow-up sessions), and paired-sample design (multiple measurements from the same person), a sample size of 30 has sufficient power (90%) to detect a moderately large effect size (d=0.8) [21]. Additionally, we recruited 15 patients as a waitlist control group who would be included in the intervention arm if drop-out or removal occurred.

    Primary Analyses

    The primary analysis included checking distributional assumptions from the primary scores and assessing relationships among covariates. Continuous variables were described using mean, standard deviation, median, minimum and maximum, and 95% confidence interval for parameter estimation.

    To determine whether any significant differences between groups existed at baseline, independent t tests were conducted on continuous baseline variables (eg, age, PHQ-9, GAD-7), and chi-square analyses were performed on categorical variables (gender, race, ethnicity).

    Descriptive in-app metrics were calculated for compliance of assessments with PHQ-9 and GAD-7 every two weeks and daily mood/anxiety Likert scales and journaling. In-app use was also explored for time-of-day frequency and weekly variations.

    • Adherence over time. Adherence was defined as the completion of in-app assessments at predefined intervals. For example, daily adherence to mood assessment was seen if a participant completed at least one of the mood or anxiety Likert scales each day. Non-adherence was recorded as days with no data. Each assessment measure was reviewed individually for the duration of the study (4 weeks) for the percentage of adherent days and was stratified for each study week in which the app was being used. Logistic regression models were carried out to examine whether adherence to in-app assessments would be associated with the severity of depressive symptoms at baseline.
    • Fidelity of app use. Fidelity defines the degree to which programs are implemented as intended by the program developer [22]. In this study we looked at five main components of fidelity for the mobile app using the mMARS scale: Engagement, Functionality, Aesthetics, Quality of Information, and Subjective Quality. Summary proportions from the mMARS were reviewed in all five areas of app use.

    Exploratory Analyses

    Pre-post analysis (Wilcoxon matched-pairs signed-rank test) was used to evaluate the short-term impacts of mobile app usage in the intervention arm and the control arm within in-person psychotherapy sessions.

    All analyses were conducted using STATA 14 (StataCorp). All statistical tests were judged for significance based upon a two-tailed alpha level of P<.05. All subjects were included in intention-to-treat (ITT) analyses.


    Results

    Participant Characteristics

    Table 1 shows the demographic information and baseline scores on clinical variables for those with data from the entire sample (n=45). Each therapist saw on average 3 patients (CI: 1-5). Participants in the intervention arm (n=30) were an average of 36 years old. Close to 60.00% of the participants were female and 29.03% reported completing college or higher level of education. Participants were mostly non-Hispanic making up 92.58%. 82.14% were Caucasian, 10.71% African American, and 7.14% identified with more than one race. Just over half (51.61%) used Android devices as their daily smartphone and the rest used Apple iOS devices. Other than the proportion of female participants (P=.02), there were no significant demographic differences between arms.

    Looking at baseline mental health status, 42.22% of the sample was in the moderately-severe or severe range of depression, as measured by the PHQ-9. GAD-7 ratings showed 9.60% of the participants were in the moderately-severe or critical range for anxiety. There were no statistical differences in the distribution of participants randomized to each arm based on baseline mental health status.

    Table 1. Demographic and baseline characteristics of study participants.
    View this table

    Adherence Over Time

    Participants in the intervention arm used the Rose mobile app over 4 weeks as part of care management. Table 2 details the descriptive statistics on participants’ daily use of the mood Likert scale, anxiety Likert scale, and online journaling; overall and by study week. Over the study period, patients adhered to completing all three assessments at an average of 69% per week (5 days a week). As seen in Table 2, adherence across all three estimates was reasonably similar. Adherence was highest in the first week, averaging at 81.17% (95% CI: 75.31%-87.04%) across all three assessments, with a linear decrease at week 4 to 56.03% (95% CI: 43.92%-68.14%) adherence. Majority adherence of 80% (24/30 days) was seen in 14 of 30 participants completing the mood Likert scale, 15 of 30 participants completing the anxiety Likert scale, and 13 of 30 participants completing the journal.

    Looking at the use of PHQ-9 and GAD-7 in-app scales (Table 3), participants adhered to filling in all three assessments at an average rate of 97% over the study period (every two weeks). Full adherence (3/3 assessments) was seen in 27/30 of the patients.

    Table 2. Weekly adherence rates for in-app mood and anxiety Likert scales and journaling.
    View this table
    Table 3. Biweekly adherence rates for in-app PHQ-9 and GAD-7 assessments.
    View this table

    Fidelity of App Use

    Using the mMARS scale we evaluated the perceived usability of the mobile app as rated by participants (Figure 2). Overall, participants thought the app was very engaging, with approximately 73% finding Rose very interesting and appropriate for the target audience. Approximately 83% of the participants reported the app to be very accurate, easy to use, and easy to navigate. Similarly, 75% reported the layout and graphics to be very pleasing and 71% felt the overall look was only adequate. Furthermore, 54% reported that the information provided through the app was of good or excellent quality, and 77% of the participants felt it covered a comprehensive range of topics. Overall, 67% said they would recommend the app outside of the study.

    We further asked participants about the perceived impact of the Rose app on their knowledge, attitudes, intentions to change, as well as the likelihood of actual change in depression and/or anxiety symptoms (Figure 3). More than two-thirds of the participants felt that their knowledge of the topic and awareness of the importance of depression and anxiety management improved through using the app. Similarly, 67% reported they were more than likely to take progressive steps to address depression and more than half stated that the app made them likely to seek help for depressive symptoms when needed.

    Figure 2. mMARS – Perceived Usability of Rose.
    View this figure
    Figure 3. Impact of App Usage on Mental Health Knowledge and Attitudes.
    View this figure

    Exploratory Analyses

    We looked for changes in depression and anxiety symptoms (measured through PHQ-9 and GAD-7) before and after the study in both the intervention and control arms (Table 4).

    Participants in the intervention arm showed statistically significant improvements in measurements of both depression and anxiety symptoms. On the PHQ-9, there was an average 4-point improvement in participants over four weeks (prepilot average score: 15 vs postpilot average score: 11; P=.01). On the GAD-7, there was an average 3-point improvement in participants over four weeks (prepilot average score: 11 vs postpilot average score: 8; P=.005). Overall, 47% of patients with moderate-to-severe depression improved to mild depression and 56% of patients with moderate-to-severe anxiety improved to mild anxiety by the end of the study. Additionally, Kruskal-Wallis H test revealed statistically significant improvements in both PHQ-9 (P=.02) and GAD-7 (P=.001) scores over the course of the study in the intervention group.

    Among the six controls who completed the pre- and postpilot assessments, neither PHQ-9 nor GAD-7 scores showed any significant changes (Table 5). Even though there was a difference in mean scores on both PHQ-9 and GAD-7 scales, statistical comparison on PHQ-9 and GAD-7 scores between intervention and control arms did not show statistical significance.

    Table 4. Pre-post change in depression and anxiety symptoms: intervention (n=30) vs control (n=15).
    View this table
    Table 5. Pre-post analysis of changes in depression and anxiety symptoms.
    View this table

    Discussion

    Principal Findings

    This study demonstrated the fidelity of using a digital mental health platform (Rose) for monitoring daily mental health and wellbeing outside of psychotherapy sessions and maintaining engagement in patients with depression and anxiety symptoms. Exploratory analyses examined changes in depressive and anxiety symptoms at baseline versus after study interventions. Improved PHQ-9 scores were observed in 76% (23/30) of intervention arm participants and 70% (21/30) showed improvements in their GAD-7 scores over the study period. Furthermore, patients rated the Rose mobile app as having high engagement and usability as well as content curation relevant to their needs.

    The study paralleled current psychotherapy care practices for moderate depression. These results showed using Rose is no worse than not using an app within psychotherapy sessions. Consistent compliance by patients between and during clinical psychotherapy sessions indicates that high levels of adherence can be achieved and retained. A larger multisite randomized controlled trial is being planned to look at the impact of the Rose digital health platform with an active (alternative mobile app) control.

    Comparison With Prior Work

    The economic impact of mental illness is marked, with the United States alone seeing $210 billion in annual medical expenditures tied to mental health disorders [23,24]. The added societal burdens of mental illness include productivity losses, absenteeism, and reduced economic growth [25]. On average, depressive symptoms lead to approximately 27 lost workdays and 18 days of reduced productivity per year in people diagnosed with depression [26]. Research attention to address these needs has exponentially increased in recent years, especially around the area of mobile mental health apps [14,27]. Research groups working in this space have found, through both observational and clinical trials, positive impacts on the utility of smartphones with onboard sensors to diagnose psychiatric disorders [12]. However, studies of adherence to use of mobile mental health apps have been limited. A meta-analysis by Tourous et al looked at 18 clinical trials that investigated the efficacy of smartphone interventions targeting depressive symptoms. The authors note that lower rates of engagement over time have been found in numerous mental health app studies [28], with higher rates being more common in shorter intervention interactions. Rates of adherence were notably higher with apps that had added components of human support, with real world engagement rates close 17% for peer support apps, but adding human interactions may reduce scalability, especially in low resource settings [28]. This suggests that intervention designs have to be adaptable and customizable to the ways in which people use smartphones. Results from the current study demonstrate the utility of smartphone-driven interventions to improve on FIC and in maintaining nonhuman-supported adherence. Furthermore, there are few studies that have looked at design, fidelity, and usability of mental health apps tied to intervention outcome and patient engagement [29,30]. This study is also one of the first to look at mobile mental health app outcome metrics and interaction principles in a longitudinal investigational format.

    Limitations

    Participants in this study were selected from a single study site and were not randomized by therapists. Thus, there may be a sampling bias that limits generalizability of the results. Furthermore, the sample consisted of 60%-90% women, which is not representative of the population of individuals with depression/anxiety in clinical settings or the community. While the pilot study had a waitlist control arm, it did not include an active comparison group (ie, using an alternate mobile app). Furthermore, only 40% of the control arm completed the postpilot follow-up. This limited the pre-post analytical comparison. A follow-up randomized control trial is needed to evaluate inferences regarding the efficacy and contrast of effect sizes between the Rose digital health platform and other interventions.

    Conclusions

    Steady compliance and high app ratings showcase the usability of Rose in practice and the utility of digital FIC models in improving patient compliance to therapy. Although exploratory, reduction in clinical depression and anxiety symptoms over the four-week study period highlights the potential utility of the Rose app for psychotherapy augmentation. Future studies are being developed to (1) further evaluate the clinical effectiveness of the Rose app through a multisite micro-randomized clinical trial, and (2) examine the impact of the Rose digital health platform on other diagnoses, including substance use and pain.

    Acknowledgments

    All authors have contributed substantially to the conceptualization, design, analysis, interpretation, and drafting of the manuscript. AJ provided expert guidance and leadership throughout the project – she was involved throughout study management, overseeing participant recruitment, staff training and writing the manuscript. MP and AA provided leadership in helping guide the app review and training of staff, technical support during the study for the health center, study design, parameterizing the models, data analysis and writing the manuscript. AP focused on participant recruitment and data collection. KM provided oversight and technical input on the mobile app integration into clinical practice and working with patient app issues. RB was involved in staff training and implementation of the clinical dashboard, gathering user feedback, and manuscript preparation. LR and NG were involved in manuscript drafting. The authors are grateful to Mr. Raghav Sambasivan, Mr. Shravan Venkatesan, Mr. Darsh Patel, Mr. Anthony Jairam, and Ms. Allison Vita for their tremendous support helping run the Rose study and for helping build the mobile app platform. The authors also appreciate the supportive partnership of Key Point Health Services, Inc.

    Conflicts of Interest

    AA, KM, and MP are on the executive team of Ask Rose Inc, the company that developed the mobile app used in this study. AV is an employee with Ask Rose, Inc. NG is a clinical consultant to Ask Rose, Inc. This study was funded by Ask Rose, Inc.

    Multimedia Appendix 1

    Modified Mobile Application Rating Scale (mMARS).

    PDF File (Adobe PDF File), 1682 KB

    References

    1. Indu PS, Anilkumar TV, Pisharody R, Russell PSS, Raju D, Sarma PS, et al. Prevalence of depression and past suicide attempt in primary care. Asian J Psychiatr 2017 Jun;27:48-52. [CrossRef] [Medline]
    2. Pratt LA, Brody DJ. Depression in the U.S. household population, 2009-2012. NCHS Data Brief 2014 Dec(172):1-8 [FREE Full text] [Medline]
    3. Pratt L, Brody DJ. Depression in the United States household population, 2005-2006. NCHS Data Brief 2008 Sep(7):1-8 [FREE Full text] [Medline]
    4. Sayers J. Mental health: new understanding, new hope. In: The world health report 2001. Geneva: World Health Organization; 2001.
    5. Wang PS, Berglund P, Olfson M, Pincus HA, Wells KB, Kessler RC. Failure and delay in initial treatment contact after first onset of mental disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005 Jun 01;62(6):603-613. [CrossRef] [Medline]
    6. Wang PS, Lane M, Olfson M, Pincus HA, Wells KB, Kessler RC. Twelve-month use of mental health services in the United States: results from the National Comorbidity Survey Replication. Arch Gen Psychiatry 2005 Jun 01;62(6):629-640. [CrossRef] [Medline]
    7. Swift JK, Greenberg RP. Premature discontinuation in adult psychotherapy: a meta-analysis. J Consult Clin Psychol 2012 Aug;80(4):547-559. [CrossRef] [Medline]
    8. Werbart A, Andersson H, Sandell R. Dropout revisited: patient- and therapist-initiated discontinuation of psychotherapy as a function of organizational instability. Psychother Res 2014 Feb 13;24(6):724-737. [CrossRef] [Medline]
    9. Miller SD, Hubble MA, Chow D, Seidel J. Beyond measures and monitoring: Realizing the potential of feedback-informed treatment. Psychotherapy (Chic) 2015 Dec;52(4):449-457. [CrossRef] [Medline]
    10. Delgadillo J, de Jong K, Lucock M, Lutz W, Rubel J, Gilbody S, et al. Feedback-informed treatment versus usual psychological treatment for depression and anxiety: a multisite, open-label, cluster randomised controlled trial. The Lancet Psychiatry 2018 Jul;5(7):564-572. [CrossRef]
    11. Delgadillo J, Overend K, Lucock M, Groom M, Kirby N, McMillan D, et al. Improving the efficiency of psychological treatment using outcome feedback technology. Behav Res Ther 2017 Dec;99:89-97. [CrossRef] [Medline]
    12. Lindhiem O, Bennett CB, Rosen D, Silk J. Mobile technology boosts the effectiveness of psychotherapy and behavioral interventions: a meta-analysis. Behav Modif 2015 Nov 17;39(6):785-804 [FREE Full text] [CrossRef] [Medline]
    13. Lui JHL, Marcus DK, Barry CT. Evidence-based apps? A review of mental health mobile applications in a psychotherapy context. Professional Psychology: Research and Practice 2017 Jun;48(3):199-210. [CrossRef]
    14. Menon V, Rajan T, Sarkar S. Psychotherapeutic Applications of Mobile Phone-based Technologies: A Systematic Review of Current Research and Trends. Indian J Psychol Med 2017;39(1):4-11 [FREE Full text] [CrossRef] [Medline]
    15. Torous J, Roberts LW. Needed Innovation in Digital Health and Smartphone Applications for Mental Health: Transparency and Trust. JAMA Psychiatry 2017 May 01;74(5):437-438. [CrossRef] [Medline]
    16. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    17. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef] [Medline]
    18. Löwe B, Decker O, Müller S, Brähler E, Schellberg D, Herzog W, et al. Validation and Standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the General Population. Medical Care 2008;46(3):266-274. [CrossRef]
    19. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [CrossRef] [Medline]
    20. Ahearn EP. The use of visual analog scales in mood disorders: A critical review. Journal of Psychiatric Research 1997 Sep;31(5):569-579. [CrossRef]
    21. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009 Dec 15;38(4):196-205. [CrossRef] [Medline]
    22. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res 2003 Apr;18(2):237-256. [CrossRef] [Medline]
    23. Roehrig C. Mental Disorders Top The List Of The Most Costly Conditions In The United States: $201 Billion. Health Aff (Millwood) 2016 Jun 01;35(6):1130-1135. [CrossRef] [Medline]
    24. Davlasheridze M, Goetz SJ, Han Y. The Effect of Mental Health on US County Economic Growth. Review of Regional Studies 2018;48(2):155-171.
    25. Trautmann S, Rehm J, Wittchen H. The economic costs of mental disorders: Do our societies react appropriately to the burden of mental disorders? EMBO Rep 2016 Sep 04;17(9):1245-1249 [FREE Full text] [CrossRef] [Medline]
    26. Brohan E, Thornicroft G. Stigma and discrimination of mental health problems: workplace implications. Occupational Medicine 2010 Aug 18;60(6):414-415. [CrossRef]
    27. Boschen MJ, Casey LM. The use of mobile telephones as adjuncts to cognitive behavioral psychotherapy. Professional Psychology: Research and Practice 2008 Oct;39(5):546-552. [CrossRef]
    28. Torous J, Lipschitz J, Ng M, Firth J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: A systematic review and meta-analysis. J Affect Disord 2020 Feb 15;263:413-419. [CrossRef] [Medline]
    29. Hays R, Henson P, Wisniewski H, Hendel V, Vaidyam A, Torous J. Assessing Cognition Outside of the Clinic: Smartphones and Sensors for Cognitive Assessment Across Diverse Psychiatric Disorders. Psychiatr Clin North Am 2019 Dec;42(4):611-625. [CrossRef] [Medline]
    30. Hilty DM, Chan S, Torous J, Luo J, Boland RJ. Mobile Health, Smartphone/Device, and Apps for Psychiatry and Medicine: Competencies, Training, and Faculty Development Issues. Psychiatr Clin North Am 2019 Sep;42(3):513-534. [CrossRef] [Medline]


    Abbreviations

    FIC: feedback-informed care
    GAD-7: Generalized Anxiety Disorder-7
    HIPPA: Health Insurance Portability and Accountability Act
    ITT: intention-to-treat (ITT)
    mMARS: modified Mobile Application Rating Scale
    PHQ-9: Patient Health Questionnaire-9


    Edited by G Eysenbach; submitted 10.01.20; peer-reviewed by A Walsh, S Jahed, J D'Arcey; comments to author 09.03.20; revised version received 05.04.20; accepted 09.04.20; published 03.07.20

    ©Atif Adam, Ameena Jain, Alexandra Pletnikova, Rishi Bagga, Allison Vita, Lisa N. Richey, Neda Gould, Supriya Munshaw, Kavi Misrilall, Matthew E Peters. Originally published in JMIR Formative Research (http://formative.jmir.org), 03.07.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.