In this 5-week study, the first week was observational and used to gather digital phenotyping data on all participants. After this first week of data collection, all participants were randomly assigned to receive app recommendations based on their digital phenotyping data or to select an app without any assistance or data. Over the next 4 weeks, participants used their designated app and completed pre- or postintervention questionnaires via web-based study visits.

All participants were recruited from ResearchMatch. Inclusion criteria included being aged >18 years, being proficient in the English language, being able to sign an informed consent form through a web-based process, having a primary care physician or psychiatrist, owning an Apple or Android smartphone, and scoring higher than 5 on the General Anxiety Questionnaire-7 (GAD-7) at the initial visit.

All participants used 2 digital health apps throughout the study. The first app that every participant used was mindLAMP, an app developed by the Division of Digital Psychiatry at Beth Israel Deaconess Medical Center (BIDMC) [24]. In this study, mindLAMP served solely as a digital phenotyping data collection tool. The second app varied between participants and served as an intervention tool. Participants downloaded 1 of 4 intervention apps: UCLA Mindful (University of California Los Angeles Health), How We Feel (HWF Project Inc)., Insomnia Coach (US Department of Veterans Affairs), or Nike Training Club (Nike, Inc).

mindLAMP is a digital phenotyping app developed at BIDMC [24,25]. mindLAMP has a customizable interface with 5 main sections: feed, learn, assess, manage, and portal. While it can be customized to offer both interventional and data capacities, this study only used its data collection capacity including custom surveys and sensors (GPS, accelerometer, and screen use metrics; Figure 1).

All participants were designated to engage with 1 of 4 health apps for 4 weeks: UCLA Mindful, How We Feel, Insomnia Coach, and Nike Training Club. UCLA Mindful offers guided meditations for users [26]. How We Feel is a mood-tracking app that offers a range of emotions for users to choose from while also tracking aspects of their physical health such as sleep and exercise [27]. Insomnia Coach guides users with their sleep through cognitive behavioral therapy and offers weekly training with a sleep coach, tips, a log, and a diary [28]. Nike Training Club Fitness offers home workouts to healthy recipes [29]. All apps were found through the Mobile Health Index and Navigation Database (MINDapps) developed by the Division of Digital Psychiatry [30]. For this study, we created a new filter on MINDapps to display the app(s) when participants were recommended or selected an app (Figure 2). The selection of apps to include was based on feedback from patients in our clinic, advisory board, volunteer MINDapps app raters, and our prior research on app engagement [21].

To assess if a personalized app recommendation would increase engagement effectively, we assessed correlations between self-reported engagement and objective engagement. To measure this relationship, an ordinary least squares (OLS) linear regression model was implemented using the statsmodels library in Python (Python Software Foundation). To assess the relationship between participants’ engagement with their respective apps and the participant’s anxiety symptoms, a series of OLS linear regressions were performed using the statsmodels library.

A regression model was created for every measure of engagement, which was either objective (mean app screen time) or subjective (participant’s self-reported engagement gathered from a survey). From these measures of engagement, we compared them to the change in structured surveys they took during the study. These surveys include the GAD-7, Flourishing Scale, UCLA Loneliness Scale, SIAS, ISI, and PSS-10. The specific change in structured survey scores was calculated from appointment 2 (when app use occurred) to appointment 3 (the final appointment).

Following the Braun and Clarke [32] framework, a group of 5 research assistants initially reviewed the raw responses to the open-ended questions in the final engagement survey. They identified themes associated with the use and engagement of the app and added them to a table: notifications, memory, ease of use, and the content of the app. Each individual’s final engagement survey was printed out and rated by at least 2 research assistants to ensure interrater reliability. They marked where they saw the theme and indicated whether it seemed positive or negative (ie, “The notifications were annoying” → notifications → negative). In the case of dispute, an additional research assistant contributed a rating until a consensus was identified. A spreadsheet was developed to indicate themes and positive, negative, or both associations.

This study was approved by the BIDMC institutional review board as protocol 2022P001143. Written informed consent regarding primary and potential secondary data analyses of research data was collected and documented for all participants via the REDCap (version 14.0.30; REDCap Consortium). The protected health information of participants was securely stored on REDCap, which is a Health Insurance Portability and Accountability Act–compliant, web-based app specifically designed for research data collection and management. Study data were subsequently deidentified during the analysis process. No identification of individual participants or users is included in this paper. Participants did not receive any form of compensation for this study.

A total of 82 adults were recruited and enrolled in the study. There were no significant differences in sex for the control and experimental groups (P=1.0). There were no significant differences in baseline anxiety or depressive symptoms in each group (P>.05).

A total of 35 (43%) participants dropped out; 22 (27%) dropped out of the study after the first meeting, divided evenly between the treatment (n=11, 13%) and control groups (n=11, 13%). Of those participants, 9 (11%) left for unknown reasons, 6 (7%) lost interest, 3 (4%) left due to the time commitment, 3 (4%) for data quality reasons, and 1 (1%) participant left for a family emergency. After the second meeting, 13 (16%) participants dropped out (control: n=7, 8%; treatment: n=6, 7%). Two (2%) left for data quality reasons while the rest (n=11, 13%) left for unknown reasons. Table 1 below shows the full breakdown of the demographics for all 47 remaining participants.

Of note, only How We Feel and UCLA Mindful had enough participants to complete the study while using the app to produce meaningful results.

To determine if a personalized app recommendation would increase engagement at a population level, we plotted the mean objective engagement (mean screen time) of the control and experimental groups broken down by app (Figure 7).

The mean screen time values across apps could not be directly compared to self-reported engagement. In order to compare all apps against each other, we used the MinMaxScaler function from the sklearn Python library to map all mean screen time values to a 0-1 range for each app separately before combining the data for analysis.

Through OLS regression we found no significant correlation (R²=0.0188; P=.39) between self-reported engagement and scaled screen time in our pilot results. When participants were asked to rate their engagement on a scale of 1-10 (10=highest engagement), the control group’s mean rating was 6.42 (SD 2.52) as compared to the experimental group’s mean rating of 6.30 (SD 2.1).

In addition to comparing self-reported engagement to mean screen time, we also compared both self-reported engagement and mean screen time to the participant’s mean score on the D-WAI scale (Figure 8) using the MinMaxScaler function noted earlier.

There was a significant positive correlation between self-reported engagement and D-WAI scores (R²=0.1199; P=.02; coefficient=1.0238) but no equivalent correlation between scaled screen time and D-WAI values (R²=0.0013; P=.83; coefficient= –0.7849). In both cases, these findings were driven more by the control group than the experimental group.

In addition to comparing different types of engagement, a preliminary analysis compared engagement scores to changes in various clinical surveys. We explored how engagement metrics correlated with clinical symptom score changes after 1 month of app use in all participants. Overall, correlations between app engagement (via any metric) and clinical changes (via any survey) were small and most results were not statistically significant (P>.01). The small sample size precludes us from making any significant claims about the findings. A full table of results from our regression analyses for the How We Feel and UCLA Mindful apps can be found in Multimedia Appendix 2.

Using digital phenotyping to guide mental health app recommendations is feasible and results in higher levels of engagement over 4 weeks, although assessing the clinical impact of that higher engagement remains complex due to challenges in assessing meaningful engagement. Through qualitative analysis, we were able to better understand that the participants’ perception of engagement was driven less by perceived ease of use and perceived use and, instead, more by habit formation. These results have implications about the potential of clinical app recommendation to drive engagement, the importance of collecting both subjective and objective engagement metrics, and the role of habit formation for sustained app use.

In this study, objective engagement was highest in both groups during the first week of interventional app use and continuously decreased throughout the study period, with the largest drop from week 1 to week 2. The high engagement across both groups at the beginning suggests that common factors may drive initial engagement, but the differing course of engagement (Figure 7) suggests that distinct factors affect sustained use. As noted in the results (Figure 7), mean weekly screen time decreased nearly every week, while the digital phenotyping app recommendation group sustained higher mean screen times, suggesting the potential of this approach.

Our study results also highlight differences between objective and subjective measures of engagement. Despite having lower objective engagement scores as measured by screen time, the control group self-reported their engagement slightly higher than the digital phenotyping group. This lack of consistency between self-reported and actual use is well known [34] but rarely explored as a methodological consideration in mental health app use studies. Today, there remains no single accepted definition of engagement, with many proposals and explanations for why engagement often wanes [35,36] and methods like personalization and prompts to encourage engagement [37,38].

Inconsistencies in self-reported app use raise concerns about research methods relying solely on this as a measure of engagement. However, the value of either subjective or objective metrics of engagement is hard to determine, as there were few statistically significant correlations with any clinical changes after using these apps for 4 weeks. This negative finding could be due to the fact that the use of self-help apps is often not associated with large clinical changes and our sample size was underpowered to detect any small effects.

Our results on the D-WAI scores (Figure 8) showed a correlation with subjective, but not objective, screen time, which were considered as measures of engagement. This result is notable as prior studies have shown that this alliance metric may be a predictor of successful clinical outcomes with self-guided mental health apps but have not explored how it may impact engagement [20]. If alliance is related to subjective engagement as our results suggest, this raises mechanistic questions about how apps function and the need for further research exploring the dual role of subjective and objective factors in driving outcomes.

Additionally, our qualitative results suggest facets of a more nuanced picture of engagement beyond metrics like screen time or alliance. Participants agreed that perceived ease of use and perceived use were important factors in engaging with an app, which aligns with prior research findings grounded in TAM [22]. But while these 2 core factors were necessary for initial engagement, results suggest that sustained engagement requires the addition of habit formation. The higher rates of engagement that we saw for the digital phenotyping recommendation group may have been driven by the feedback and digital phenotyping information that could have helped participants create routines and patterns around app use.

Only 2 apps were picked for final analysis in order to produce meaningful results, because the other 2 apps (Insomnia Coach and Nike Training Club) did not produce an adequate sample size. The lack of uptake of those 2 apps was related to our clinical algorithm consistently recommending participants to target depression and anxiety symptoms, leading to disproportionate recommendations of their respective apps. Participants in the control group most frequently chose to download the other apps as well. The predominant self-selection of the 2 apps may be due to the high prevalence of anxiety and depression symptoms among adults, coupled with participants’ awareness that those apps targeted anxiety and depression. Additionally, both apps incorporate mindfulness techniques, an increasingly popular self-guided intervention. The visual appearance may have also influenced app selection, as it has been shown that app aesthetics play a large role in consumer appeal [39]. While we picked only 4 apps for participants to select from in this study, future studies could pick a larger number of different apps given the generalizability of this approach. Future studies should also use a method to ensure a more balanced representation across all app categories.

Another limitation of the study was the group design. The experimental group included the effect of both clinical and digital phenotypic recommendation, whereas the control group was based on a third condition of participant choice. This made it difficult to distinguish between the effect of clinical and data-based recommendations. Additionally, the control group still received app treatment, making it difficult to surmise results in the absence of intervention. Future studies should consider four distinct groups: (1) clinical recommendation; (2) digital phenotypic–based recommendation; (3) participant choice; and (4) no app (control). The small sample size (n=47) and high dropout rate (35/82, 43%) may have introduced bias and limited robust generalizability of study findings. Most participants dropped out after the first meeting for unknown reasons. A lower number of participants reported loss of interest or time commitment issues. Other participants finished the study but were excluded for data quality issues, a problem ubiquitous in digital phenotyping research. The lack of compensation and the remote nature of the study were most likely contributing factors to the small sample size and high dropout rate. Paying participants to attend study visits would have likely increased engagement but would have also confounded results. Holding in-person study visits may have decreased the dropout rate but would simultaneously made recruitment less feasible and led to a less geographically diverse sample.

The study was not designed to assess clinical impact and instead engagement. Future studies, powered appropriately, can explore if high engagement (both subjective and objective) is actually associated with improvements in depression or anxiety.

Digital phenotyping app recommendation is feasible and may increase rates of engagement. However, such models need to be carefully assessed before use in larger-scale studies as they may bias recommendations toward a subset of apps. Assessing the mechanism of how this approach increases engagement, whether through digital working alliance or habit formation, can help advance the use of digital phenotyping for app recommendation.