Since its earliest use in the mid-1990s, digital health promised personalized treatments that patients could access from home. It was anticipated that treatment would have a broad reach, resulting in improved health outcomes and decreased costs [1-3]. Over the past 2 decades, and with increasing consistency, research examining the efficacy of self-guided digital health interventions show evidence of efficacy, especially for those with mental health concerns [4-7].

Although digital health interventions appear to be effective, poor adherence and lack of compliance have remained consistent patterns in research [8-10]. This pattern was first recognized in 2005 and coined The Law of Attrition [11].

As early as 2009, systematic reviews identified poor adherence and lack of compliance as engagement issues that required attention [12]. These issues persist, as demonstrated by a recent meta-analytic review on digital interventions for depression, which showed efficacy but highlighted compliance as a major challenge [13].

Adherence and compliance are complex and rooted in several systemic and individual factors [14-19]. However, it is an important topic as evidence indicates a dose-response relationship, and higher levels of engagement are associated with improved health outcomes [20,21].

Moreover, digital health interventions are becoming increasingly common and accessible. Patients’ use of, and trust, in these interventions has been intensified by the COVID-19 pandemic. The use of self-guided digital health interventions for mental health concerns is growing [22,23] because of access barriers such as high cost, stigma, and lack of access due to a shortage of professionals who can meet this growing demand [24,25].

How do we increase engagement in digital health programs to maximize their efficacy?

Our primary hypothesis (H1) was that members would engage with the tips and to-do checklist. Engagement of the members was determined by the percentage of tips that were clicked on compared to the number shown. Because there is no prior literature, we first needed to establish baseline data. Data were reported as observational and served as a benchmark for future studies.

Our secondary hypothesis was that members would prefer to engage with directive tips in arm 2 versus social proof or present bias tips in arm 3. To assess the preference for engagement, we compared the number of tips clicked in arm 2 versus those clicked in arm 3.

Our tertiary hypothesis was that the addition of tips and a to-do checklist would increase completion rates with course tools. Increased completion rates were determined by comparing completion rates in arms 2 and 3 with those in arm 1 (control group).

In each hypothesis, we assessed whether engagement was influenced by gender or age.

The digital health platform used in this study was managed by Evolution Health. Evolution Health is an evidence-based, self-guided digital health platform that features courses and brief interventions based on behavior change techniques including CBT, stages of change, structured relapse prevention, normative feedback, and harm reduction.

The platform offers interactive courses and quizzes for people with mental health, addiction, and obesity issues. The platform contains a moderated community based on social cognitive theory.

Memberships are available to individuals who register through the organization’s free-to-consumer program and white-label instances that are licensed by employers, insurance companies, employee assistance programs, educational institutions, nonprofit organizations, for-profit health care organizations, and individual therapists.

The research protocol for this study was published in JMIR Research Protocols on August 15, 2022 [44]. The International Registered Report Identifier is DERR1-10.2196/37231.

The 2 interventions in this study contain self-guided interactive behavior change treatment courses based on best practices, and both have been examined extensively in the literature [9,45-53].

The 2 interventions have undergone several iterations. For example, Overcoming Anxiety was the first intervention noted in The Law of Attrition (previously known as The Panic Center) paper by Eysenbach [11]. In that iteration, the course contained a tunnel design with 12 successive sessions. The course now has a gamified free-form matrix design, among other technical and usability enhancements.

Table 2 outlines each course’s current theoretical construct and evidence base. Table 3 outlines the main course components.

At registration, all members endorsed a checkbox to confirm that they consented to have their nonidentifiable data used for research purposes and approved the platform’s privacy policy. The participants did not receive compensation for their involvement. The platform is available in several languages, and the English language privacy policy and terms of use are presented in Multimedia Appendix 1. All data collection policies and procedures adhered to the international privacy guidelines [54-56] and the Helsinki Declaration of 1975 [57].

This study was conducted on self-guided treatment for depression and anxiety, but it did not measure clinical outcomes. Although the study participants were randomly assigned to a control or intervention arm, the study was a randomized controlled trial and not a randomized clinical trial [58]. This study did not assess whether the study participants’ engagement with course tools decreased the depressive symptoms, severity of panic attacks, or frequency of panic attacks.

As described earlier, evidence indicates that higher levels of engagement are associated with improved health outcomes, and the literature observes dose-response relationships. However, any clinical outcomes related to the engagement strategies used in this study will need to be tested in future research.

As the study was based on unidentifiable data and no clinical measures were tested, it was deemed exempt from further review by the Evolution Health Institutional Review Board (IRB#000014034, FWA00033737).

The study was designed to have a power of 0.95, indicating a 95% probability of correctly detecting a statistically significant difference in engagement with tips between the 2 treatment groups (arm 2 vs arm 3). On the basis of a 2-tailed t test with a conventional α level (.05) for statistical significance, we required a sample of 1302 members with 651 in each group to detect a small effect size of Cohen d=0.2. Refer to Multimedia Appendix 2 for the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist [59].

During the registration process, new members were assigned to 1 of the 3 arms using a random number generator (Figure 4). Randomization was conducted using simple randomization.

A custom data collection interface and reporting mechanism were developed by Evolution Health. Age and gender data were collected at registration or through a secure sign on with various white-label instances. Data were collected for each member who was randomized into the experiment. The course components promoted by tips and to-do checklist items are listed in Textbox 1. Members who participated in both courses were not counted twice.

The following behaviors were tracked in the custom database for each tip and to-do checklist item that was randomly presented to a member:

It was possible for a single member to participate in both courses. However, the study design was to test the behavioral prompts, not the courses. The participants were randomized to an intervention arm, in which behavioral nudges and prompts were consistent across courses.

As there were no barriers to registration and many new members registered with the platform for purposes other than treatment, we removed those who registered but did not return to the intervention (nonparticipants) from the analysis.

All data were analyzed using mixed effect logistic regression with members as a random variable. Mixed effect logistic regression was conducted using the glmer() function from the lme4 package in R software (R Foundation for Statistical Computing), with the default optimizer Bound Optimization BY Quadratic Approximation, a derivative-free optimization algorithm used for problems with bound constraints.

For the tip-of-the-day section, the outcome variable was whether a user clicked on a tip presented to them (no or yes), and the predictors were arm (2 or 3), gender (female or male), and age group (18 to 30 years to >60 years).

For the to-do checklist items, the outcome variable was whether a user clicked on a checklist item (no or yes), and the predictors were gender and age group.

For the course components, the outcome variable was whether a user completed a component (no or yes), and the predictors were arm (1, 2, or 3), gender, and age group.

Between November 2021 and May 2022, data were collected from new members who self-registered for Evolution Health’s self-guided treatment program for anxiety and depression. All members were randomized into 1 of the 3 arms.

First, members with test accounts and unauthenticated accounts were removed from the data set, resulting in a population of 13,224 members. Then, members whose accounts were missing demographic data (376/13,224, 2.84%) were removed from the data set, followed by members aged <18 (129/13,224, 0.97%), resulting in a population of 12,719 members.

Finally, of the 13,224 members, 8567 (65.46%) with nonparticipant accounts were removed, resulting in a study population of 4567 (34.53%). Of the 4567 members, 1788 (39.15%) were randomized into arm 1, 865 (18.94%) into arm 2, and 1914 (41.9%) into arm 3 (Figure 8).

The average age of the members in arms 1, 2, and 3 was 44.4 (SD 10.69), 45.4 (SD 10.62), and 41.8 (SD 11.17), respectively (Table 4). The 51 to 60 years age group was the most populous age category in each of the arms (arm 1: 777/1788, 43.45%; arm 2: 420/865, 48.6%; and arm 3: 586/1914, 30.61%). The members of arm 3 were somewhat younger (F_2,4564=40.97; P<.001) than the other 2 arms.

Regarding gender (Table 5), most members identified as women (arm 1: 1466/1788, 81.99%; arm 2: 751/865, 86.8%; and arm 3: 1770/1914, 92.48%). Few members identified as transgender individuals, nonbinary or third gender individuals, or preferred not to say (arm 1: 5/1788, 0.28%; arm 2: 1/865, 1.2%; and arm 3: 2/1914, 0.1%). Arm 3 had more female members (χ²₄=92.21; P<.001) than the other 2 arms.

During the study period, there were 13,510 total log-ins, with initial registration considered as 1 log-in (Table 6). The average number of log-ins for male members was 3.89 and female members was 2.83. The average visit duration (AVD) for members who engaged with depression and anxiety course tools ranged from 6 minutes and 45 seconds, with 7.65 pages viewed, to 24 minutes and 21 seconds, with 12.56 pages viewed.

In arm 2, there were 31 revolving tips, and in arm 3, there were 30 revolving tips. Therefore, the probability of a member assigned to the arm 2 seeing a specific tip at least once is .088, and the probability of a member assigned to the arm 3 seeing a specific tip at least once is .091.

A total of 11,431 tips were displayed, of which 564 (4.93%) were clicked on (Table 7). In arm 2, 3622 tips were shown, of which 190 (5.24%) were clicked on. In arm 3, 7809 tips were shown, of which 374 (4.79%) were clicked on. Mixed effect logistic regression revealed no statistically significant differences between the 2 arms regarding the number of tips clicked on (P=.25).

In arm 2, female members clicked on 171 (5.82%) tips out of the 2937 tips shown to them and male members clicked on 19 (2.8%) tips out of the 682 tips shown to them, whereas in arm 3, females members clicked on 350 (4.94%) out of the 7081 tips shown to them and male members clicked on 24 (3.3%) out of the 722 tips shown to them (Table 8).

There were no statistically significant differences between number of tips clicked on between age groups (Table 9).

Mixed effect logistic regression (Table 10) revealed that female members clicked on more tips than male members (z score=2.58; P=.01). Although the role of gender and engagement should be more thoroughly examined in future studies, this finding is consistent with other research on platform components [60].

There were no substantial differences between age groups.

As outlined in Textbox 1 and illustrated in Figure 5, the to-do checklist contained 8 items. The members of arm 3 completed an average of 2.7 (34%) out of 8 course components.

The checklist item with the highest engagement was complete the depression and anxiety test, which was completed by 51.4% (55/107 of the members; Table 11). The second most popular item was watch the getting started video, with 48.2% (923/1914) of the members engaging with the behavioral cue. The checklist item with lowest engagement was upload my image with nearly one-fifth (360/1914, 18.8%) of the members engaging with the behavioral cue.

It should be noted that 2 items read a community post and complete the depression and anxiety test were hidden for some members. This is due to these elements being feature flags and some Evolution Health clients and research partners choosing to hide these features from their membership base.

Although this resulted in a lower number of members seeing these cues and having access to the course components, the percentage of members who viewed the cues and engaged with the components was noteworthy.

Table 12 lists the engagement results for checklist use and gender. Mixed effect logistic regression found that female members clicked on more checklist items than male members (z score=2.07; P=.04). Although the role of gender and engagement should be more thoroughly examined in future studies, this finding is consistent with other research on platform components [60].

Tables 13 and 14 list the engagement with checklist use and age. Members aged 41 to 50 years and 51 to 60 years were significantly less likely to click on a to-do checklist item (z score=2.1; P<.04) than members aged 18 to 30 years (z score=4.35; P<.001). No other differences were found between the age groups.

Tables 15 and 16 outline course component completion rates by arm. Using the set goals example, 17% (304/1788) of the members in the control arm, 19.8% (171/865) in arm 2, and 31.35% (600/1914) in arm 3 completed the goals exercise.

The course component with the highest completion rate was complete the depression and anxiety test, with 51.4% (60/117) in arm 3, 47% (17/36) in arm 2, and 24.1% (47/195) in arm 1. The course component with the lowest completion rate was upload my image, with 18.65% (357/1914) in arm 3, 1.9% (16/865) in arm 2, and 0.73% (13/1788) in arm 1.

As mentioned previously, the 2 items read a community post and complete the depression and anxiety test were not seen by all members. Some Evolution Health clients do not offer these 2 tools to their membership base. Consequently, there are high engagement percentages but low member use.

For technical reasons, the completion of 2 components review a worksheet and watch the getting started video were not captured in arm 1. However, about one-third of the members in arm (606/1914, 31.66%) reviewed a worksheet compared to about 2% of those (21/865, 2.4%) in arm 2, and almost half (923/1914, 48.22%) of the members in arm 3 watched the getting started video compared to about 3% of those (24/865, 2.8%) in arm 2.

Engagement with 6 of the 8 measurable components was higher in arm 2 than in arm 1 (z score=3.59; P<.001) and higher in arm 3 than in arm 1 (z score=10.91; P<.001). Engagement in 8 of the 8 components was higher in arm 3 than in arm 2 (z score=4.93; P<.001).

Overall, the use of the tips and to-do checklist items resulted in increased engagement. Members in the control arm completed an average of 1.52 course components versus 1.8 course components in arm 2 and 2.11 course components in arm 3.

The purpose of CBT is to help individuals deal with overwhelming problems; this is achieved by teaching people how to deal with negative thoughts and beliefs [61]. In traditional in-person therapy, the duration of CBT treatment depends on a variety of factors, including health disorders (eg, depression, anxiety, posttraumatic stress disorder, and sleep disorders), symptom severity, duration of therapy, practitioner availability, cost, willingness of the patient to do homework, convenience, culture, treatment settings, and the therapeutic alliance [62].

The recommended duration of CBT varies. Harvard Medical School recommends 30- to 60-minute sessions over 12 to 20 weeks [63], the United Kingdom’s National Health Services recommends 30- to 60-minute sessions once a week or every 2 weeks for 6 to 20 sessions [64], and the Mayo Clinic states that therapeutic encounters can range from 5 to 20 sessions [65].

According to the US National Institutes of Health’s National Library of Medicine, some people undergoing CBT feel much better after a few sessions, whereas others need treatment for several months [66]. In Canada, the Centre for Addiction and Mental Health notes that some people improve in 4 to 6 sessions, whereas others may need >20 sessions [67].

Structured CBT sessions are 1 of the 8 course components in the Evolution Health platform, and like the 7 other course components examined in this study, members’ engagement is optional. The courses Overcoming Depression and Overcoming Anxiety both comprised 9 sessions. On average, 41% (1851/4567) of the members chose to engage with and complete session 1 (Table 15).

If members completed a course, they received a course completion certificate (Figure 9).

Unlike the platform’s addiction courses, where some members are incentivized by customer sponsors to complete a course (eg, some human resource clients reward their employees for completing the quit smoking behavior change course), to the best of our knowledge, participants in the Overcoming Depression and Overcoming Anxiety courses were not rewarded for course completion or receiving a certificate.

Although the goal of CBT is to decrease symptoms and the duration of treatment varies widely per individual, course completion may be a poor indicator of wellness. However, as engagement is the primary measure of this study and the literature indicates that higher levels of engagement are associated with improved health outcomes, it may be beneficial to form a baseline observation for course completion rates.

Regarding course completion, 31 members in the Overcoming Depression course and 88 members in the Overcoming Anxiety course received course completion certificates. There were no statistically significant differences in the course completion rates between the 3 arms.

In addition, 256 of 4567 members of the N members received a certificate for completing a supplementary 1-session more help course, which was less intense and dealt with specific subjects such as overcoming grief, problems in relationships, or role transitions. A greater proportion of members in arm 2 received more certificates for the more help course than did the members in arm 1 (P=.02). There were no differences of interaction between gender or age groups.

The average age of the members was 43.8 years, and 87.3% (3987/4567) identified as female members. The average number of log-ins for female members was 2.83 versus 3.89 for male members. The average number of log-ins should be interpreted with caution, as in other studies on the Evolution Health population; engagement appears to follow the properties of power laws [53].

Our H1 was to analyze whether the members would engage with the tips and to-do checklist.

Regarding gender, mixed effect logistic regression found that female members clicked on more tips than male members (z score=2.58; P=.01).

Members in arm 3 completed an average of 2.7 (34%) out of the 8 course components featured on the to-do checklist. Engagement ranged from 18.81% (360/1914; upload my image) to 51.4% (55/107; complete the depression and anxiety test).

Mixed effect logistic regression found that female members clicked on more checklist items than male members (z score=2.07; P=.04). Members aged 41 to 50 years and 51 to 60 years were significantly less likely to click on a to-do checklist item (z score=2.1; P=.04) than members aged 18 to 30 years (z score=4.35; P<.001). No other differences were found between the age groups.

There were no statistically significant differences in the number of tips clicked on between directive tips in arm 2 and social proof and present bias tips in arm 3 (P=.25).

Completion rates of 6 out of the 8 course components increased in both experimental arms from baseline use in arm 1. Completion rates in arm 2 were greater than those in arm 1 (z score=3.59; P<.001), and completion rates in arm 3 were greater than those in arm 1 (z score=10.91; P<.001).

A total of 2 out of the 8 course tools (review a worksheet and watch the getting started video) did not have arm 1 baseline statistics. However, completion rates for review a worksheet in arm 2 versus arm 3 increased from 2.4% to 31.66%. The completion rate for watch the getting started video increased from 2.8% to 48.22%.

Completion rates for the course component complete CBT session 1 were slightly higher in arm 2 (376/865, 43.5%) than in arm 3 (806/1914, 42.11%). Completion rates of arm 2 were greater than those in arm 1 (z score=2.99; P=.003), and completion rates in arm 3 were greater than those in arm 1 (z score=2.91; P=.004). Completion rates of the both arms 2 and 3 were significantly higher than those in arm 1 (669/1788, 37.42%).

The highest increase in course component use was for the upload my image component, with 0.73% (13/1788) of the members uploading an image in arm 1, 1.9% (16/865) in arm 2, and 18.65% (357/1914) in arm 3. This is interesting as personalization in digital applications is now common; however, the Evolution Health platform is designed to be anonymous, and before uploading, members were advised to only upload nonidentifying images.

However, high z scores require further investigation. For example, high scores may be due to arm 3 members being more likely to complete components than arm 1 members.

In the literature, efficacy studies analyzing general CBT completion rates are rare, and this is due to complexities surrounding the delivery of CBT for different indications, symptom severity, duration of therapy, practitioner availability, cost, willingness of the patient to do homework, convenience, culture, treatment settings, and therapeutic alliance.

Owing to the nature of the medium, measuring course completion rates in digital health courses is relatively simple. However, digital health courses should not be held to different standards than traditional in-person therapy, where treatment success is often measured by the therapist by observing decreases in symptoms.

In 2005, Eysenbach [11] published The Law of Attrition, which recognized that a substantial portion of users drop out of eHealth (digital health) trials before completion and that the high dropout rate makes the efficacy of digital health programs less believable. The paper noted that researchers often compare digital health dropout rates with those of clinical drug trials.

Similar to traditional CBT, where treatment can run from a few weekly sessions to >20 sessions, the digital health literature has shown dose-response effects. Users of self-guided digital health courses interact with devices, not trained therapists. Until digital health platforms can unobtrusively and accurately detect symptom severity, measuring course completion rates will continue to be an inaccurate measure of efficacy.

As higher levels of engagement are associated with improved health outcomes [20,21] and due to the complex methodological issues associated with establishing efficacy rates in population-based digital health CBT programs with high reach, research should continue to focus on strategies that increase adherence and engagement [8-10,12].

The arm 3 home page is presently the default setting for the home pages for the Overcoming Anxiety and Overcoming Depression courses. We expect to see increases in course engagement based on the use of behavioral nudges and prompts. Ongoing data collection from all members will contribute to future research.

A strength of this study is that it was conducted in an ad libitum environment. Unlike many digital health studies, a large study population in the thousands was leveraged rather than smaller groups. As participants were not aware of participating in the experiment, this limited participant bias and the Hawthorne effect.

A limitation is that, especially due to the anonymity of members, we have no way of identifying participants or validating their demographic information. Although we have no way of knowing whether the registrants are people with depression or anxiety who are seeking help, the removal of nonparticipants should mitigate the effects on the overall results.

Some Evolution Health clients promote certain course tools, or health care professionals may direct their clients to use certain platform attributes. Therefore, the tips or to-do checklist items may not be a factor in their engagement. Second, many members may simply ignore the behavioral cues and complete certain course tools based on their own preference.

The behavioral cues related to read a community post and complete the depression and anxiety test were not seen by all members. This is due to these elements being feature flags and some Evolution Health clients choosing not to offer these tools to their membership base. Although this resulted in fewer members seeing these cues and having access to the course components, the percentage of members who viewed the cues and engaged with the components was noteworthy.

As this study was designed to form a baseline for future research, there are some methodological issues that can be explored further. For example, unlike the tips presented in arm 2, the use of tips in arm 3 may or may not have been influenced by adding the to-do checklist. Alternatively, using directive tips in arm 3, rather than social proof and present bias tips, may increase the use of the to-do checklist.

Notably, our evidence indicates that nudges and prompts increase engagement in self-guided treatment programs for depression and anxiety. On the basis of these encouraging yet preliminary results, we can proceed with more sophisticated studies that will examine more enhanced strategies designed to increase platform engagement.

As mentioned earlier, there has been scant published research regarding the implementation of behavioral economic strategies designed to increase engagement in digital health programs. At minimum, this randomized controlled trial was successful in confirming member engagement through the use of these strategies.

As recommended by Aschbacher et al [68] in a recent study involving digital mental health and dose responses, machine learning models can help enable precision by analyzing engagement patterns over time. Combined with another recent paper by Forbes et al [24] analyzing digital interventions for depression, which noted that it is important to develop standardized ways of reporting adherence and engagement so that effective comparisons across different interventions could be measured, baseline outcomes need to be established.

The baseline outcomes of this study can serve as guideposts for future studies. With machine learning models in place and with the goal of improving member outcomes and platform efficacy, Evolution Health considers the following research questions:

It cannot be assumed that the outcomes observed on the Evolution Health platform will be replicated in other environments. Further research in the combined fields of digital health and behavioral economics is required.

Members of the Evolution Health platform’s self-guided digital health courses engaged with behavioral nudges and prompts. From this preliminary analysis, it appears that both the tips and to-do checklists increased engagement in course components.

To the best of our knowledge, this is the first randomized controlled trial designed to test the implementation of behavioral nudges and prompts in web-based self-guided courses for mood disorders. The results of this study may be important because efficacy is related to increased engagement.

Owing to its novel approach, the outcomes of this study should be interpreted with caution but may be used as a guideline for future research.