This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2491) Registered Report (579) Anxiety and Stress Disorders (1081) Web-based and Mobile Health Interventions (3357) Depression and Mood Disorders; Suicide Prevention (1392) Psychological, -behavioural, -social, and/or -cultural Experiments and Interventions (53) Behavior Change (723)

Published on in Vol 8 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/52558, first published .
Nudges and Prompts Increase Engagement in Self-Guided Digital Health Treatment for Depression and Anxiety: Results From a 3-Arm Randomized Controlled Trial

Nudges and Prompts Increase Engagement in Self-Guided Digital Health Treatment for Depression and Anxiety: Results From a 3-Arm Randomized Controlled Trial

Nudges and Prompts Increase Engagement in Self-Guided Digital Health Treatment for Depression and Anxiety: Results From a 3-Arm Randomized Controlled Trial

Authors of this article:

Trevor van Mierlo1 Author Orcid Image ;   Renante Rondina2 Author Orcid Image ;   Rachel Fournier1 Author Orcid Image
Article Authors Cited by (2) Tweetations (3) Metrics Author Video

    Original Paper

    1Evolution Health, Toronto, ON, Canada

    2Rotman School of Managment, University of Toronto, Toronto, ON, Canada

    *all authors contributed equally

    Corresponding Author:

    Trevor van Mierlo, MBA, MScCH, MScBMR, GEMBA, DBA

    Evolution Health

    206-90 Eglinton Avenue East

    Toronto, ON, M4P 2Y3

    Canada

    Phone: 1 4166448476

    Email: tvanmierlo@evolutionhealth.care


    Background: Accessible and effective approaches to mental health treatment are important because of common barriers such as cost, stigma, and provider shortage. The effectiveness of self-guided treatment is well established, and its use has intensified because of the COVID-19 pandemic. Engagement remains important as dose-response relationships have been observed. Platforms such as Facebook (Meta Platform, Inc), LinkedIn (Microsoft Corp), and X Corp (formerly known as Twitter, Inc) use principles of behavioral economics to increase engagement. We hypothesized that similar concepts would increase engagement in self-guided digital health.

    Objective: This 3-arm randomized controlled trial aimed to test whether members of 2 digital self-health courses for anxiety and depression would engage with behavioral nudges and prompts. Our primary hypothesis was that members would click on 2 features: tips and a to-do checklist. Our secondary hypothesis was that members would prefer to engage with directive tips in arm 2 versus social proof and present bias tips in arm 3. Our tertiary hypothesis was that rotating tips and a to-do checklist would increase completion rates. The results of this study will form a baseline for future artificial intelligence–directed research.

    Methods: Overall, 13,224 new members registered between November 2021 and May 2022 for Evolution Health’s self-guided treatment courses for anxiety and depression. The control arm featured a member home page without nudges or prompts. Arm 2 featured a home page with a tip-of-the-day section. Arm 3 featured a home page with a tip-of-the-day section and a to-do checklist. The research protocol for this study was published in JMIR Research Protocols on August 15, 2022.

    Results: Arm 3 had significantly younger members (F2,4564=40.97; P<.001) and significantly more female members (χ24=92.2; P<.001) than the other 2 arms. Control arm members (1788/13,224, 13.52%) completed an average of 1.5 course components. Arm 2 members (865/13,224, 6.54%) clicked on 5% of tips and completed an average of 1.8 course components. Arm 3 members (1914/13,224, 14.47%) clicked on 5% of tips, completed 2.7 of 8 to-do checklist items, and completed an average of 2.11 course components. Completion rates in arm 2 were greater than those in arm 1 (z score=3.37; P<.001), and completion rates in arm 3 were greater than those in arm 1 (z score=12.23; P<.001). Engagement in all 8 components in arm 3 was higher than that in arm 2 (z score=1.31; P<.001).

    Conclusions: Members engaged with behavioral nudges and prompts. The results of this study may be important because efficacy is related to increased engagement. Due to its novel approach, the outcomes of this study should be interpreted with caution and used as a guideline for future research in this nascent field.

    International Registered Report Identifier (IRRID): RR2-10.2196/37231

    JMIR Form Res 2024;8:e52558

    doi:10.2196/52558

    Keywords

    behavioral economics; digital health; attrition; engagement; nudges; depression; anxiety; mood disorders 


    Background

    Since its earliest use in the mid-1990s, digital health promised personalized treatments that patients could access from home. It was anticipated that treatment would have a broad reach, resulting in improved health outcomes and decreased costs [1-3]. Over the past 2 decades, and with increasing consistency, research examining the efficacy of self-guided digital health interventions show evidence of efficacy, especially for those with mental health concerns [4-7].

    Although digital health interventions appear to be effective, poor adherence and lack of compliance have remained consistent patterns in research [8-10]. This pattern was first recognized in 2005 and coined The Law of Attrition [11].

    As early as 2009, systematic reviews identified poor adherence and lack of compliance as engagement issues that required attention [12]. These issues persist, as demonstrated by a recent meta-analytic review on digital interventions for depression, which showed efficacy but highlighted compliance as a major challenge [13].

    Adherence and compliance are complex and rooted in several systemic and individual factors [14-19]. However, it is an important topic as evidence indicates a dose-response relationship, and higher levels of engagement are associated with improved health outcomes [20,21].

    Moreover, digital health interventions are becoming increasingly common and accessible. Patients’ use of, and trust, in these interventions has been intensified by the COVID-19 pandemic. The use of self-guided digital health interventions for mental health concerns is growing [22,23] because of access barriers such as high cost, stigma, and lack of access due to a shortage of professionals who can meet this growing demand [24,25].

    How do we increase engagement in digital health programs to maximize their efficacy?

    Behavioral Economics

    Overview

    Behavioral economics leverages psychological experimentation to develop theories about human decision-making. The field has identified a range of unconscious biases around how people think and feel [26,27].

    The utility of behavioral economics is vast. Digital health has leveraged the discipline to investigate how people use digital health programs and to gain insights into the characteristics of people who use them. Several digital health studies have investigated the use of these strategies, including cooperative games and incentives [28], gamification [29,30], serious games [31,32], and positive behavioral support [33,34].

    Our Use of Behavioral Economics

    In our study, we examined the effectiveness of the nudge theory and behavioral prompts in 2 ad libitum self-guided digital behavior change courses.

    Nudge Theory

    Nudge theory, popularized in the 2008 book Nudge: Improving Decisions About Health, Wealth, and Happiness [27], leverages indirect, positive suggestions to influence decision-making and behavior.

    There is a lack of quality research analyzing the use of nudges in digital health. A 2019 scoping review examined the use of nudges in both web-based and real-world settings in physical activity interventions [35]. Of the 35 publications reviewed, 8 were web-based studies. The authors concluded that although nudging may be an effective approach to promote physical activity, there are large gaps in research, and further studies that are explicitly based on nudge insights are needed.

    A 2020 editorial in Personalized Medicine addressed the meaningful adoption of nudges in digital health [36]. The authors acknowledged that using nudges in digital health interventions is rare and advocated for the use of nudges to promote positive behavior change.

    Behavioral Prompts

    In applied behavioral analysis, behavioral prompts are cues specifically designed to encourage a specific task [34]. In this study, we used 2 types of behavioral prompts anchored in the nudge theory: daily tips and a to-do checklist (Table 1).

    Table 1. Example nudges and prompts.
    Delivery formatContent typeExample text from our study
    TipDirective contentExpress yourself by uploading your image
    TipSocial proofMany members have similar goals as you. Reviewing other members’ goals can help you reach your goal.
    TipPresent biasFeel better sooner by learning from others. Read what others have posted on the community.
    To-do checklistDirective contentWatch the getting started video
    Directive, Social Proof, and Present Bias Tips

    Directive content, as the name suggests, offers concrete suggestions to members [37]. These types of tips are brief and instructional.

    Social proof is derived from the behavioral economics concept where we tend to copy the actions of those around us. These tips speak of our tendency to be swayed by other people’s choices, which we attempt to mirror [38,39].

    Present bias is the inclination to prefer a smaller present reward now over a larger reward later. These tips encourage users to perform a task that provides an immediate benefit [40,41].

    Our Use of Behavioral Prompts and Technical Functionality of Tips and to-Do Checklist Items

    An example of a course tool in both Overcoming Anxiety and Overcoming Depression courses is goal setting. Goal setting is an important component of cognitive behavioral therapy (CBT). However, in a self-guided environment, many members are unsure of how to set personal goals. Evolution Health encourages members to review goals set by others as examples.

    Figure 1 shows an example of a tip of the day that encourages members to review the goals of other members.

    Figure 2 shows an example of the completed to-do checklist item, set goals. In this figure, the item is marked complete by a check box. This signifies that a member has clicked on the item and visited the page.

    By clicking on either the tip (Figure 2) or the to-do checklist item (Figure 3), the member is brought to member goals, a course component that allows members to browse various goals set by other members (Figure 3).

    Engagement experiments in popular non–health care digital platforms are common. Although they are scientific in nature, they are not typically published, as they are conducted within private companies.

    For example, social network sites such as Facebook (Meta Platforms, Inc), LinkedIn (Microsoft Corp), and X Corp. (formerly known as Twitter) generate revenue based on ad revenue derived from page views and the time members spend on their site. In a 2015 presentation, it was revealed that LinkedIn had >400 controlled experiments being conducted per day [42]. Similar studies with an ad libitum population are required for digital health, and this study is an attempt to fill this gap.

    We have not observed sufficient evidence in the literature to determine whether nudges and prompts can be strategically applied to increase engagement with and decrease attrition in the courses for depression and anxiety [43]. Furthermore, because of the nascent state of behavioral economics within digital health, we did not find any quantitative benchmarks that would help us determine whether our use of nudges and prompts was successful.

    ‎
    Figure 1. Present bias tip: review other members’ goals.
    ‎
    Figure 2. Completed to-do checklist item, set goals.
    ‎
    Figure 3. Other members’ goals.

    Objectives

    Our primary hypothesis (H1) was that members would engage with the tips and to-do checklist. Engagement of the members was determined by the percentage of tips that were clicked on compared to the number shown. Because there is no prior literature, we first needed to establish baseline data. Data were reported as observational and served as a benchmark for future studies.

    Our secondary hypothesis was that members would prefer to engage with directive tips in arm 2 versus social proof or present bias tips in arm 3. To assess the preference for engagement, we compared the number of tips clicked in arm 2 versus those clicked in arm 3.

    Our tertiary hypothesis was that the addition of tips and a to-do checklist would increase completion rates with course tools. Increased completion rates were determined by comparing completion rates in arms 2 and 3 with those in arm 1 (control group).

    In each hypothesis, we assessed whether engagement was influenced by gender or age.


    Overview

    The digital health platform used in this study was managed by Evolution Health. Evolution Health is an evidence-based, self-guided digital health platform that features courses and brief interventions based on behavior change techniques including CBT, stages of change, structured relapse prevention, normative feedback, and harm reduction.

    The platform offers interactive courses and quizzes for people with mental health, addiction, and obesity issues. The platform contains a moderated community based on social cognitive theory.

    Memberships are available to individuals who register through the organization’s free-to-consumer program and white-label instances that are licensed by employers, insurance companies, employee assistance programs, educational institutions, nonprofit organizations, for-profit health care organizations, and individual therapists.

    The research protocol for this study was published in JMIR Research Protocols on August 15, 2022 [44]. The International Registered Report Identifier is DERR1-10.2196/37231.

    The Interventions

    The 2 interventions in this study contain self-guided interactive behavior change treatment courses based on best practices, and both have been examined extensively in the literature [9,45-53].

    The 2 interventions have undergone several iterations. For example, Overcoming Anxiety was the first intervention noted in TheLaw of Attrition (previously known as The Panic Center) paper by Eysenbach [11]. In that iteration, the course contained a tunnel design with 12 successive sessions. The course now has a gamified free-form matrix design, among other technical and usability enhancements.

    Table 2 outlines each course’s current theoretical construct and evidence base. Table 3 outlines the main course components.

    Table 2. Theoretical constructs and evidence base.
    Theoretical constructOvercoming Depression courseOvercoming Anxiety course
    Brief intervention✓✓
    Cognitive behavioral therapy✓✓
    Gamification✓✓
    Health belief model✓✓
    Motivational interviewing✓✓
    Normative feedback✓✓
    Social cognitive theory✓✓
    Targeting and tailoring✓✓
    Table 3. Main course components.
    Course componentOvercoming Depression courseOvercoming Anxiety course
    Avatar upload✓✓
    Course completion certificate✓✓
    Course diary (mood tracker or symptom tracker)✓✓
    Course worksheets✓✓
    Gamified CBTa course✓✓
    Getting started video✓✓
    Goals exercise✓✓
    Moderated community✓✓
    Private messaging✓✓
    Statistics interface (for corporate clients)✓✓
    Tailored depression and anxiety test✓✓
    Therapist interface✓✓

    aCBT: cognitive behavioral therapy.

    Ethical Considerations

    At registration, all members endorsed a checkbox to confirm that they consented to have their nonidentifiable data used for research purposes and approved the platform’s privacy policy. The participants did not receive compensation for their involvement. The platform is available in several languages, and the English language privacy policy and terms of use are presented in Multimedia Appendix 1. All data collection policies and procedures adhered to the international privacy guidelines [54-56] and the Helsinki Declaration of 1975 [57].

    This study was conducted on self-guided treatment for depression and anxiety, but it did not measure clinical outcomes. Although the study participants were randomly assigned to a control or intervention arm, the study was a randomized controlled trial and not a randomized clinical trial [58]. This study did not assess whether the study participants’ engagement with course tools decreased the depressive symptoms, severity of panic attacks, or frequency of panic attacks.

    As described earlier, evidence indicates that higher levels of engagement are associated with improved health outcomes, and the literature observes dose-response relationships. However, any clinical outcomes related to the engagement strategies used in this study will need to be tested in future research.

    As the study was based on unidentifiable data and no clinical measures were tested, it was deemed exempt from further review by the Evolution Health Institutional Review Board (IRB#000014034, FWA00033737).

    Power and Sample Size

    The study was designed to have a power of 0.95, indicating a 95% probability of correctly detecting a statistically significant difference in engagement with tips between the 2 treatment groups (arm 2 vs arm 3). On the basis of a 2-tailed t test with a conventional α level (.05) for statistical significance, we required a sample of 1302 members with 651 in each group to detect a small effect size of Cohen d=0.2. Refer to Multimedia Appendix 2 for the CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklist [59].

    Randomization

    During the registration process, new members were assigned to 1 of the 3 arms using a random number generator (Figure 4). Randomization was conducted using simple randomization.

    ‎
    Figure 4. Recruitment process.

    Intervention Groups

    Arm 1

    Members randomized to arm 1 were presented with a dashboard that did not contain behavioral nudges. Figure 5 presents a screenshot of the arm 1 dashboard for a member who chose to engage with the depression course.

    ‎
    Figure 5. Member home page for arm 1.
    Arm 2

    Members randomized to arm 2 were presented with a dashboard that contained a tip-of-the-day section containing directive content. The randomization strategy for the 31 directive tips was randomization without replacement.

    Figure 6 presents a screenshot of the arm 2 dashboard for a member who chose to engage with the depression course.

    ‎
    Figure 6. Member home page for arm 2.
    Arm 3

    Members randomized to arm 3 were presented with 2 sections that contained nudges. The first was a tip-of-the-day section containing social proof and present bias cues. At each log-in, members saw a new tip. The randomization strategy for the tips was randomization without replacement. There were 15 social proof tips and 15 present bias tips.

    In addition to the tip of the day, arm 3 featured a to-do checklist that listed 8 course components. When a member clicked on a component, they were brought to the exercise. As mentioned previously, if a member clicks on a to-do checklist item, it is marked complete with a check mark.

    Figure 7 presents a screenshot of the arm 2 dashboard for a member who chose to engage with the depression course.

    ‎
    Figure 7. Member home page for arm 3.

    Data Collection

    A custom data collection interface and reporting mechanism were developed by Evolution Health. Age and gender data were collected at registration or through a secure sign on with various white-label instances. Data were collected for each member who was randomized into the experiment. The course components promoted by tips and to-do checklist items are listed in Textbox 1. Members who participated in both courses were not counted twice.

    The following behaviors were tracked in the custom database for each tip and to-do checklist item that was randomly presented to a member:

    • If the nudge was shown
    • If the nudge was clicked on
    • If a member completed the course component described in the tip or to-do checklist item.

    It was possible for a single member to participate in both courses. However, the study design was to test the behavioral prompts, not the courses. The participants were randomized to an intervention arm, in which behavioral nudges and prompts were consistent across courses.

    Textbox 1. Course components tracked for engagement.

    Action code and course component

    • 1 (item in the to-do checklist): Uploading a personal image (avatar) to their profile
    • 2 (item in the to-do checklist): Completing cognitive behavioral therapy session 1
    • 3 (item in the to-do checklist): Use of the program diary (mood tracker or symptom tracker)
    • 4 (item in the to-do checklist): Read a community post
    • 5 (item in the to-do checklist): Review a worksheet
    • 6 (item in the to-do checklist): Set personal goals
    • 7 (item in the to-do checklist): Complete the depression and anxiety test
    • 8 (item in the to-do checklist): Watch the getting started video
    • 9: Review another member’s profile
    • 10: Post in the community
    • 11: Read other member’s goals
    • 12: Give a community member a “thumbs up”
    • 13: Encourage a community member by clicking their “show support” icon
    • 14: Private message a community moderator

    Participants

    As there were no barriers to registration and many new members registered with the platform for purposes other than treatment, we removed those who registered but did not return to the intervention (nonparticipants) from the analysis.

    Data Analyses

    All data were analyzed using mixed effect logistic regression with members as a random variable. Mixed effect logistic regression was conducted using the glmer() function from the lme4 package in R software (R Foundation for Statistical Computing), with the default optimizer Bound Optimization BY Quadratic Approximation, a derivative-free optimization algorithm used for problems with bound constraints.

    For the tip-of-the-day section, the outcome variable was whether a user clicked on a tip presented to them (no or yes), and the predictors were arm (2 or 3), gender (female or male), and age group (18 to 30 years to >60 years).

    For the to-do checklist items, the outcome variable was whether a user clicked on a checklist item (no or yes), and the predictors were gender and age group.

    For the course components, the outcome variable was whether a user completed a component (no or yes), and the predictors were arm (1, 2, or 3), gender, and age group.


    Overview

    Between November 2021 and May 2022, data were collected from new members who self-registered for Evolution Health’s self-guided treatment program for anxiety and depression. All members were randomized into 1 of the 3 arms.

    First, members with test accounts and unauthenticated accounts were removed from the data set, resulting in a population of 13,224 members. Then, members whose accounts were missing demographic data (376/13,224, 2.84%) were removed from the data set, followed by members aged <18 (129/13,224, 0.97%), resulting in a population of 12,719 members.

    Finally, of the 13,224 members, 8567 (65.46%) with nonparticipant accounts were removed, resulting in a study population of 4567 (34.53%). Of the 4567 members, 1788 (39.15%) were randomized into arm 1, 865 (18.94%) into arm 2, and 1914 (41.9%) into arm 3 (Figure 8).

    ‎
    Figure 8. Study population.

    Demographic Characteristics

    The average age of the members in arms 1, 2, and 3 was 44.4 (SD 10.69), 45.4 (SD 10.62), and 41.8 (SD 11.17), respectively (Table 4). The 51 to 60 years age group was the most populous age category in each of the arms (arm 1: 777/1788, 43.45%; arm 2: 420/865, 48.6%; and arm 3: 586/1914, 30.61%). The members of arm 3 were somewhat younger (F2,4564=40.97; P<.001) than the other 2 arms.

    Regarding gender (Table 5), most members identified as women (arm 1: 1466/1788, 81.99%; arm 2: 751/865, 86.8%; and arm 3: 1770/1914, 92.48%). Few members identified as transgender individuals, nonbinary or third gender individuals, or preferred not to say (arm 1: 5/1788, 0.28%; arm 2: 1/865, 1.2%; and arm 3: 2/1914, 0.1%). Arm 3 had more female members (χ24=92.21; P<.001) than the other 2 arms.

    During the study period, there were 13,510 total log-ins, with initial registration considered as 1 log-in (Table 6). The average number of log-ins for male members was 3.89 and female members was 2.83. The average visit duration (AVD) for members who engaged with depression and anxiety course tools ranged from 6 minutes and 45 seconds, with 7.65 pages viewed, to 24 minutes and 21 seconds, with 12.56 pages viewed.

    Table 4. Age breakdown per arm (N=4567).
    Age range (years)Arm 1 (n=1788), n (%)Arm 2 (n=865), n (%)Arm 3 (n=1914), n (%)Total age category, n (%)
    18 to 30231 (12.91)108 (12.5)368 (19.22)707 (15.48)
    31 to 40393 (21.97)159 (18.4)492 (25.7)1044 (22.86)
    41 to 50329 (18.4)147 (17.0)415 (21.68)891 (19.51)
    51 to 60777 (43.45)420 (48.6)586 (30.61)1783 (39.04)
    >6058 (3.24)32 (3.7)53 (2.76)142 (3.11)
    Table 5. Gender categories per arm (N=4567).
    GenderArm 1 (n=1788), n (%)Arm 2 (n=856), n (%)Arm 3 (n=1914), n (%)Total, n (%)
    Female members1466 (81.99)751 (86.8)1770 (92.48)3987 (87.3)
    Male members317 (17.72)113 (13.1)142 (7.42)572 (12.52)
    Other gender identity5 (0.28)1 (1.2)2 (0.1)8 (0.17)
    Table 6. Number of log-ins by gender category.
    GenderMaximum number of log-ins, nMinimum number of log-ins, nAverage number of log-ins, mean (SD)
    Female members27612.83 (6.04)
    Male members24313.89 (12.13)
    Other gender identity612.6 (1.85)
    Range27612.38 (7.09)

    Tip of the Day

    In arm 2, there were 31 revolving tips, and in arm 3, there were 30 revolving tips. Therefore, the probability of a member assigned to the arm 2 seeing a specific tip at least once is .088, and the probability of a member assigned to the arm 3 seeing a specific tip at least once is .091.

    A total of 11,431 tips were displayed, of which 564 (4.93%) were clicked on (Table 7). In arm 2, 3622 tips were shown, of which 190 (5.24%) were clicked on. In arm 3, 7809 tips were shown, of which 374 (4.79%) were clicked on. Mixed effect logistic regression revealed no statistically significant differences between the 2 arms regarding the number of tips clicked on (P=.25).

    Table 7. Tips shown and clicked on (N=11,431).

    		Arm 2 (n=3622), n (%)Arm 3 (n=7809), n (%)Total, n (%)
    Tips shown3622 (100)7809 (100)11,431 (100)
    Tips clicked on190 (5.24)374 (4.79)564 (4.93)

    In arm 2, female members clicked on 171 (5.82%) tips out of the 2937 tips shown to them and male members clicked on 19 (2.8%) tips out of the 682 tips shown to them, whereas in arm 3, females members clicked on 350 (4.94%) out of the 7081 tips shown to them and male members clicked on 24 (3.3%) out of the 722 tips shown to them (Table 8).

    There were no statistically significant differences between number of tips clicked on between age groups (Table 9).

    Mixed effect logistic regression (Table 10) revealed that female members clicked on more tips than male members (z score=2.58; P=.01). Although the role of gender and engagement should be more thoroughly examined in future studies, this finding is consistent with other research on platform components [60].

    There were no substantial differences between age groups.

    Table 8. Tips shown and clicked on by gender category (N=10,018).

    		Arm 2, n (%)Arm 3, n (%)Total, n (%)
    Tips shown to female members2937 (100)7081 (100)10,018 (100)

    		Tips clicked on171 (5.82)350 (4.94)521 (5.20)
    Tips shown to male members682 (100)722 (100)1404 (100)

    		Tips clicked on19 (2.8)24 (3.3)43 (3.06)
    Tips shown to members of other gender identities3 (100)6 (100)9 (100)

    		Tips clicked on0 (0)0 (0)0 (0)
    Table 9. Tip engagement by age group.
    Age groupArm 2, n (%)Arm 3, n (%)Total, n (%)
    Tips shown to members aged 18 to 30 years449 (100)1393 (100)1842 (100)

    		Tips clicked on26 (5.8)67 (4.81)93 (5.05)
    Tips shown to members aged 31 to 40 years669 (100)2130 (100)2799 (100)

    		Tips clicked on42 (6.3)110 (5.16)152 (5.43)
    Tips shown to members aged 41 to 50 years677 (100)1818 (100)2495 (100)

    		Tips clicked on36 (5.3)95 (5.23)131 (5.25)
    Tips shown to members aged 51 to 60 years1493 (100)2260 (100)3753 (100)

    		Tips clicked on73 (4.89)91 (4.16)164 (4.37)
    Tips shown to members aged >60 years334 (100)208 (100)542 (100)

    		Tips clicked on13 (3.9)11 (5.3)24 (4.4)
    Table 10. Summary table for tip-of-the-day section.
    PredictorsORa (95% CI)SEz scoreP value
    Arm 30.87 (0.69-1.10)0.1−1.15.25
    Gender (male members)0.59 (0.39-0.88)0.12−2.58.01
    Age group (years)

    		31 to 401.08 (0.78-1.49)0.180.44.66

    		41 to 500.99 (0.71-1.40)0.17−0.03.98

    		51 to 600.82 (0.59-1.13)0.13−1.22.22

    		>601.2 (0.66-2.20)0.370.6.55

    aOR: odds ratio.

    To-Do Checklist

    As outlined in Textbox 1 and illustrated in Figure 5, the to-do checklist contained 8 items. The members of arm 3 completed an average of 2.7 (34%) out of 8 course components.

    The checklist item with the highest engagement was complete the depression and anxiety test, which was completed by 51.4% (55/107 of the members; Table 11). The second most popular item was watch the getting started video, with 48.2% (923/1914) of the members engaging with the behavioral cue. The checklist item with lowest engagement was upload my image with nearly one-fifth (360/1914, 18.8%) of the members engaging with the behavioral cue.

    It should be noted that 2 items read a community post and complete the depression and anxiety test were hidden for some members. This is due to these elements being feature flags and some Evolution Health clients and research partners choosing to hide these features from their membership base.

    Although this resulted in a lower number of members seeing these cues and having access to the course components, the percentage of members who viewed the cues and engaged with the components was noteworthy.

    Table 12 lists the engagement results for checklist use and gender. Mixed effect logistic regression found that female members clicked on more checklist items than male members (z score=2.07; P=.04). Although the role of gender and engagement should be more thoroughly examined in future studies, this finding is consistent with other research on platform components [60].

    Tables 13 and 14 list the engagement with checklist use and age. Members aged 41 to 50 years and 51 to 60 years were significantly less likely to click on a to-do checklist item (z score=2.1; P<.04) than members aged 18 to 30 years (z score=4.35; P<.001). No other differences were found between the age groups.

    Table 11. To-do checklist item clicked on (N=1914).
    To-do checklist itemClicked, n (%)
    Upload my image360 (18.81)
    Complete CBTa session 1580 (30.3)
    Use of the mood tracker or symptom tracker616 (32.18)
    Read a community post24 (22.4)
    Review a worksheet606 (31.66)
    Set goals605 (31.61)
    Complete the depression and anxiety test55 (51.4)
    Watch the getting started video923 (48.22)

    aCBT: cognitive behavioral therapy.

    Table 12. Engagement with the to-do checklist items by gender (N=1914).
    To-do checklist itemWomen (n=1770, 92.48%), n (%)Men (n=142, 7.42%), n (%)
    Upload my image338 (19.1)21 (14.79)
    Complete CBTa session 1549 (31.02)31 (21.83)
    Use the mood tracker or symptom tracker572 (32.32)43 (30.28)
    Read a community post20 (29.4)4 (10.8)
    Review a worksheet572 (32.32)33 (23.24)
    Set goals566 (31.98)38 (26.76)
    Complete the depression and anxiety test32 (47.0)21 (56.8)
    Watch the getting started video862 (48.7)60 (42.25)

    aCBT: cognitive behavioral therapy.

    Table 13. Engagement with the to-do checklist items by age (N=1914).
    To-do checklist itemAge group (years; n=1914, 100%), n (%)

    		18 to 3031 to 4041 to 5051 to 60>60
    Upload my image5 (31.25)73 (20.74)100 (20.33)96 (23.13)80 (13.65)
    Complete CBTa session 18 (50)121 (34.38)162 (32.93)147 (35.42)129 (22.01)
    Use the mood tracker or symptom tracker7 (43.75)120 (34.09)163 (33.13)157 (37.83)155 (26.45)
    Read a community post1 (50)11 (30.56)2 (7.14)5 (22.73)4 (26.67)
    Review a worksheet8 (50)124 (35.23)158 (32.11)155 (37.35)144 (24.57)
    Set goals3 (18.75)119 (33.81)158 (32.11)157 (37.83)151 (25.77)
    Complete the depression and anxiety test2 (100)16 (44.44)13 (46.43)12 (54.55)10 (66.67)
    Watch the getting started video9 (56.25)165 (46.88)224 (45.53)244 (58.8)252 (43)

    aCBT: cognitive behavioral therapy.

    Table 14. Summary table for the to-do checklist items.
    PredictorsORa (95% CI)SEz scoreP value
    Gender (male members)0.8 (0.65-0.99)0.08−2.07.04
    Age group (years)

    		31 to 400.89 (0.76-1.05)0.07−1.4.02

    		41 to 500.84 (0.71-0.99)0.07−2.1.04

    		51 to 600.7 (0.59-0.82)0.06−4.35<.001

    		>600.73 (0.52-1.03)0.13−1.79.07

    aOR: odds ratio.

    Comparison of Course Components Completed by Arm

    Tables 15 and 16 outline course component completion rates by arm. Using the set goals example, 17% (304/1788) of the members in the control arm, 19.8% (171/865) in arm 2, and 31.35% (600/1914) in arm 3 completed the goals exercise.

    Table 15. Course components completed by arm (N=4567).
    Course componentArm 1 (control; n=1788), n (%)Arm 2 (RCa; n=865), n (%)Arm 3 (PBb, SPc, and TDd; n=1914), n (%)
    Upload my image13 (0.73)16 (1.85)357 (18.65)
    Complete CBTe session 1669 (37.42)376 (43.47)806 (42.11)
    Use the mood tracker or symptom tracker366 (20.47)224 (25.9)629 (32.86)
    Read a community post46 (39.66)13 (43.33)51 (47.66)
    Review a worksheetN/Af21 (2.4)606 (31.66)
    Set goals304 (17)171 (19.77)600 (31.35)
    Complete the depression and anxiety test47 (24.14)17 (46.67)60 (51.4)
    Watch the getting started videoN/A24 (2.8)923 (48.22)

    aRC: randomized control.

    bPB: present bias.

    cSP: social proof.

    dTD: to-do checklist.

    eCBT: cognitive behavioral therapy.

    fN/A: not applicable.

    Table 16. Summary table for course completion by arm.
    PredictorsORa (95% CI)SEz scoreP value
    Arm 21.31 (1.13-1.52)0.13.59<.001
    Arm 31.93 (1.71-2.17)0.1210.91<.001
    Gender (male members)1.05 (0.90-1.23)0.090.6.55
    Age group (years)

    		31 to 400.97 (0.82-1.15)0.08−0.32.75

    		41 to 501.12 (0.94-1.33)0.11.26.21

    		51 to 600.81 (0.69-0.94)0.06−2.7.007

    		>600.76 (0.54-1.05)0.13−1.66.10

    aOR: odds ratio.

    The course component with the highest completion rate was complete the depression and anxiety test, with 51.4% (60/117) in arm 3, 47% (17/36) in arm 2, and 24.1% (47/195) in arm 1. The course component with the lowest completion rate was upload my image, with 18.65% (357/1914) in arm 3, 1.9% (16/865) in arm 2, and 0.73% (13/1788) in arm 1.

    As mentioned previously, the 2 items read a community post and complete the depression and anxiety test were not seen by all members. Some Evolution Health clients do not offer these 2 tools to their membership base. Consequently, there are high engagement percentages but low member use.

    For technical reasons, the completion of 2 components review a worksheet and watch the getting started video were not captured in arm 1. However, about one-third of the members in arm (606/1914, 31.66%) reviewed a worksheet compared to about 2% of those (21/865, 2.4%) in arm 2, and almost half (923/1914, 48.22%) of the members in arm 3 watched the getting started video compared to about 3% of those (24/865, 2.8%) in arm 2.

    Engagement with 6 of the 8 measurable components was higher in arm 2 than in arm 1 (z score=3.59; P<.001) and higher in arm 3 than in arm 1 (z score=10.91; P<.001). Engagement in 8 of the 8 components was higher in arm 3 than in arm 2 (z score=4.93; P<.001).

    Overall, the use of the tips and to-do checklist items resulted in increased engagement. Members in the control arm completed an average of 1.52 course components versus 1.8 course components in arm 2 and 2.11 course components in arm 3.

    CBT Course Completion Rates

    The purpose of CBT is to help individuals deal with overwhelming problems; this is achieved by teaching people how to deal with negative thoughts and beliefs [61]. In traditional in-person therapy, the duration of CBT treatment depends on a variety of factors, including health disorders (eg, depression, anxiety, posttraumatic stress disorder, and sleep disorders), symptom severity, duration of therapy, practitioner availability, cost, willingness of the patient to do homework, convenience, culture, treatment settings, and the therapeutic alliance [62].

    The recommended duration of CBT varies. Harvard Medical School recommends 30- to 60-minute sessions over 12 to 20 weeks [63], the United Kingdom’s National Health Services recommends 30- to 60-minute sessions once a week or every 2 weeks for 6 to 20 sessions [64], and the Mayo Clinic states that therapeutic encounters can range from 5 to 20 sessions [65].

    According to the US National Institutes of Health’s National Library of Medicine, some people undergoing CBT feel much better after a few sessions, whereas others need treatment for several months [66]. In Canada, the Centre for Addiction and Mental Health notes that some people improve in 4 to 6 sessions, whereas others may need >20 sessions [67].

    Structured CBT sessions are 1 of the 8 course components in the Evolution Health platform, and like the 7 other course components examined in this study, members’ engagement is optional. The courses Overcoming Depression and Overcoming Anxiety both comprised 9 sessions. On average, 41% (1851/4567) of the members chose to engage with and complete session 1 (Table 15).

    If members completed a course, they received a course completion certificate (Figure 9).

    Unlike the platform’s addiction courses, where some members are incentivized by customer sponsors to complete a course (eg, some human resource clients reward their employees for completing the quit smoking behavior change course), to the best of our knowledge, participants in the Overcoming Depression and Overcoming Anxiety courses were not rewarded for course completion or receiving a certificate.

    Although the goal of CBT is to decrease symptoms and the duration of treatment varies widely per individual, course completion may be a poor indicator of wellness. However, as engagement is the primary measure of this study and the literature indicates that higher levels of engagement are associated with improved health outcomes, it may be beneficial to form a baseline observation for course completion rates.

    Regarding course completion, 31 members in the Overcoming Depression course and 88 members in the Overcoming Anxiety course received course completion certificates. There were no statistically significant differences in the course completion rates between the 3 arms.

    In addition, 256 of 4567 members of the N members received a certificate for completing a supplementary 1-session more help course, which was less intense and dealt with specific subjects such as overcoming grief, problems in relationships, or role transitions. A greater proportion of members in arm 2 received more certificates for the more help course than did the members in arm 1 (P=.02). There were no differences of interaction between gender or age groups.

    ‎
    Figure 9. Course completion certificate.

    This 3-arm randomized controlled trial tested whether behavioral nudges and prompts could be successfully applied within 2 self-guided treatment digital health programs for anxiety and depression.

    Demographic Characteristics

    The average age of the members was 43.8 years, and 87.3% (3987/4567) identified as female members. The average number of log-ins for female members was 2.83 versus 3.89 for male members. The average number of log-ins should be interpreted with caution, as in other studies on the Evolution Health population; engagement appears to follow the properties of power laws [53].

    H1 Summary: Engagement With Behavioral Nudges and Prompts

    Our H1 was to analyze whether the members would engage with the tips and to-do checklist.

    Regarding gender, mixed effect logistic regression found that female members clicked on more tips than male members (z score=2.58; P=.01).

    Members in arm 3 completed an average of 2.7 (34%) out of the 8 course components featured on the to-do checklist. Engagement ranged from 18.81% (360/1914; upload my image) to 51.4% (55/107; complete the depression and anxiety test).

    Mixed effect logistic regression found that female members clicked on more checklist items than male members (z score=2.07; P=.04). Members aged 41 to 50 years and 51 to 60 years were significantly less likely to click on a to-do checklist item (z score=2.1; P=.04) than members aged 18 to 30 years (z score=4.35; P<.001). No other differences were found between the age groups.

    Secondary Hypothesis Summary: Members’ Preference of Directive Tips in Arm 2 or Social Proof or Present Bias Tips in Arm 3

    There were no statistically significant differences in the number of tips clicked on between directive tips in arm 2 and social proof and present bias tips in arm 3 (P=.25).

    Tertiary Hypothesis Summary: Tips and to-Do Checklist

    Completion rates of 6 out of the 8 course components increased in both experimental arms from baseline use in arm 1. Completion rates in arm 2 were greater than those in arm 1 (z score=3.59; P<.001), and completion rates in arm 3 were greater than those in arm 1 (z score=10.91; P<.001).

    A total of 2 out of the 8 course tools (review a worksheet and watch the getting started video) did not have arm 1 baseline statistics. However, completion rates for review a worksheet in arm 2 versus arm 3 increased from 2.4% to 31.66%. The completion rate for watch the getting started video increased from 2.8% to 48.22%.

    Completion rates for the course component complete CBT session 1 were slightly higher in arm 2 (376/865, 43.5%) than in arm 3 (806/1914, 42.11%). Completion rates of arm 2 were greater than those in arm 1 (z score=2.99; P=.003), and completion rates in arm 3 were greater than those in arm 1 (z score=2.91; P=.004). Completion rates of the both arms 2 and 3 were significantly higher than those in arm 1 (669/1788, 37.42%).

    The highest increase in course component use was for the upload my image component, with 0.73% (13/1788) of the members uploading an image in arm 1, 1.9% (16/865) in arm 2, and 18.65% (357/1914) in arm 3. This is interesting as personalization in digital applications is now common; however, the Evolution Health platform is designed to be anonymous, and before uploading, members were advised to only upload nonidentifying images.

    However, high z scores require further investigation. For example, high scores may be due to arm 3 members being more likely to complete components than arm 1 members.

    CBT Course Completion Rates

    In the literature, efficacy studies analyzing general CBT completion rates are rare, and this is due to complexities surrounding the delivery of CBT for different indications, symptom severity, duration of therapy, practitioner availability, cost, willingness of the patient to do homework, convenience, culture, treatment settings, and therapeutic alliance.

    Owing to the nature of the medium, measuring course completion rates in digital health courses is relatively simple. However, digital health courses should not be held to different standards than traditional in-person therapy, where treatment success is often measured by the therapist by observing decreases in symptoms.

    In 2005, Eysenbach [11] published The Law of Attrition, which recognized that a substantial portion of users drop out of eHealth (digital health) trials before completion and that the high dropout rate makes the efficacy of digital health programs less believable. The paper noted that researchers often compare digital health dropout rates with those of clinical drug trials.

    Similar to traditional CBT, where treatment can run from a few weekly sessions to >20 sessions, the digital health literature has shown dose-response effects. Users of self-guided digital health courses interact with devices, not trained therapists. Until digital health platforms can unobtrusively and accurately detect symptom severity, measuring course completion rates will continue to be an inaccurate measure of efficacy.

    As higher levels of engagement are associated with improved health outcomes [20,21] and due to the complex methodological issues associated with establishing efficacy rates in population-based digital health CBT programs with high reach, research should continue to focus on strategies that increase adherence and engagement [8-10,12].

    Practical Implications

    The arm 3 home page is presently the default setting for the home pages for the Overcoming Anxiety and Overcoming Depression courses. We expect to see increases in course engagement based on the use of behavioral nudges and prompts. Ongoing data collection from all members will contribute to future research.

    Strengths and Limitations

    A strength of this study is that it was conducted in an ad libitum environment. Unlike many digital health studies, a large study population in the thousands was leveraged rather than smaller groups. As participants were not aware of participating in the experiment, this limited participant bias and the Hawthorne effect.

    A limitation is that, especially due to the anonymity of members, we have no way of identifying participants or validating their demographic information. Although we have no way of knowing whether the registrants are people with depression or anxiety who are seeking help, the removal of nonparticipants should mitigate the effects on the overall results.

    Some Evolution Health clients promote certain course tools, or health care professionals may direct their clients to use certain platform attributes. Therefore, the tips or to-do checklist items may not be a factor in their engagement. Second, many members may simply ignore the behavioral cues and complete certain course tools based on their own preference.

    The behavioral cues related to read a community post and complete the depression and anxiety test were not seen by all members. This is due to these elements being feature flags and some Evolution Health clients choosing not to offer these tools to their membership base. Although this resulted in fewer members seeing these cues and having access to the course components, the percentage of members who viewed the cues and engaged with the components was noteworthy.

    As this study was designed to form a baseline for future research, there are some methodological issues that can be explored further. For example, unlike the tips presented in arm 2, the use of tips in arm 3 may or may not have been influenced by adding the to-do checklist. Alternatively, using directive tips in arm 3, rather than social proof and present bias tips, may increase the use of the to-do checklist.

    Notably, our evidence indicates that nudges and prompts increase engagement in self-guided treatment programs for depression and anxiety. On the basis of these encouraging yet preliminary results, we can proceed with more sophisticated studies that will examine more enhanced strategies designed to increase platform engagement.

    Future Directions

    As mentioned earlier, there has been scant published research regarding the implementation of behavioral economic strategies designed to increase engagement in digital health programs. At minimum, this randomized controlled trial was successful in confirming member engagement through the use of these strategies.

    As recommended by Aschbacher et al [68] in a recent study involving digital mental health and dose responses, machine learning models can help enable precision by analyzing engagement patterns over time. Combined with another recent paper by Forbes et al [24] analyzing digital interventions for depression, which noted that it is important to develop standardized ways of reporting adherence and engagement so that effective comparisons across different interventions could be measured, baseline outcomes need to be established.

    The baseline outcomes of this study can serve as guideposts for future studies. With machine learning models in place and with the goal of improving member outcomes and platform efficacy, Evolution Health considers the following research questions:

    • Will members of addiction-focused courses (eg, managing drinking and quitting smoking) follow similar engagement patterns if nudges and prompts are made available to them? Which nudges and prompts can be used universally, and which work best for specific mental health or addiction indications?
    • The average member logged into the platform 2.38 times. Can specific tips be introduced at onset to promote log-ins?
    • There were no statistically significant differences in the number of tips clicked on between directive tips in arm 2 and social proof and present bias tips in arm 3 (P=.25). However, arm 2 featured 31 rotating directive tips and arm 3 featured 15 social proof tips and 15 present bias tips. Which tips were most engaging (eg, directive, social proof, or present bias tips)? Which content areas were most engaging (eg, goal setting, community themed, or specific exercises such as the depression and anxiety tests)? Which tips were the most engaging genders or age groups?
    • The average arm 3 member clicked on 2.7 out of 8 to-do checklist items. Which items were these members most likely to click on first? Were there specific patterns of engagement that may influence which tips should be shown to specific members?
    • The course components differ in the effort required to complete them. For example, completing a CBT session is more intensive than the few minutes required to complete the depression and anxiety test. Furthermore, one member may take several minutes to contemplate their goals, whereas others may have concrete goals already established. Future research should analyze duration in relation to each course component, as these data may be leveraged to create tailored tips or to-do content for specific user engagement patterns.
    • The AVD for members who engaged with depression and anxiety course tools ranged from 6 minutes and 45 seconds, with 7.65 pages viewed, to 24 minutes and 21 seconds, with 12.56 pages viewed. In the future, AVD may be used as a benchmark, as overall engagement may be an important metric to calculate when observing dose-response relationships.
    • Members arrived at the platform through the free-to-consumer program and white-label instances licensed by clients that range from employers, insurance companies, employee assistance programs, educational institutions, nonprofit organizations, for-profit health care organizations, and individual therapists. Measuring AVD from these referral sources and patterns of course tool use may assist in creating targeted engagement recommendations.
    • The depression and anxiety test has been validated in a separate study, and the algorithm reports whether members qualify for 30 separate mood and anxiety disorders [48]. Symptoms related to these disorders were also collected and reported to the members. Future research should analyze these data to investigate the possible relationships between symptomology, symptom severity, and engagement patterns.
    • A recent paper by Forbes et al [24] analyzing patient adherence with digital interventions for depression noted that it is important to standardize reporting adherence and engagement. Our future work will focus on establishing baseline metrics. For example, an acceptable click rate for a tip is x% or an acceptable click rate for a to-do checklist item is y%.
    • The platform contains courses for addictive behaviors (eg, smoking cessation and problem drinking), in which members who complete these courses receive a certificate of completion. We are aware that some workforce members received incentives from their human resources department for completing courses and receiving certificates. It may be worthwhile to compare the course completion rates between incentivized addictive behavior courses and nonincentivized mood disorder courses.
    • This experiment examined the use of behavioral prompts. Future studies may examine overcoming behavioral barriers such as bounded rationality or choice architecture.

    It cannot be assumed that the outcomes observed on the Evolution Health platform will be replicated in other environments. Further research in the combined fields of digital health and behavioral economics is required.

    Conclusions

    Members of the Evolution Health platform’s self-guided digital health courses engaged with behavioral nudges and prompts. From this preliminary analysis, it appears that both the tips and to-do checklists increased engagement in course components.

    To the best of our knowledge, this is the first randomized controlled trial designed to test the implementation of behavioral nudges and prompts in web-based self-guided courses for mood disorders. The results of this study may be important because efficacy is related to increased engagement.

    Owing to its novel approach, the outcomes of this study should be interpreted with caution but may be used as a guideline for future research.

    Data Availability

    The data sets generated during and analyzed during this study are not publicly available due to Evolution Health’s data privacy policy, terms of use, and user agreement but are available from the corresponding author on reasonable request.

    Research emerging from this and ongoing data collection will be leveraged to train artificial intelligence models to better understand how to increase and encourage healthy behavior changes. Making these data publicly available may have the opposite effect and enable the development of models that can detect and target susceptible populations.

    The rich data set from this study, ongoing engagement data that are continually collected, or other platform data sets can be made available to researchers interested in conducting studies for noncommercial purposes. Interested researchers are encouraged to contact Evolution Health.

    Conflicts of Interest

    TVM is the chief executive officer and founder of Evolution Health, the owner of the Evolution Health digital health care platform. RF is an Evolution Health board member. No funds were received for conducting this study.

    Multimedia Appendix 1

    Privacy policy and terms of use.

    PDF File (Adobe PDF File), 197 KB
    Multimedia Appendix 2

    CONSORT-eHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 37389 KB
    1. DeChant HK, Tohme WG, Mun SK, Hayes WS, Schulman KA. Health systems evaluation of telemedicine: a staged approach. Telemed J. 1996;2(4):303-312. [CrossRef] [Medline]
    2. Picot J, Cradduck T. The telehealth industry in Canada: industry profile and capability analysis. The Keston Group and Infotelmed Communications. Mar 2000. URL: https:/​/citeseerx.​ist.psu.edu/​document?repid=rep1&type=pdf&doi=c7f26186673239a22d731d13857780ba0a5c7918 [accessed 2024-02-26]
    3. Schneider SJ, Walter R, O'Donnell R. Computerized communication as a medium for behavioral smoking cessation treatment: controlled evaluation. Comput Hum Behav. 1990;6(2):141-151. [CrossRef]
    4. Barak A, Hen L, Boniel-Nissim M, Shapira N. A comprehensive review and a meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. J Technol Hum Serv. 2008;26(2-4):109-160. [CrossRef]
    5. Krämer R, Köhne-Volland L, Schumacher A, Köhler S. Efficacy of a web-based intervention for depressive disorders: three-arm randomized controlled trial comparing guided and unguided self-help with waitlist control. JMIR Form Res. Apr 04, 2022;6(4):e34330. [FREE Full text] [CrossRef] [Medline]
    6. Mohr DC, Kwasny MJ, Meyerhoff J, Graham AK, Lattie EG. The effect of depression and anxiety symptom severity on clinical outcomes and app use in digital mental health treatments: meta-regression of three trials. Behav Res Ther. Dec 2021;147:103972. [FREE Full text] [CrossRef] [Medline]
    7. Mamukashvili-Delau M, Koburger N, Dietrich S, Rummel-Kluge C. Long-term efficacy of internet-based cognitive behavioral therapy self-help programs for adults with depression: systematic review and meta-analysis of randomized controlled trials. JMIR Ment Health. Aug 22, 2023;10:e46925. [FREE Full text] [CrossRef] [Medline]
    8. Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res. Dec 22, 2004;6(4):e46. [FREE Full text] [CrossRef] [Medline]
    9. Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res. Mar 26, 2005;7(1):e7. [FREE Full text] [CrossRef] [Medline]
    10. Wu RC, Delgado D, Costigan J, Maciver J, Ross H. Pilot study of an internet patient-physician communication tool for heart failure disease management. J Med Internet Res. Mar 26, 2005;7(1):e8. [FREE Full text] [CrossRef] [Medline]
    11. Eysenbach G. The law of attrition. J Med Internet Res. Mar 31, 2005;7(1):e11. [FREE Full text] [CrossRef] [Medline]
    12. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res. Apr 24, 2009;11(2):e13. [FREE Full text] [CrossRef] [Medline]
    13. Moshe I, Terhorst Y, Philippi P, Domhardt M, Cuijpers P, Cristea I, et al. Digital interventions for the treatment of depression: a meta-analytic review. Psychol Bull. Aug 2021;147(8):749-786. [CrossRef] [Medline]
    14. Claxton AJ, Cramer J, Pierce C. A systematic review of the associations between dose regimens and medication compliance. Clin Ther. Aug 2001;23(8):1296-1310. [CrossRef] [Medline]
    15. DiMatteo MR. Variations in patients' adherence to medical recommendations: a quantitative review of 50 years of research. Med Care. Mar 2004;42(3):200-209. [CrossRef] [Medline]
    16. Krueger KP, Berger BA, Felkey B. Medication adherence and persistence: a comprehensive review. Adv Ther. Jul 2005;22:313-356. [CrossRef]
    17. van Mierlo T, Fournier R, Ingham M. Targeting medication non-adherence behavior in selected autoimmune diseases: a systematic approach to digital health program development. PLoS One. Jun 24, 2015;10(6):e0129364. [FREE Full text] [CrossRef] [Medline]
    18. Viswanathan M, Golin CE, Jones CD, Ashok M, Blalock SJ, Wines RC, et al. Interventions to improve adherence to self-administered medications for chronic diseases in the United States: a systematic review. Ann Intern Med. Dec 04, 2012;157(11):785-795. [FREE Full text] [CrossRef] [Medline]
    19. Zivin K, Kales HC. Adherence to depression treatment in older adults: a narrative review. Drugs Aging. 2008;25(7):559-571. [CrossRef] [Medline]
    20. Enrique A, Palacios JE, Ryan H, Richards D. Exploring the relationship between usage and outcomes of an internet-based intervention for individuals with depressive symptoms: secondary analysis of data from a randomized controlled trial. J Med Internet Res. Aug 01, 2019;21(8):e12775. [FREE Full text] [CrossRef] [Medline]
    21. McVay MA, Bennett GG, Steinberg D, Voils CI. Dose-response research in digital health interventions: concepts, considerations, and challenges. Health Psychol. Dec 2019;38(12):1168-1174. [FREE Full text] [CrossRef] [Medline]
    22. Alharbi F. The use of digital healthcare platforms during the COVID-19 pandemic: the consumer perspective. Acta Inform Med. Mar 2021;29(1):51-58. [FREE Full text] [CrossRef] [Medline]
    23. Twenge JM, Joiner TE. Mental distress among U.S. adults during the COVID-19 pandemic. J Clin Psychol. Dec 2020;76(12):2170-2182. [FREE Full text] [CrossRef] [Medline]
    24. Forbes A, Keleher MR, Venditto M, DiBiasi F. Assessing patient adherence to and engagement with digital interventions for depression in clinical trials: systematic literature review. J Med Internet Res. Aug 11, 2023;25:e43727. [FREE Full text] [CrossRef] [Medline]
    25. Bond RR, Mulvenna MD, Potts C, O’Neill S, Ennis E, Torous J. Digital transformation of mental health services. Npj Ment Health Res. Aug 22, 2023;2:13. [CrossRef]
    26. Kahneman D. Thinking, Fast & Slow. London, UK. Penguin Books; 2015.
    27. Thaler R. In: Thaler RH, Sunstein CR, editors. Nudge : Improving Decisions About Health. London, UK. Penguin Books; 2009.
    28. van Mierlo T, Hyatt D, Ching AT, Fournier R, Dembo RS. Behavioral economics, wearable devices, and cooperative games: results from a population-based intervention to increase physical activity. JMIR Serious Games. Jan 28, 2016;4(1):e1. [FREE Full text] [CrossRef] [Medline]
    29. Six SG, Byrne KA, Tibbett TP, Pericot-Valverde I. Examining the effectiveness of gamification in mental health apps for depression: systematic review and meta-analysis. JMIR Ment Health. Nov 29, 2021;8(11):e32199. [FREE Full text] [CrossRef] [Medline]
    30. Zhang M, Ying J, Ho RC. Gamified cognitive bias interventions for psychiatric disorders: protocol of a systematic review. JMIR Res Protoc. Oct 16, 2018;7(10):e10154. [FREE Full text] [CrossRef] [Medline]
    31. Susi T, Johannesson M, Backlund P. Serious games: an overview. School of Humanities and Informatics, University of Skövde. Feb 5, 2007. URL: https://www.diva-portal.org/smash/get/diva2:2416/fulltext01.pdf [accessed 2024-02-26]
    32. Tan JW, Zary N. Diagnostic markers of user experience, play, and learning for digital serious games: a conceptual framework study. JMIR Serious Games. Jul 16, 2019;7(3):e14620. [FREE Full text] [CrossRef] [Medline]
    33. Black M, Brunet J. A wearable activity tracker intervention with and without weekly behavioral support emails to promote physical activity among women who are overweight or obese: randomized controlled trial. JMIR Mhealth Uhealth. Dec 16, 2021;9(12):e28128. [FREE Full text] [CrossRef] [Medline]
    34. What is positive behavior support? The Association for Positive Behavior Support. URL: https://www.apbs.org/pbs [accessed 2024-02-26]
    35. Forberger S, Reisch L, Kampfmann T, Zeeb H. Nudging to move: a scoping review of the use of choice architecture interventions to promote physical activity in the general population. Int J Behav Nutr Phys Act. Sep 03, 2019;16(1):77. [FREE Full text] [CrossRef] [Medline]
    36. Shah N, Adusumalli S. Nudges and the meaningful adoption of digital health. Per Med. Nov 2020;17(6):429-433. [CrossRef] [Medline]
    37. Milkman K, Patel MS, Gandhi L, Graci HN, Gromet DM, Ho H, et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor's appointment. Proc Natl Acad Sci U S A. May 18, 2021;118(20):e2101165118. [FREE Full text] [CrossRef] [Medline]
    38. Laibson D. Life-cycle consumption and hyperbolic discount functions. Eur Econ Rev. May 31, 1998;42(3-5):861-871. [CrossRef]
    39. O'Donoghue T, Rabin M. Present bias: lessons learned and to be learned. Am Econ Rev. May 2015;105(5):273-279. [CrossRef]
    40. Burnkrant RE, Cousineau A. Informational and normative social influence in buyer behavior. J Consum Res. Dec 01, 1975;2(3):206-215. [CrossRef]
    41. van Teunenbroek C, Bekkers R, Beersma B. Look to others before you leap: a systematic literature review of social information effects on donation amounts. Nonprofit Volunt Sect Q. 2020;49(1):53-73. [CrossRef]
    42. Xu Y, Chen N, Fernandez A, Sinno O, Bhasin A. From infrastructure to culture: A/B testing challenges in large scale social networks. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2015. Presented at: KDD '15; August 10-13, 2015; Sydney, Australia. [CrossRef]
    43. O'Brien HL, Toms EG. What is user engagement? A conceptual framework for defining user engagement with technology. J Am Soc Inf Sci Technol. Feb 28, 2008;59(6):938-955. [CrossRef]
    44. Rondina R, van Mierlo T, Fournier R. Testing behavioral nudges and prompts in digital courses for the self-guided treatment of depression and anxiety: protocol for a 3-arm randomized controlled trial. JMIR Res Protoc. Aug 15, 2022;11(8):e37231. [FREE Full text] [CrossRef] [Medline]
    45. Cunningham JA, van Mierlo T, Fournier R. An online support group for problem drinkers: AlcoholHelpCenter.net. Patient Educ Couns. Feb 2008;70(2):193-198. [CrossRef] [Medline]
    46. Davis J. The Panic Center. Child Adolesc Ment Health. Feb 2007;12(1):49-50. [CrossRef] [Medline]
    47. Farvolden P, Cunningham J, Selby P. Using e-health programs to overcome barriers to the effective treatment of mental health and addiction problems. J Technol Hum Serv. 2009;27(1):5-22. [CrossRef]
    48. Farvolden P, McBride C, Bagby RM, Ravitz P. A web-based screening instrument for depression and anxiety disorders in primary care. J Med Internet Res. 2003;5(3):e23. [FREE Full text] [CrossRef] [Medline]
    49. Mulvaney-Day N, Marshall T, Downey Piscopo K, Korsen N, Lynch S, Karnell LH, et al. Screening for behavioral health conditions in primary care settings: a systematic review of the literature. J Gen Intern Med. Mar 2018;33(3):335-346. [FREE Full text] [CrossRef] [Medline]
    50. Selby P, Farvolden P, Van Mierlo T. Demographic characteristics, cessation history and stages of change status among users of a web-based smoking cessation program. In: Proceedings of the National Conference on Tobacco or Health. 2002. Presented at: NCTOH 2002; November 19-21, 2002; San Francisco, CA.
    51. Selby P, van Mierlo T, Voci SC, Parent D, Cunningham JA. Online social and professional support for smokers trying to quit: an exploration of first time posts from 2562 members. J Med Internet Res. Aug 18, 2010;12(3):e34. [FREE Full text] [CrossRef] [Medline]
    52. Urbanoski K, van Mierlo T, Cunningham J. Investigating patterns of participation in an online support group for problem drinking: a social network analysis. Int J Behav Med. Oct 2017;24(5):703-712. [CrossRef] [Medline]
    53. van Mierlo T, Hyatt D, Ching AT. Mapping power law distributions in digital health social networks: methods, interpretations, and practical implications. J Med Internet Res. Jun 25, 2015;17(6):e160. [FREE Full text] [CrossRef] [Medline]
    54. Directive 95/46/EC of the European parliament and of the council on the protection of individuals with regard to the processing of personal data and on the free movement of such data. European Union. 1995. URL: https://www.refworld.org/legal/reglegislation/eu/1995/en/13712 [accessed 2024-02-26]
    55. Personal Information Protection and Electronic Documents Act (S.C. 2000, c. 5). Government of Canada. 2000. URL: https://laws-lois.justice.gc.ca/eng/acts/p-8.6/ [accessed 2024-02-26]
    56. Summary of the HIPAA privacy rule. U.S. Department of Health and Human Services. URL: https://www.hhs.gov/hipaa/for-professionals/privacy/laws-regulations/index.html [accessed 2024-02-26]
    57. WMA declaration of Helsinki – ethical principles for medical research involving human subjects. World Medical Association. URL: https:/​/www.​wma.net/​policies-post/​wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/​ [accessed 2024-02-26]
    58. The "C" in your RCT - and what it means for the current state of Digital Health. Evolution Health. Nov 30, 2023. URL: https:/​/www.​linkedin.com/​pulse/​c-your-rct-what-means-current-state-digital-health-jvsfc?trk=public_post_feed-article-content [accessed 2024-02-26]
    59. Eysenbach G, CONSORT-EHEALTH Group. CONSORT-EHEALTH: improving and standardizing evaluation reports of web-based and mobile health interventions. J Med Internet Res. Dec 31, 2011;13(4):e126. [FREE Full text] [CrossRef] [Medline]
    60. van Mierlo T, Li X, Hyatt D, Ching AT. Demographic and indication-specific characteristics have limited association with social network engagement: evidence from 24,954 members of four health care support groups. J Med Internet Res. Feb 17, 2017;19(2):e40. [FREE Full text] [CrossRef] [Medline]
    61. Serafini G, Costanza A, Aguglia A, Amerio A, Placenti V, Magnani L, et al. Overall goal of cognitive-behavioral therapy in major psychiatric disorders and suicidality: a narrative review. Med Clin North Am. Jan 2023;107(1):143-167. [CrossRef] [Medline]
    62. Fernandez E, Salem D, Swift JK, Ramtahal N. Meta-analysis of dropout from cognitive behavioral therapy: magnitude, timing, and moderators. J Consult Clin Psychol. Dec 2015;83(6):1108-1122. [CrossRef] [Medline]
    63. Youn SJ, Marques L. Intensive CBT: how fast can I get better? Harvard Health Publishing. Oct 23, 2018. URL: https://tinyurl.com/ckmb56ur [accessed 2024-02-26]
    64. Overview - Cognitive behavioural therapy (CBT). National Health Service. URL: https://tinyurl.com/msbfzt8h [accessed 2023-01-04]
    65. Cognitive behavioral therapy. Mayo Clinic. URL: https://www.mayoclinic.org/tests-procedures/cognitive-behavioral-therapy/about/pac-20384610 [accessed 2023-01-04]
    66. Cognitive behavioral therapy. National Institutes of Health National Library of Medicine. URL: https://www.ncbi.nlm.nih.gov/books/NBK279297/ [accessed 2024-02-26]
    67. Cognitive behavioual therapy. Centre for Addiction and Mental Health. URL: https://www.camh.ca/en/health-info/mental-illness-and-addiction-index/cognitive-behavioural-therapy [accessed 2024-01-04]
    68. Aschbacher K, Rivera LM, Hornstein S, Nelson BW, Forman-Hoffman VL, Peiper NC. Longitudinal patterns of engagement and clinical outcomes: results from a therapist-supported digital mental health intervention. Psychosom Med. Sep 01, 2023;85(7):651-658. [CrossRef] [Medline]

    ‎
    AVD: average visit duration
    CBT: cognitive behavioral therapy
    CONSORT-eHEALTH: Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth
    H1: primary hypothesis

    Edited by A Mavragani; submitted 07.09.23; peer-reviewed by JC Buckey, ES Silverman, L Callender; comments to author 21.12.23; revised version received 04.01.24; accepted 13.02.24; published 09.04.24.

    Copyright

    ©Trevor van Mierlo, Renante Rondina, Rachel Fournier. Originally published in JMIR Formative Research (https://formative.jmir.org), 09.04.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.