Background: Mobile apps have shown considerable promise for reducing alcohol consumption among problem drinkers, but like many mobile health apps, they frequently report low utilization, which is an important limitation, as research suggests that effectiveness is related to higher utilization. Interactive chatbots have the ability to provide a conversational interface with users and may be more engaging and result in higher utilization and effectiveness, but there is limited research into this possibility.
Objective: This study aimed to develop a chatbot alcohol intervention based on an empirically supported app (Step Away) for reducing drinking and to conduct a pilot trial of the 2 interventions. Included participants met the criteria for hazardous drinking and were interested in reducing alcohol consumption. The study assessed utilization patterns and alcohol outcomes across the 2 technology conditions, and a waitlist control group.
Methods: Participants were recruited using Facebook advertisements. Those who met the criteria for hazardous consumption and expressed an interest in changing their drinking habits were randomly assigned to three conditions: the Step Away app, Step Away chatbot, and waitlist control condition. Participants were assessed on the web using the Alcohol Use Disorders Identification Test, Adapted for Use in the United States, Readiness to Change Questionnaire, Short Inventory of Problems-Revised, and Timeline Followback at baseline and at 12 weeks follow-up.
Results: A total of 150 participants who completed the baseline and follow-up assessments were included in the final analysis. ANOVA results indicated that participants in the 3 conditions changed their drinking from baseline to follow-up, with large effect sizes noted (ie, η2=0.34 for change in drinks per day across conditions). However, the differences between groups were not significant across the alcohol outcome variables. The only significant difference between conditions was in the readiness to change variable, with the bot group showing the greatest improvement in readiness (F2,147=5.6; P=.004; η2=0.07). The results suggested that the app group used the app for a longer duration (mean 50.71, SD 49.02 days) than the bot group (mean 27.16, SD 30.54 days; P=.02). Use of the interventions was shown to predict reduced drinking in a multiple regression analysis (β=.25, 95% CI 0.00-0.01; P=.04).
Conclusions: Results indicated that all groups in this study reduced their drinking considerably from baseline to the 12-week follow-up, but no differences were found in the alcohol outcome variables between the groups, possibly because of a combination of small sample size and methodological issues. The app group reported greater use and slightly higher usability scores than the bot group, but the bot group demonstrated improved readiness to change scores over the app group. The strengths and limitations of the app and bot interventions as well as directions for future research are discussed.
Trial Registration: ClinicalTrials.gov NCT04447794; https://clinicaltrials.gov/ct2/show/NCT04447794
Access to evidence-based treatment and support for addressing excessive alcohol use is a public health priority given that alcohol continues to be the third leading preventable cause of death in the United States, and its excessive use is responsible for >95,000 deaths each year [, ]. A recent review of death certificates found that the annual number of deaths from alcohol-related causes doubled between 1999 and 2017 for individuals aged ≥16 years [ ]. Although the effectiveness of behavioral interventions for alcohol addiction is well established [ , ], the National Epidemiological Survey on Alcohol and Related Conditions found that treatment utilization is very low; in 2019, among the 14.5 million people aged ≥12 years reporting a past year alcohol use disorder, only 7.6% received treatment for alcohol use at any location [ ].
Technology-Delivered Alcohol Interventions and Health Equity
Technology-based interventions, including mobile apps, have great potential to meaningfully expand access to treatment and have been shown to be acceptable among alcohol and other substance users [- ]. Over the past 15 years, numerous behavior-change interventions have been created to capitalize on the potential of the internet, including several alcohol interventions with demonstrated effectiveness in reducing alcohol consumption without the guidance of a counselor [ - ]. For example, the Drinker’s Checkup, a web-based brief motivational intervention that provides alcohol-use assessment, individualized feedback, and an intervention to develop a plan of behavior change, reduced alcohol consumption among problem drinkers by 50%, with reductions maintained at the 12-month follow-up [ ]. A meta-analysis found that effect sizes from technology-based interventions were “as effective or nearly as effective as face-to-face therapy” [ ], with clinical benefits found in other reviews [ , ]. In addition to their effectiveness, technology-based interventions have great potential to reduce health disparities in hidden populations [ , ], including homeless individuals [ ]. Whether technology is used to deliver direct treatment services [ ], or to provide behavioral support for reducing alcohol use or preventing relapse [ ], technology-based alcohol interventions overcome several personal and access-related barriers to in-person alcohol treatment, including poor or inadequate availability of services; cost and inadequate insurance; convenience in the face of childcare, work, and transportation challenges; and beliefs about help-seeking as shameful or a sign of weakness [ , ]—barriers to treatment that are particularly salient for women, minorities, and those in rural locations [ - ]. Relatedly, technology-based interventions have the potential to address concerns about privacy and stigma that may be associated with attending alcohol treatment facilities [ , ].
Mobile Devices and Contextual Relevance
One important advantage of technology-based alcohol interventions is that they can be accessed from home and the intervention is situated within daily contexts [- ]. Much of what leads to continued alcohol use or relapse occurs in an individual’s everyday environment, where, unlike a traditional treatment context, exposure to alcohol-related cues, stress, and negative affect are uncontrolled [ , ]. These contextual and situational cues may overwhelm an individual’s coping resources and other skills that were discussed during treatment sessions with the hope that they would be used when facing cravings for alcohol triggered by internal and external factors encountered in daily life [ , ]. The increased use of smartphones to access the internet is especially promising, as these mobile devices provide a way to support behavior-change goals whenever and wherever needed. While digital divide issues persist with regard to high-speed internet access, it is estimated that 81% of all American adults, 79% aged 50 to 64 years and 53% aged ≥65 years, own smartphones [ ]. These data also indicate that minority groups are at least as likely as White individuals to own a smartphone, and recent trends show a preference for smartphone use over desktop access to the internet [ ], suggesting an opportunity to improve treatment access for historically underserved groups.
Similar to other internet-based interventions, smartphone-based interventions help bridge the gap between those in need of treatment and those receiving it by addressing stigma concerns associated with treatment program attendance, leveraging the desire to independently manage an alcohol problem, and eliminating the need for physical travel to a treatment facility. In addition, as smartphones are carried at almost all times by their owners, they have greater potential to provide timely interventions in the actual environment in which drinking occurs, and given that most mobile health apps are free or sell for <US $5, they are much more affordable than traditional treatment. That being said, the growing volume of publicly available apps varies in quality and effectiveness, and even those developed using theories of behavior change and evidence-based content may lack rigorous evaluation. In a recent systematic review of health behavior change apps, including diet, physical activity, and alcohol use, many reported improvements in targeted outcomes, but few demonstrated significant treatment effects over comparison groups in randomized trials . The reported limitations included study design issues (eg, nonrepresentative samples and inadequate comparison groups), intervention design issues (eg, features offered are not based on theory or existing evidence), and limited or short-lived engagement with the app [ , ].
Treatment Effectiveness and Effective Management
Maintained engagement with behavior change apps has been associated with app effectiveness in a number of studies and may depend on specific design features that improve adherence and efficacy. Engagement-enhancing features include visually appealing and easily navigated content; features based on behavior-change principles including feedback, self-monitoring, and data-driven adaptation; and features that promote therapeutic alliance including acceptance and support, relatability, and positive expectations . For example, conversational agents that use artificial intelligence to proactively guide, prompt, and check in with participants and encourage general use of the app have been found to improve the therapeutic alliance between the participant and the intervention, resulting in more engagement [ ]. In other studies, the ability to set goals, self-monitor progress, and receive feedback increased engagement across a variety of health behaviors, including alcohol use [ ]. An advantage of apps is their ability to generate utilization data, which are commonly used as a proxy for quantifying engagement. Utilization metrics typically include the number of log-ins, proportion of features accessed, and frequency and duration of use [ ]. However, interpreting the relative importance of these metrics to determine the amount and type of engagement that is sufficient to achieve the desired behavioral and health outcomes (ie, effective engagement) has not been established [ ]. Given the lack of practical measures to comprehensively assess effective engagement, researchers continue to rely on utilization data with the assumption that more engagement is better [ ].
Step Away: A Behavior Change App for Reducing Excessive Alcohol Use
Step Away is a smartphone app designed to deliver empirically based alcohol assessment and intervention for individuals who drink at hazardous levels that may present health risks. Step Away is the next generation of an earlier app that we tested (the Location-Based Monitoring and Intervention System-Alcohol) with individuals with an alcohol use disorder, which demonstrated significant 6-week reductions in alcohol consumption, along with ratings by participants as being very helpful in changing their drinking habits . This study also indicated that the amount of use of Location-Based Monitoring and Intervention System-Alcohol features was related to changes in alcohol consumption. The design of Step Away is informed by three theoretical constructs that are considered the key active ingredients for person-centered, behavioral-based intervention and the treatment of addictions: motivational enhancement [ ], relapse prevention [ ], and community reinforcement [ ]. The app offers eight modules in addition to daily alcohol consumption and craving tracking: (1) assessment and feedback on alcohol consumption relative to age-specific norms, drinking-related problems, and monetary costs of drinking, including daily prompting to complete a brief questionnaire on drinking behavior and cravings during the prior 24 hours, and weekly feedback highlighting progress toward goals; (2) goal setting, which asks participants to select abstinence or moderation as a goal; (3) rewards, which prompts them to set up a reward for meeting their goal and reminds them to reward themselves when their goal is met (eg, 30 days of no drinking); (4) cravings, which offers 6 in-the-moment interventions for coping with cravings; (5) moderation or abstinence strategies, which consists of simple behavioral strategies tailored to the participant’s goal; (6) supportive persons, which provides tools for connecting with participant-identified friends or family when additional support is needed; (7) reminders, which encourages the creation of visual reminders of their reasons for changing their drinking habits, including the ability to upload inspirational photos to make a change; and (8) new activities, which recommends healthy behaviors and the ability to schedule selected activities within the smartphone calendar. Step Away also provides real-time intervention options; that is, when a participant clicks on the “Get Help” icon, they are provided with strategies for managing cravings or negative emotions and contacting a national treatment finder service to receive help finding in-person treatment in their area.
Although engagement with Step Away has been shown to be relatively high with a recent study of the app in a veteran sample showing that approximately 40% of participants were still engaged with the app at 6 months , use of the treatment steps or other modules was found to be relatively low [ ]. In an attempt to increase intervention engagement, we developed a chatbot version of Step Away that incorporates the app modules into a chat-delivered intervention, which we postulated could result in higher use and engagement over the Step Away app. A chatbot is a computer program that simulates human conversation powered by pre-established rules and artificial intelligence. They have become common in e-commerce, call centers, and internet gaming. They serve as digital assistants that communicate with a user through texting to help a user with numerous tasks, such as planning air travel or helping with a bank transaction. In the smartphone context, they are being used in texting interfaces, in such a way that it appears to the user that they are having an actual conversation with the service. An important distinction between apps and bots is that a bot user has the perception of talking with the service. Bots are emerging as promising tools in contemporary health care [ ]. An example of a health care chatbot is Melody, which has a text conversation with a patient about their symptoms and provides potential diagnoses that a health care practitioner can then use to develop a treatment plan [ ].
A key difference between the app and bot is that the app relies on the user to launch a new feature on their own, whereas the bot guides the user through a conversational interface and essentially serves as an alcohol reduction personal assistant. The bot is also designed to perform a daily check-in regarding ongoing drinking and cravings and offers a menu of in-the-moment tools that can be used to manage the situation or experience.
This paper presents results from a 3-month pilot study that compared effectiveness and participant engagement differences among individuals randomly assigned to one of three groups: (1) Step Away app, (2) Step Away bot, and (3) assessment-only delayed condition (control). Our study builds on the existing literature by indicating whether a chatbot-delivered version of a smartphone-delivered intervention has superior engagement and alcohol outcomes compared with an app with similar intervention features over a 12-week duration, and whether these interventions produce superior outcomes over a waitlist control condition.
The study used a randomized controlled study design and a mixed methods approach (ClinicalTrials.gov NCT04447794). Participants were enrolled and randomly assigned to one of three groups: Step Away app (for iPhone and Android smartphones), Step Away chatbot, and assessment-only delay (control). Participants were assessed for their alcohol consumption and related behaviors when they enrolled in the study (baseline) and again 12 weeks later (follow-up). Age and gender stratification were used to ensure a relatively even distribution across each intervention or control group.
Participant recruitment was conducted through Facebook advertisements, which provided a link to the study website and the web-based prescreening survey. Recruitment began in early June 2021 and was completed in early September 2021 when the target sample size was reached. Advertisements were targeted to all Facebook users who may meet the following criteria: age ≥18 years; have either an iPhone or an Android phone; reside in the United States; not in another form of alcohol treatment or using another mobile health alcohol intervention, be an active drinker, and have proficiency in English language.
The prescreening survey asked potential study participants about these criteria as well as all 10 questions from the Alcohol Use Disorders Identification Test, Adapted for Use in the United States (USAUDIT). Those who met these criteria, and who had a USAUDIT score between 8 and 24 inclusive for males aged ≤65 years and a score between 7 and 24 inclusive for females as well as males aged ≥65 years, were invited to complete the consent form and the baseline survey. The target audience for Step Away is those whose drinking patterns fall in the at-risk or high-risk USAUDIT zones. Therefore, those with USAUDIT scores of ≥8 for males and ≥7 for females met the criteria for risky drinking; those with scores of ≥25 were not eligible to participate. A US $25 Amazon e-gift card was provided to each participant when they completed the baseline survey, and another US $25 Amazon e-gift card was emailed after the completion of the follow-up survey.
The following measures were assessed at baseline and follow-up. The USAUDIT was assessed at the time of screening for eligibility and again at baseline for comparison. Demographic characteristics such as age, sex, education, and ethnicity were collected.
Alcohol Use Disorder Identification Test, Adapted for Use in the United States
The USAUDIT  is a 10-item measure used to identify hazardous drinking. The scale response is scored from the least frequent, 0 (never) to the most frequent, 6 (daily). Scores of ≥8 have been shown to detect hazardous drinking, and a cutoff point <8 has been suggested as a cutoff with higher sensitivity for women [ , ]. The Alcohol Use Disorders Identification Test was found to have a median Cronbach α of .83 and a test-retest reliability of 0.92 [ ].
Short Inventory of Problems-Revised
The Short Inventory of Problems-Revised (SIP-R)  assesses alcohol-related problems through 15 questions scored on a 4-point Likert scale from 1 (never) to 4 (daily or almost daily). Higher scores indicate more life problems related to alcohol use [ ]. Among problem drinkers, the SIP-R demonstrated good concurrent validity and internal consistency [ - ].
The Timeline Followback (TLFB)  assesses the quantity and frequency of alcohol consumption. Participants reported the number of drinks they had consumed each day for the past 30 days [ ]. When compared with daily interviews using a smartphone, the TLFB showed concurrent validity; however, this diminished over time as participants were less likely to remember the number of drinks they had per day [ ].
Readiness to Change Questionnaire
The Readiness to Change Questionnaire (RTCQ)  is a 12-item questionnaire that measures the stage of change. The scores were calculated to categorize a participant in the “precontemplation,” “contemplation,” or “action” stage of change regarding changing their drinking behavior [ ]. The RTCQ demonstrated a test-retest reliability of about 0.80 per scale and a Cronbach α of about .80 per scale [ ].
System Usability Scale
The 10-item System Usability Scale (SUS)  measures a user’s experience of a product’s usability. The SUS was found to be reliable, with a Cronbach α of .91 [ ]. Participants in the app and chatbot groups completed the SUS to assess the usability of both tools.
Screening and administering consent forms and baseline and follow-up surveys were all done using the web-based survey platform Qualtrics. Data were downloaded from Qualtrics in CSV format for review and analysis.
Screening and Consent
Completed screenings, including the USAUDIT, were automatically scored using Qualtrics to determine eligibility. Scores of ≤8 indicated that they did not answer all 9 of the demographic criteria questions required for eligibility (eg, age and residency in the United States); scores of ≥9 were considered eligible. Individuals meeting the demographic criteria were eligible if their USAUDIT score fell between 7 and 24 for females or between 8 and 24 for males. The cutoff criteria were set to reflect the minimum cutoff score for detecting hazardous drinking . Those who scored >24 were not eligible owing to safety concerns; USAUDIT scores of >25 indicate a moderate to severe alcohol use disorder or dependence [ ]. These individuals instead received an automated message providing treatment options and mental health resources, including the SAMHSA treatment locator and the national suicide prevention hotline. If they met the eligibility criteria, Qualtrics was used to automatically link them to the web-based consent form.
Baseline survey links were emailed to participants once they were manually reviewed by the study team to confirm their eligibility. Up to 3 reminder emails were sent to complete the baseline survey, if necessary.
Follow-up surveys were matched to the baseline surveys for each participant. An email with a link to the follow-up survey was manually sent to each participant as they became eligible (ie, after they had used the app or chatbot for 12 weeks, or 12 weeks after their study enrollment for the control group). To encourage a high follow-up response rate, up to 3 reminder emails were sent and up to 3 phone reminder calls were made.
To ensure that the study did not enroll participants who were only seeking renumeration (“phishers”), further participant validation checks were conducted before emailing the baseline survey link, including reviewing the collected data for identical email addresses, multiple duplicate IP addresses, and geolocation data. At follow-up, additional validation checks were conducted. We compared the participants’ demographic responses from baseline to follow-up. In this step, 13 participants reported significant inconsistencies in demographics from baseline to follow-up (eg, they reported inconsistent answers to questions about their ethnicity and gender between baseline and follow-up). We were unable to verify these participants through a telephone call, and they were thus removed from the analyses to ensure valid responses.
We recruited 1417 participants in total, including those who completed the prescreen. Of these, 417 patients were eligible and consented to participate. After examining participant responses regarding study eligibility, additional participants were removed because of having numerous prescreen submissions under the same IP address which represented phishing, or the automatic scoring through the prescreen allowing ineligible participants into the study (eg, they indicated being currently in alcohol treatment which was an exclusion criteria). Subsequently, the baseline surveys were sent to 197 participants. A few participants (n=6) were further found to be ineligible after examining their prescreening surveys, leaving 191 eligible baseline surveys. At follow-up, 163 participants completed the survey. After removing 13 participants owing to failing our validation checks, we analyzed data from 150 participants, 55 (36.7%) app users, 50 (33.3%) bot users, and 45 (30%) participants in the delay group.shows the flow of this study. All analyses were conducted using SPSS (version 27; IBM Corp).
The study was reviewed and approved by the University of Alaska Anchorage institutional review board (1521800).
shows demographics from each intervention group.
|Characteristics||App (n=55)||Bot (n=50)||Delay (n=45)|
|Age (years), mean (SD)||42.58 (13.49)||40.82 (12.54)||41.51 (13.07)|
|Sex, n (%)|
|Male||25 (46)||22 (44)||16 (36)|
|Female||30 (54)||28 (56)||29 (64)|
|Race and ethnicity, n (%)|
|African American||5 (9)||8 (16)||3 (7)|
|White||44 (80)||36 (72)||37 (82)|
|Asian American||2 (4)||0 (0)||5 (11)|
|Alaska Native or American Indian||0 (0)||1 (2)||0 (0)|
|Hispanic or Latinx||3 (6)||4 (8)||0 (0)|
|Other||1 (2)||1 (2)||0 (0)|
Change in Drinking Measure From Baseline to Follow-up
Missing data were imputed using multiple imputation. TLFB data correlated with age, gender, and SIP-R data. Multiple imputation was performed using age, gender, SIP-R, and TLFB data as predictor variables for missing TLFB data with 5 data sets imputed. A total of 36 respondents were excluded from the TLFB analyses owing to insufficient TLFB data, resulting in a total of 114 participants from the app (n=42, 36.8%), bot (n=39, 34.2%), and delay (n=33, 29%) groups for the TLFB analyses. These participants did not complete any of the 30-day TLFB survey questions at either baseline or follow-up; therefore, multiple imputation was not possible with these participants. We calculated drinks per day (DPD), percent days abstinent (PDA), and heavy drinking days (HDD).shows the descriptive data for each intervention group regarding drinking variables at baseline and follow-up. shows the percentage of days spent drinking hazardously (our primary outcome variable) from baseline to follow-up among the 3 groups.
|Timeline followback measures (N=114), n (%)||42 (36.8)||42 (36.8)||39 (34.2)||39 (34.2)||33 (28.9)||33 (28.9)|
|DPD,a mean (SD)||2.69 (1.72)||1.51 (1.04)||2.64 (1.26)||1.75 (1.17)||2.50 (1.14)||1.88 (1.41)|
|PDAb (%), mean (SD)||24 (28)||44 (31)||22 (22)||36 (29)||21 (24)||39 (32)|
|HDD,c mean (SD)||7.69 (8.33)||3.29 (5.50)||7.49 (7.98)||3.54 (6.76)||6.30 (5.65)||4.94 (6.92)|
|Drinking outcome measures (N=150), n (%)||55 (36.7)||55 (36.7)||50 (33.3)||50 (33.3)||45 (30)||45 (30)|
|AUDIT,d mean (SD)||15.25 (4.35)||12.62 (7.24)||16.28 (3.49)||14.76 (7.01)||15.60 (4.42)||14.62 (7.50)|
|RTCQ,e mean (SD)||2.35 (0.55)||2.40 (0.68)||2.14 (0.50)||2.62 (0.57)||2.35 (0.57)||2.35 (0.65)|
|SIP-R,f mean (SD)||11.69 (7.25)||8.16 (6.80)||12.68 (10.28)||9.86 (7.95)||13.51 (9.46)||9.53 (8.45)|
aDPD: drinks per day.
bPDA: percent days abstinent.
cHDD: heavy drinking days (as defined by ≥4 DPD for females and ≥5 DPD for males).
dAUDIT: Alcohol Use Disorder Identification Test, Adapted for Use in the United States.
eRTCQ: Readiness to Change Questionnaire.
fSIP-R: Short Inventory of Problems-Revised.
Alcohol Consumption Results by Group
Repeated measures ANOVA were conducted on the 3 dependent drinking variables calculated from the TLFB, DPD, PDA, and HDD. The within-subjects variable was time (baseline to follow-up), and the between-subjects variable was the intervention group (app, bot, or delay). There was a significant effect of time on DPD (F2,111=55.93; P<.001; η2=0.34), PDA (F2,111=42.00; P<.001; η2=0.27), and HDD (F2,111=28.18; P<.001; η2=0.20), and all were large effect sizes. There was no significant interaction between time and group for both DPD (F2,111=1.74; P=.18; η2=0.03) and HDD (F2,111=2.27; P=.11; η2=0.04). These results indicate that all participants reduced their drinking significantly between baseline and follow-up, and there was no significant difference in the reduction in drinking based on the group they were randomized to.
Repeated measures ANOVA were conducted on three dependent drinking-related variables: AUDIT, RTCQ, and SIP-R. The within-subjects variable was time (baseline to follow-up), and the between-subjects variable was the intervention group (app, bot, or delay). There was a significant effect of time on the SIP (F2,147=24.76; P<.001; η2=0.14), with a large effect size. There was a significant effect of time with a medium effect size on AUDIT (F2,147=10.97; P=.001; η2=0.07) and RTCQ (F2,147=7.79; P=.006; η2=0.05). Between- and within-subjects ANOVAs were conducted to compare the intervention groups for the 3 dependent drinking-related variables. There was no significant interaction between time and group for AUDIT or SIP-R scores, indicating that all 3 intervention groups significantly improved on these drinking measures between baseline and follow-up but did not differ from one another. There was a significant interaction between time and group for the RTCQ (F2,147=5.62; P=.004; η2=0.07) with a medium effect size, indicating that readiness to change scores varied by intervention group, with those in the bot group improving significantly more between baseline and follow-up than those in the app or delay group. The intervention group did not effect change in SIP or AUDIT scores for participants.
App Versus Bot Utilization Results
Three different variables were calculated to measure utilization. We calculated from the Step Away database that the app and the bot collected, the number of times users clicked on a function of the app as the total number of visits they made to each of the interventions. Duration of use was calculated as the time between the date the user downloaded the app and their date of last use. Active days used was calculated as the number of days that a user was actively using the app, which was defined as having entered the app and clicked on at least one feature. Utilization data from 18 bot users were missing, indicating that these participants either did not use the bot at all or did not enter their personal identification number at the start of the study, which was required to match their utilization data to their study ID. Therefore, the results must be interpreted with caution and the means for the bot utilization variables are likely lower than those presented here. Independent sample 2-tailed t tests were conducted between the means of each variable to determine whether there were significant differences between app and bot users. Duration of use significantly differed between app users and bot users (F85=12.23; t85=2.45; P=.02), with duration of use being higher among app users (mean 50.71, SD 49.02) than among bot users (mean 27.16, SD 30.54). Total visits were higher for app users (mean 33.96, SD 38.16) than for bot users (mean 24.56, SD 28.68; F85=1.53; t85=1.21; P=.22), and the average active days used was higher for app users (mean 22.62, SD 29.08) than bot users (mean 19.63, SD 26.30; F85=.02; t85=.48; P=.88), although neither of these differences were statistically significant.
Effect of Utilization of the Intervention on Change in Drinking Variables
Significant direct effects were shown for the duration of use of the app or bot on the change in average DPD and PDA. Increased duration of use predicted a greater decrease in DPD (β=.01, SE 0.00; β=.25, 95% CI 0.00-0.01; P=.04), and increased duration of use predicted a greater change in the increase of PDA (β=−.18, SE 0.07; β=−.29, 95% CI −0.32 to −0.03; P=.02). There were trends of utilization increasing change in drinking that were not significant, including an effect of total visits on the change in PDA (β=−.16, SE 0.10; β=−.20, 95% CI −0.35 to 0.03; P=.10), of active days of use on the change in PDA (β=−.21, SE 0.12; β=−.21, 95% CI −0.45 to 0.03; P=.08), and of duration of use on the change in HDD (β=.03, SE 0.02; β=.23, 95% CI −0.00 to 0.07; P=.06). These trends indicated that increased use of the app or bot led to greater decreases in drinking and increases in abstinence.
Usability Ratings for the App Versus Bot
Means from the overall SUS score were compared between app and bot users using independent sample 2-tailed t tests. App users rated a higher mean SUS score (mean 66.35, SD 19.68) than bot users (mean 61.70, SD 25.40), but this difference was not statistically significant. A typical acceptable SUS score is approximately 70, and scores <50 are considered unacceptable . presents the mean (SD) for the app and bot participants for each SUS question.
|Items||App (n=52), mean (SD)||Bot (n=48), mean (SD)|
|1. I think I would use the Step Away app/chatbot frequently.||3.33 (1.31)||3.15 (1.41)|
|2. I found the Step Away app/chatbot unnecessarily complex.||2.46 (1.21)||2.65 (1.44)|
|3. I thought the Step Away app/chatbot was easy to use.||3.83 (1.20)||3.56 (1.41)|
|4. I think that I would need assistance to be able to use the Step Away app/chatbot.||1.92 (1.25)||2.19 (1.35)|
|5. I found the various functions in the Step Away app/chatbot well-integrated.||3.42 (1.18)||3.35 (1.42)|
|6. I thought there was too much inconsistency in the Step Away app/chatbot.||2.35 (1.06)||2.85 (1.38)|
|7. I would imagine that most people would learn to use the Step Away app/chatbot very quickly.||3.94 (0.94)||3.88 (1.20)|
|8. I found the Step Away app/chatbot very cumbersome to use.||2.67 (1.31)||2.94 (1.48)|
|9. I felt very confident using the Step Away app/chatbot.||3.75 (1.05)||3.75 (1.25)|
|Total score||66.35 (19.68)||61.70 (25.40)|
This study sought to develop a chatbot version of the empirically supported app, Step Away, and conduct a pilot trial to determine if a chatbot version could provide enhanced use and outcome effectiveness over the app version. We also sought to pilot trial a methodology that included randomly assigning participants who were hazardously consuming alcohol and interested in making a change to their drinking to one of three conditions: the Step Away app, the Step Away bot, and a waitlist control condition.
Results from this pilot study indicated that self-reported alcohol consumption from baseline to the 12-week follow-up decreased substantially in all groups. Effect sizes suggested that changes in alcohol consumption and drinking-related problems were within the large effect range. However, the results also suggested that there were no statistically significant differences in alcohol consumption variables between the 3 groups, suggesting that the waitlist control condition changed their drinking at a similar level to the intervention groups. The lack of statistical significance could potentially be related to an inadequate sample size (the purpose of the study was not to provide a robust test of effectiveness and power calculations were not performed). It is also likely that the waitlist control condition, owing to their expressed desire to change their drinking, had a strong (and positive) reaction to being assessed at baseline for their alcohol consumption and drinking-related variables, such as life problems stemming from alcohol consumption . A delayed assessment of the control group (until the intervention groups are assessed at follow-up) is a highly recommended strategy for managing this possibility.
Another focus of this study, which is perhaps equally important for alcohol outcome analysis, is the utilization assessment of the 2 interventions. In contrast to our expectations, our results suggested that the app was used more frequently and for a longer time than the bot. There are 2 possible explanations for this observation. The Step Away app was originally developed in 2013 and has undergone 4 major revisions based on user input and recommendations. A finding that has emerged frequently from research on Step Away is that the app’s ability to provide self-monitoring and tailored feedback to the user is a key driver of utilization [, , ], which has resulted in the feedback feature being prominent and well-developed. This was the first version of the bot, and, like many first versions, it had some unforeseen limitations. First, similar to all Facebook Messenger bots, it provided a daily SMS text message prompting the user to check in with the bot; however, if they did not respond, the bot did not continue to provide prompts. It relied on the user to return to it on subsequent days to re-engage with it. We heard from some users that they thought the bot disengaged from them as it stopped providing reminders to complete their daily interviews. This notification problem did not exist with the app and app users were notified daily regardless of their response to the prompts.
Another factor related to the feedback is that although an app can provide sophisticated graphs and other pictographic representations of progress, bots are much more limited in this regard (a limitation that existed when we developed the bot). The bot’s feedback, although similar in content to the app, was more simplistic and perhaps not captivating. Finally, on 2 occasions the bot crashed (ie, ceased to function) during the study, possibly owing to complications related to its dependence on Facebook Messenger, and making it difficult for our developers to resolve promptly. The app also crashed once during the study, but the problem was immediately corrected.
Although this study indicated numerous benefits of the app over the bot, one finding favored the bot. Bot participants were found to have a greater change in motivation to change their drinking compared with the app. It is possible that the conversational tone and the feeling that the bot was more like talking to a person could have enhanced users’ motivation to change their drinking habits. Future research should explore this possibility.
Limitations and Future Directions
This study had numerous limitations, many because of its pilot nature, which are related to recommendations for future research. First, the sample was insufficient for detecting small to medium effect sizes, and our findings reflected this limitation. We estimated that a sample size of 195 would provide enough power to detect between-group effect sizes, as shown in this study. Second, the methodology for assessing the waitlist control condition likely resulted in reactivity to assessment phenomena, which made detecting differences between the interventions and the control condition challenging. Future studies in this area would be wise, if using a waitlist control condition, to delay assessment, a method that has been used successfully in other studies . We also had limited time to undertake this study and our 12-week follow-up period may have been insufficient for differences between interventions to emerge. Previous research with Step Away showed that participants continued to reduce their alcohol intake at 6 months [ ] and that 45% of participants were still actively engaged at the 6-month follow-up. A 12-month follow-up would provide a more detailed picture of how users remain engaged with the interventions over time and how this engagement is related to improvement. Finally, although this study set out to contrast the 2 interventions and determine which had higher use and effectiveness, this contrast is perhaps not ideal. A more beneficial strategy may involve combining these technologies. Throughout the development of the bot, we realized that both bots and apps each have their unique strengths. Apps can provide highly captivating images and graphs depicting progress, while bots provide the feeling that the user is interacting with someone or something, which has the potential to increase motivation and overall engagement. Apps are also more self-guided than bots and allow the user to interact in a more self-directed, timely manner than a bot, which guides the user by suggesting modules and responding in a conversational manner. It is possible to merge these 2 technologies into 1 system by placing a bot within an app. Thus, it may be that the best in class smartphone-based system for reducing drinking is a hybrid of a bot and an app, a promising direction for future interventions.
Research on smartphone-based interventions for alcohol problems has been promising, but more research into factors related to use and long-term effectiveness is urgently needed. This study indicated that the app and bot interventions were related to substantial improvements in drinking, as was the waitlist control condition, which resulted in inconclusive results. We found that the app was more highly used, which may have been related to some bot limitations that are easily addressable, such as improved prompting over time and greater system stability, which has undoubtedly improved since the development of the Step Away bot. Future research should use longer follow-ups, more sophisticated methodology regarding the timing of the assessment for the control condition, and perhaps leveraging the 2 technologies to develop a hybrid system that has the beneficial elements of an app and a bot.
Research reported in this publication was supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award number R34AA026440. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Conflicts of Interest
PD has a financial interest in the company that owns the Step Away app and bot. He did not participate in data collection and analysis in this study.
CONSORT-EHEALTH checklist (V 1.6.1).PDF File (Adobe PDF File), 1248 KB
- Mokdad AH, Marks JS, Stroup DF, Gerberding JL. Actual causes of death in the United States, 2000. JAMA 2004 Mar 10;291(10):1238-1245. [CrossRef] [Medline]
- Alcohol and public health: alcohol-related disease impact (ARDI): annual average for United States 2011-2015: alcohol-attributable deaths due to excessive alcohol use, all ages. Centers for Disease Control and Prevention. 2020. URL: https://nccd.cdc.gov/DPH_ARDI/Default/Default.aspx [accessed 2022-03-23]
- White AM, Castle IP, Hingson RW, Powell PA. Using death certificates to explore changes in alcohol-related mortality in the United States, 1999 to 2017. Alcohol Clin Exp Res 2020 Jan;44(1):178-187. [CrossRef] [Medline]
- Whitlock EP, Polen MR, Green CA, Orleans T, Klein J, U.S. Preventive Services Task Force. Behavioral counseling interventions in primary care to reduce risky/harmful alcohol use by adults: a summary of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med 2004 Apr 06;140(7):557-568 [FREE Full text] [CrossRef] [Medline]
- Miller WR, Brown J, Simpson TL, Handmaker NS, Bien TH, Luckie LF, et al. What works? A methodological analysis of the alcohol treatment outcome literature. In: Hester RK, Miller WR, editors. Handbook of alcoholism treatment approaches: effective alternatives. 2nd ed. Boston, MA, USA: Allyn & Bacon; 1995:12-24.
- Substance Abuse and Mental Health Services Administration. Key substance use and mental health indicators in the United States: results from the 2019 national survey on drug use and health (HHS Publication No. PEP20-07-01-001, NSDUH Series H-55). Rockville, MD, USA: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration; 2020.
- Vernon ML. A review of computer-based alcohol problem services designed for the general public. J Subst Abuse Treat 2010 Apr;38(3):203-211 [FREE Full text] [CrossRef] [Medline]
- Choo EK, Ranney ML, Wong Z, Mello MJ. Attitudes toward technology-based health information among adult emergency department patients with drug or alcohol misuse. J Subst Abuse Treat 2012 Dec;43(4):397-401 [FREE Full text] [CrossRef] [Medline]
- Lieberman DZ, Huang SW. A technological approach to reaching a hidden population of problem drinkers. Psychiatr Serv 2008 Mar;59(3):297-303. [CrossRef] [Medline]
- Cunningham JA. Comparison of two internet-based interventions for problem drinkers: randomized controlled trial. J Med Internet Res 2012 Aug 01;14(4):e107 [FREE Full text] [CrossRef] [Medline]
- Cunningham JA, Wild TC, Cordingley J, van Mierlo T, Humphreys K. A randomized controlled trial of an internet-based intervention for alcohol abusers. Addiction 2009 Dec;104(12):2023-2032 [FREE Full text] [CrossRef] [Medline]
- Hester RK, Delaney HD, Campbell W. ModerateDrinking.Com and moderation management: outcomes of a randomized clinical trial with non-dependent problem drinkers. J Consult Clin Psychol 2011 Apr;79(2):215-224 [FREE Full text] [CrossRef] [Medline]
- Riper H, Spek V, Boon B, Conijn B, Kramer J, Martin-Abello K, et al. Effectiveness of E-self-help interventions for curbing adult problem drinking: a meta-analysis. J Med Internet Res 2011 Jun 30;13(2):e42 [FREE Full text] [CrossRef] [Medline]
- Hester RK, Squires DD, Delaney HD. The Drinker's Check-up: 12-month outcomes of a controlled clinical trial of a stand-alone software program for problem drinkers. J Subst Abuse Treat 2005 Mar;28(2):159-169. [CrossRef] [Medline]
- Barak A, Hen L, Boniel-Nissim M, Shapira N. A comprehensive review and a meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. J Technol Hum Serv 2008;26(2-4):109-160 [FREE Full text] [CrossRef]
- Newman MG, Szkodny LE, Llera SJ, Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy? Clin Psychol Rev 2011 Feb;31(1):89-103. [CrossRef] [Medline]
- Gainsbury S, Blaszczynski A. A systematic review of Internet-based therapy for the treatment of addictions. Clin Psychol Rev 2011 Apr;31(3):490-498. [CrossRef] [Medline]
- Postel MG, de Jong CA, de Haan HA. Does e-therapy for problem drinking reach hidden populations? Am J Psychiatry 2005 Dec;162(12):2393. [CrossRef] [Medline]
- Lin LA, Casteel D, Shigekawa E, Weyrich MS, Roby DH, McMenamin SB. Telemedicine-delivered treatment interventions for substance use disorders: a systematic review. J Subst Abuse Treat 2019 Jun;101:38-49. [CrossRef] [Medline]
- McInnes DK, Li AE, Hogan TP. Opportunities for engaging low-income, vulnerable populations in health care: a systematic review of homeless persons' access to and use of information technologies. Am J Public Health 2013 Dec;103 Suppl 2:e11-e24. [CrossRef] [Medline]
- Burner E, Zhang M, Terp S, Ford Bench K, Lee J, Lam CN, et al. Feasibility and acceptability of a text message-based intervention to reduce overuse of alcohol in emergency department patients: controlled proof-of-concept trial. JMIR Mhealth Uhealth 2020 Jun 04;8(6):e17557 [FREE Full text] [CrossRef] [Medline]
- Wallhed Finn S, Bakshi AS, Andréasson S. Alcohol consumption, dependence, and treatment barriers: perceptions among nontreatment seekers with alcohol dependence. Subst Use Misuse 2014 May;49(6):762-769. [CrossRef] [Medline]
- McCrady BS, Epstein EE, Fokas KF. Treatment interventions for women with alcohol use disorder. Alcohol Res 2020;40(2):08 [FREE Full text] [CrossRef] [Medline]
- Alvanzo AA, Storr CL, Mojtabai R, Green KM, Pacek LR, La Flair LN, et al. Gender and race/ethnicity differences for initiation of alcohol-related service use among persons with alcohol dependence. Drug Alcohol Depend 2014 Jul 01;140:48-55 [FREE Full text] [CrossRef] [Medline]
- Gilbert PA, Pro G, Zemore SE, Mulia N, Brown G. Gender differences in use of alcohol treatment services and reasons for nonuse in a national sample. Alcohol Clin Exp Res 2019 Apr;43(4):722-731 [FREE Full text] [CrossRef] [Medline]
- Verissimo AD, Grella CE. Influence of gender and race/ethnicity on perceived barriers to help-seeking for alcohol or drug problems. J Subst Abuse Treat 2017 Apr;75:54-61 [FREE Full text] [CrossRef] [Medline]
- Heron KE, Smyth JM. Ecological momentary interventions: incorporating mobile technology into psychosocial and health behaviour treatments. Br J Health Psychol 2010 Feb;15(Pt 1):1-39 [FREE Full text] [CrossRef] [Medline]
- Burns MN, Begale M, Duffecy J, Gergle D, Karr CJ, Giangrande E, et al. Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res 2011 Aug 12;13(3):e55 [FREE Full text] [CrossRef] [Medline]
- Waters AJ, Schoenmakers TM, Snelleman M, Szeto EH, Franken IH, Hendriks VM, et al. Affect, motivation, temptation, and drinking among alcohol-dependent outpatients trying to maintain abstinence: an Ecological Momentary Assessment study. Drug Alcohol Depend 2020 Jan 01;206:107626. [CrossRef] [Medline]
- Bell L, Garnett C, Qian T, Perski O, Williamson E, Potts HW. Engagement with a behavior change app for alcohol reduction: data visualization for longitudinal observational study. J Med Internet Res 2020 Dec 11;22(12):e23369 [FREE Full text] [CrossRef] [Medline]
- Starcke K, Antons S, Trotzke P, Brand M. Cue-reactivity in behavioral addictions: a meta-analysis and methodological considerations. J Behav Addict 2018 Jun 01;7(2):227-238 [FREE Full text] [CrossRef] [Medline]
- Kalivas PW, Volkow ND. The neural basis of addiction: a pathology of motivation and choice. Am J Psychiatry 2005 Aug;162(8):1403-1413. [CrossRef] [Medline]
- Anderson M. Mobile technology and home broadband 2019. Pew Research Center. 2019. URL: https://www.pewresearch.org/internet/2019/06/13/mobile-technology-and-home-broadband-2019/ [accessed 2022-03-27]
- Milne-Ives M, Lam C, De Cock C, Van Velthoven MH, Meinert E. Mobile apps for health behavior change in physical activity, diet, drug and alcohol use, and mental health: systematic review. JMIR Mhealth Uhealth 2020 Mar 18;8(3):e17046 [FREE Full text] [CrossRef] [Medline]
- Baumel A, Kane JM. Examining predictors of real-world user engagement with self-guided eHealth interventions: analysis of mobile apps and websites using a novel dataset. J Med Internet Res 2018 Dec 14;20(12):e11491 [FREE Full text] [CrossRef] [Medline]
- Ly KH, Ly AM, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv 2017 Dec;10:39-46 [FREE Full text] [CrossRef] [Medline]
- Spaulding EM, Marvel FA, Piasecki RJ, Martin SS, Allen JK. User engagement with smartphone apps and cardiovascular disease risk factor outcomes: systematic review. JMIR Cardio 2021 Feb 03;5(1):e18834 [FREE Full text] [CrossRef] [Medline]
- Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, et al. Measuring engagement in eHealth and mHealth behavior change interventions: viewpoint of methodologies. J Med Internet Res 2018 Nov 16;20(11):e292 [FREE Full text] [CrossRef] [Medline]
- Dulin PL, Gonzalez VM, Campbell K. Results of a pilot test of a self-administered smartphone-based treatment system for alcohol use disorders: usability and early outcomes. Subst Abus 2014;35(2):168-175 [FREE Full text] [CrossRef] [Medline]
- Miller WR. Enhancing motivation for change in substance abuse treatment: TIP Series 35. Substance Abuse and Mental Health Services Administration. Rockville, MD: US Department of Health and Human Services; 1999. URL: https://store.samhsa.gov/sites/default/files/d7/priv/tip35_final_508_compliant_-_02252020_0.pdf [accessed 2022-03-27]
- Marlatt GA, Donovan DM. Relapse prevention: maintenance strategies in the treatment of addictive behaviors. 2nd edition. New York, NY, USA: The Guilford Press; 2005.
- Meyers RJ, Villanueva M, Smith JE. The community reinforcement approach: history and new directions. J Cogn Psychother 2005;19(3):247-260. [CrossRef]
- Malte CA, Dulin PL, Baer JS, Fortney JC, Danner AN, Lott AM, et al. Usability and acceptability of a mobile app for the self-management of alcohol misuse among veterans (Step Away): pilot cohort study. JMIR Mhealth Uhealth 2021 Apr 08;9(4):e25927 [FREE Full text] [CrossRef] [Medline]
- Furness D. The chatbot will see you now: AI may play doctor in the future of healthcare. Digital Trends. 2016. URL: https://www.digitaltrends.com/cool-tech/artificial-intelligence-chatbots-are-revolutionizing-healthcare/ [accessed 2022-03-27]
- Taylor H. How China's biggest search engine aims to fix a huge crisis in health care: a bot. CNBC. 2016. URL: https://www.cnbc.com/2016/10/11/chinese-baidu-unveils-ai-health-chatbot-for-patients-and-doctors.html [accessed 2022-03-27]
- Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on early detection of persons with harmful alcohol consumption--II. Addiction 1993 Jun;88(6):791-804. [CrossRef] [Medline]
- Reinert DF, Allen JP. The alcohol use disorders identification test: an update of research findings. Alcohol Clin Exp Res 2007 Feb;31(2):185-199. [CrossRef] [Medline]
- Miller WR, Tonigan JS, Longabaugh R. The drinker inventory of consequences (DrInC): an instrument for assessing adverse consequences of alcohol abuse: test manual. National Institute on Alcohol Abuse and Alcoholism. Rockville, MD: National Institutes of Health; 1995. URL: https://pubs.niaaa.nih.gov/publications/projectmatch/match04.pdf [accessed 2022-03-27]
- Kiluk BD, Dreifuss JA, Weiss RD, Morgenstern J, Carroll KM. The short inventory of problems - revised (SIP-R): psychometric properties within a large, diverse sample of substance use disorder treatment seekers. Psychol Addict Behav 2013 Mar;27(1):307-314 [FREE Full text] [CrossRef] [Medline]
- Alterman AI, Cacciola JS, Ivey MA, Habing B, Lynch KG. Reliability and validity of the alcohol short index of problems and a newly constructed drug short index of problems. J Stud Alcohol Drugs 2009 Mar;70(2):304-307 [FREE Full text] [CrossRef] [Medline]
- Feinn R, Tennen H, Kranzler HR. Psychometric properties of the short index of problems as a measure of recent alcohol-related problems. Alcohol Clin Exp Res 2003 Sep;27(9):1436-1441. [CrossRef] [Medline]
- Sobell LC, Sobell MB. Timeline follow-back: a technique for assessing self-reported alcohol consumption. In: Litten RZ, Allen JP, editors. Measuring alcohol consumption: psychosocial and biological methods. Totowa, NJ, USA: Humana Press; 1992:41-72.
- Dulin PL, Alvarado CE, Fitterling JM, Gonzalez VM. Comparisons of alcohol consumption by time-line follow back vs. smartphone-based daily interviews. Addict Res Theory 2017;25(3):195-200 [FREE Full text] [CrossRef] [Medline]
- Rollnick S, Heather N, Gold R, Hall W. Development of a short 'readiness to change' questionnaire for use in brief, opportunistic interventions among excessive drinkers. Br J Addict 1992 May;87(5):743-754. [CrossRef] [Medline]
- Brooke J. SUS: a 'quickdirty' usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors. Usability evaluation in industry. London, UK: CRC Press; 1996:189-194.
- Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact 2008 Jul 30;24(6):574-594. [CrossRef]
- Planning and implementing screening and brief intervention for risky alcohol use: a step-by-step guide for primary care practices. Centers for Disease Control and Prevention. 2014. URL: https://www.cdc.gov/ncbddd/fasd/documents/alcoholsbiimplementationguide.pdf [accessed 2022-03-27]
- Meier E, Miller MB, Lombardi N, Leffingwell T. Assessment reactivity: a randomized controlled trial of alcohol-specific measures on alcohol-related behaviors. Addict Behav 2017 Apr;67:44-48 [FREE Full text] [CrossRef] [Medline]
- Blonigen DM, Harris-Olenak B, Kuhn E, Timko C, Humphreys K, Smith JS, et al. Using peers to increase veterans' engagement in a smartphone application for unhealthy alcohol use: a pilot study of acceptability and utility. Psychol Addict Behav 2021 Nov;35(7):829-839. [CrossRef] [Medline]
- Gonzalez VM, Dulin PL. Comparison of a smartphone app for alcohol use disorders with an Internet-based intervention plus bibliotherapy: a pilot study. J Consult Clin Psychol 2015 Apr;83(2):335-345 [FREE Full text] [CrossRef] [Medline]
|DPD: drinks per day|
|HDD: heavy drinking days|
|PDA: percent days abstinent|
|RTCQ: Readiness to Change Questionnaire|
|SIP-R: Short Inventory of Problems-Revised|
|SUS: System Usability Scale|
|TLFB: Timeline Followback|
|USAUDIT: Alcohol Use Disorders Identification Test, Adapted for Use in the United States|
Edited by A Mavragani; submitted 19.08.21; peer-reviewed by W Peng, T Fazzino; comments to author 23.10.21; revised version received 02.12.21; accepted 14.03.22; published 16.05.22Copyright
©Patrick Dulin, Robyn Mertz, Alexandra Edwards, Diane King. Originally published in JMIR Formative Research (https://formative.jmir.org), 16.05.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.