Binge drinking, defined as consuming 5 (for men) or 4 (for women) standard drinks within approximately 2 hours on a single occasion [1], continues to be a public health concern, particularly among college students [1]. In this millennium, there has been an increase of over 200% in alcohol-related deaths and a 25% increase in alcohol overdose–related hospitalizations in the United States [2]. Binge drinking can result in short-term effects such as poor academic performance, but more seriously, it can lead to detrimental health outcomes such as alcohol use disorder and other health risks [3]. College binge drinking habits can predict the course of alcohol dependence and problems over a lifetime [4,5]. Developing effective interventions to reduce binge drinking among college students has the potential to improve lifetime drinking outcomes and prevent alcohol-related health and psychosocial problems among college students across the United States.

Undergraduate students are commonly exposed to evidence-based binge drinking preventive and clinical interventions that aim to promote drinking safety [6-8]. An example of a common intervention across universities in the United States is the Brief Alcohol Screening and Intervention for College Students program, which provides a harm reduction intervention that uses personalized normative feedback and structured prompts to guide behavior change for students who have risky drinking habits or alcohol-related consequences [9]. Despite universities continually improving alcohol programming, these efforts yield minimal change and rates of dangerous alcohol consumption have not changed significantly; worse, the effects of these programs diminish over time [10-13]. Therefore, novel interventions that engage students and promote lower risk drinking behavior by college students are still needed. A tailored mobile app for students could provide personalized alcohol safety information in real time, either before or during a drinking episode.

With extensive student involvement, and based on findings from other formative work [14-16], we designed a draft mobile app for safer student drinking. The aim of this study is to determine college students’ perspectives on the appeal, relevance, and perceived utility of the draft app.

The setting was a public university in Virginia with extensive alcohol programming, alcohol interventions, and student involvement in governance, which has experienced several high-profile alcohol-related injuries and deaths of students in the past decades. We sought to identify new ways to reduce hazardous drinking among college students. We conducted 2 studies before this one. First, we conducted in-depth interviews and focus groups with students and university stakeholders to understand the culture of the university, the role of drinking in student and alumni life, the history and impact of various alcohol interventions, the continuing ubiquity of binge drinking despite deployment of numerous interventions, and students’ impressions of the impact of alcohol interventions to which they had been exposed (Alamiri, N, unpublished data, January 2022). Second, we conducted a review of public social media posts to understand how alcohol use and its consequences at the university were documented on the web by students and bystanders [17]. These formative studies demonstrated that (1) risky drinking is a prominent feature of student life, (2) most students have participated in risky drinking behavior, (3) students report minimal change in their own drinking in response to existing alcohol programming, (4) mobile app use by students is ubiquitous, and (5) students wanted flexible intervention tools that are accessible during drinking episodes (their own or when witnessing excessive drinking by their peers). Students requested a novel intervention with evidence-based and student-preferred features and functions. An estimated blood alcohol content (eBAC) calculator (to estimate the percentage of alcohol in a person’s bloodstream [15]) was the feature students were most interested in when the concept was presented during the focus groups and surveys. Overall, the formative findings indicated that novel interventions must be highly relevant to drinking behaviors and be usable during drinking.

Study team members created a draft prototype app based on prior experience in mobile app development and prior research on alcohol intervention development. Previous research indicates the benefits of self-monitoring for behavior change [18,19]. Logging risky behaviors, such as drinking, helps reduce those behaviors [14], leading us to include a drink tracker. To boost engagement, the app included personalized encouragement [20,21] to users who tracked several days in a row (streak). We used push notifications [22] to provide safety messages during heavy drinking episodes to alert the user of potential danger and provide recommended actions. In-app messaging, called the message center [22], provided testimonials from real students about drinking situations. To appeal to college student users’ needs, we added a secondary behavior (hydration or sleep) to track that could prompt users to engage with the app even when not drinking alcohol to keep users engaged [23]. The settings feature within the app allowed users to turn off push notifications and adjust their weight to reflect a more accurate eBAC. Once the prototype app was developed, 2 classes of undergraduate and graduate students in a mobile engineering class provided initial feedback on the draft app, and some features were slightly redesigned. Thus, the final prototyped app contained eight features and a home screen (Figure 1): alcoholic drink tracker (Figure 2); additional behavior tracker (hydration or sleep); self-monitoring feedback of days tracked; push messages about drinking and safety; eBAC calculator that provided an eBAC based on weight, sex, and timing or specific drinks logged; learning topics (Figure 3); a message center containing student testimonials (Figure 4); and settings.

The study included a single 1-hour qualitative interview of each undergraduate student participant using standardized think-aloud methods [24], described below. This study was approved by the Social and Behavioral Science Institutional Review Board of the University of Virginia (protocol 3282).

To be eligible, a study applicant had to be (1) a student at the university, (2) able to read and speak English, and (3) aged ≥18 years. Participants reviewed an information form before completing any study activities and provided verbal consent that was audio recorded for documentation. The first round of interviews consisted of students who were not members of Greek organizations or student athletes. Participants in the second round included students who were fraternity and sorority members and student athletes. These groups tested the prototype app separately because previous research has shown that student athletes or who are in Greek organizations are at a higher risk for dangerous drinking episodes than their non-Greek peers [25-29], and we sought to gather feedback from groups who may have different alcohol use patterns.

Once all of the notes were captured, the RAs deleted the audio recording. Researchers reviewed the notes and presented the major findings to the app development team for discussion and identification of features and functions to be modified. Reporting was in aggregate only (ie, “5 of the 20 participants said x, while 12 said y and 3 said z”). The app development team continuously iterated and updated the look and feel of the mock-up based on the feedback. Data included audio recordings of the think-aloud protocol process, notes taken by researchers observing participants, and usability questionnaire responses. All participants were assigned a unique ID unrelated to their identity. Stored data were linked only to this unique ID. All data were stored behind the Health Insurance Portability and Accountability Act firewall of an institutional review board-approved School of Medicine highly secure data drive.

Demographic characteristics of the overall study sample and by condition are presented in Table 1. Participants were mostly women (7/10, 70%) and aged 20 years (5/10, 50%), and 80% (8/10) were enrolled in the most populous school within the university, containing departments in the Arts and Sciences.

The feedback provided by the participants in person during round 1 is presented in Table 2. These students provided feedback on the eight essential features of the app: drink tracker, alternate behavior tracker (hydration or sleep), days tracked, messages, settings, eBAC calculator, learn topics, and a message center containing student testimonials. During the first round, RAs asked participants to focus on usability, rather than aesthetics, of the app. Participants expressed confusion about the purpose of the features and provided useful feedback about how they were displayed. For example, the drink tracker had drink choices listed that the students wanted to rename (eg, they recommended renaming seltzer or cider, to hard seltzer and hard cider, respectively, to ensure that future users would log alcoholic versions of these drinks) and indicated that such changes would increase utility and relevance. Participants reported that they would prefer having brand name examples (eg, Corona, Coors Lite, and White Claw) for each of the choices (ie, beer, liquor, seltzer, mixed drink, cider, and wine) to help clarify the different types of alcoholic drinks, again increasing utility and relevance as well as appeal. They reported that this would help future student users accurately track their drinks and that they believed many students do not know the alcohol by volume percentages for typical drinks.

Participants also provided feedback on whether they were interested in and would use certain features. For example, students were not interested in a hydration tracker feature and considered its utility and appeal low. The hydration tracker asks students to rate how hydrated they are on a scale of 1 to 5. Participants expressed confusion about the hydration tracking as it was not as specific as the (alcoholic) drink tracker. Students questioned how the hydration tracker related to the app, failing to see any connection between hydration and drinking alcohol. Thus, the hydration tracker showed low relevance.

Participants in round 2 met with the RAs on the web using video meeting technology, but other than this, all other procedures remained similar. In the second round of think-aloud testing, RAs guided participants to focus on the look and feel (eg, aesthetics and design) more than the functionality of the app. For example, the app’s default setting is in dark mode, with a black background; a participant expressed preference for a light mode option to increase its appeal. In addition, participants expressed a preference for symbols for sex rather than the terms male and female for personal data entry screens. Participant feedback in round 2 is presented in Table 3. Overall, participants expressed interest in several of the features of the app, but not in others. The look and feel of the app were reported to be important to the students. Student navigation across app features indicated that the improved look and feel helped guide them to the features they were most interested in using.

The objective of this study is to assess the appeal, relevance, and perceived utility of a draft mobile app for college students that could intervene at or around the time of alcohol use. We sought to determine if students would want and use the app and if it would be perceived as useful and engaging. Participants reported that minor changes to the functionality of the initial versions of the app would improve the usability of the app (eg, having examples of each alcoholic drink for clarification and stating whether they believe certain features would be used or not). In addition, participants reported that minor changes to the aesthetics would increase the usability (eg, incorporating the logo colors throughout the app to make it more vibrant and having an option to turn off push notifications). The changes were made between rounds of think-aloud user testing to enact modifications based on user feedback.

This process was an iterative step within a series of formative studies to create and improve an app to reduce college binge drinking. Feedback provided by college students assisted the investigators and development team in designing features that were useful and appealing. The purpose was to test usability, and student feedback helped the team modify the app so that it would perform as students hoped in real-life scenarios to address binge drinking and alcohol consumption. This process was designed as a feedback loop beginning with the science that drafted features about learning what a standard drink is, self-monitoring tools with immediate and longer-term feedback about risks when drinking, options to get help in an emergency, and vetted information about sex differences in alcohol processing in the body. The features were conceptualized and then mocked up by developer team members with expertise in user experience and user interface design). Finally, the team obtained qualitative feedback on relevance and usability from students regarding the aesthetics and appeal of the app and its ability to engage them.

Although the practical insights from students about the app proved valuable and guided us to make important app modifications, there are several limitations to be considered. Because of COVID-19, students had to participate in the second round of think-aloud trials virtually over Zoom using an adapted protocol. Although participants navigated through the app with ease, the researchers could not always see the participants’ phone screen and sometimes had difficulty determining which features the participants were commenting on. RAs asked what feature the participant was viewing and where in the app they were navigating to and prompted the participant to turn their phone screen toward the camera. These moments of clarification seemed to interrupt the natural rhythm and flow of the think-aloud format and may have resulted in reduced information. In person, the researcher could easily see the participant’s phone screen without prompting, which allowed for the conversation to flow uninterrupted while the RA could take notes more inconspicuously. During round 1, the participants’ comments were prominent in the conversation, whereas researchers listened and asked fewer questions. Conversely, researchers asked more questions during round 2, whereas participants provided less feedback, and their comments were either positive or more superficial in nature.

Previous studies have used apps to help reduce dangerous drinking episodes among college students. However, these apps did not have or include features that tailor to individual personalized feedback such as logging drinks and providing an eBAC [34-36]. One of these previous interventions has justified the use of a digital platform as users did indicate that it offered help and was easy to use [36]. Other alcohol app interventions have reported to significantly reduce drinking [34,35].

Data and conclusions from this study will be synthesized with information gained from other qualitative and quantitative formative work, to determine the final features to be tested. We plan a pilot randomized trial to assess the use and impact of the app on drinking behavior over several months; we plan to conduct a prospective brief observational and feasibility study with about 255 students to (1) measure student demographics, attitudes, experiences, and behaviors related to drinking; (2) evaluate the acceptability, feasibility, relevance, and actual use of the app over the intervention period among students; (3) measure student attitudes, experiences, and behaviors related to drinking in a final survey; (4) determine the usability of the app and desired improvements using short interviews after about 4 weeks of app use; and (5) test the impact of incentives on app use to provide crucial information for implementation of a larger subsequent trial. If the pilot of this mobile app intervention shows acceptable use and reporting of frequency and volume of binge drinking, we will consider testing it further for impact on drinking behaviors.

Student feedback clearly indicated preferences for the eBAC calculator and days tracked features. In contrast, student feedback showed a dislike of or lower interest in using the water or hydration tracker, learn topics, and messages or testimonials. On the basis of student ratings of appeal, relevance, and utility, we revised the app, and the finalized app included the drink tracker, days tracked, eBAC calculator with an eBAC level, sleep quality tracker, learn topics, message center, and testimonials. By asking student participants to navigate through, and comment on, the app’s features using standard think-aloud procedures, the researchers gained real time perspective on the appeal, relevance, and utility of the app that could be translated into a refined version of the app.