Usability and Satisfaction Testing of Game-Based Learning Avatar-Navigated Mobile (GLAm), an App for Cervical Cancer Screening: Mixed Methods Study

Introduction

Adherence to cervical cancer screening guidelines among eligible individuals aged 21-29 years decreased between 2005 and 2016 [1]. Barriers to screening in young adults include lack of knowledge or awareness, negative perceptions of testing, and systemic barriers [2]. Mobile health (mHealth) technology has the potential to improve knowledge outside of the clinical encounter [3-5]. A meta-analysis of various mHealth platforms showed technology was associated with increased uptake of cancer screening, with the largest effect for cervical cancer screening (pooled OR 1.71, 95% CI 1.34-2.19) [6]. A scoping review concluded mHealth interventions may be an effective way to reach individuals in need of screening and improve cervical cancer screening uptake [7]. These results suggest mHealth has a role in cervical cancer screening uptake, but interventions are varied and most often comprise SMS text messages, telephone calls, or social media messaging, and less often mobile apps. The purpose of this study was to test the ease of use, usefulness, and satisfaction of the Game-Based Learning Avatar-Navigated Mobile (GLAm) app, designed to increase knowledge and awareness of cervical cancer and screening [2] and elucidate the screening process.

Methods

GLAm App Development

The GLAm app was developed in collaboration with the University of Minnesota Education Technology Innovations team (Multimedia Appendix 1). The objective of the app is to educate and motivate guideline-adherent cervical cancer screening among average-risk individuals aged 21-29 years. The Fogg Health Behavior Model was used as a theoretic guide [8], and the app design was informed by results from testing an app previously developed by our group [5]. The current app was tailored to a younger population and incorporated more gamified aspects. The content was informed by a literature review of barriers to cervical cancer screening in young adults and a focus group study conducted by the study team. A user-created avatar navigates the app, which comprises a single education module in testing mode. The app uses a multiple-choice quiz format to test knowledge and to provide information on each topic. Some explanations included short videos (eg, how a Pap test is performed). Trophies are awarded for app registration, avatar creation, education module completion, videos watched, and logging completed cervical cancer screening (Multimedia Appendix 1). The app is programmed to send SMS text message reminders for the next recommended screening test, but this feature could not be tested in this study. The app includes links to additional cervical cancer screening information.

Recruitment

Participants were recruited through a social media (Facebook and Instagram)–promoted message over a 2-week time period. Inclusion criteria were being female, being at average risk for cervical cancer, being aged 21-29 years, and residing in the United States, per self-report. Interested participants submitted a form with their contact information on the internet through the secure Research Electronic Data Capture (REDCap) web app [9] managed by the University of Minnesota Clinical and Translational Science Institute. Potential study participants were then contacted by the study team by email or telephone, per their preference. The anticipated number of usability problems was small given the simple design of the GLAm app, and sample size was determined through standard practices for data obtained through think-aloud methodologies [10,11]. Goal accrual for the qualitative study was 6 participants (maximum 9). The goal accrual for the quantitative survey study was 12 participants (maximum 24) and included the qualitative study participants.

Ethics Approval

The University of Minnesota Institutional Review Board approved the study (#STUDY00010234). Participants provided written informed consent and Health Insurance Portability and Accountability Act (HIPAA) authorization form electronically through REDCap [12], and participants were emailed a PDF of the signed consent. All study data were deidentified. Each study participant was assigned a unique study number, and the link between the study number and participant identifiers was maintained in a REDCap database that was separate from any study data. Survey data were stored in REDCap [9]. Other deidentified study data, including videos of the think-aloud play, were stored in a HIPAA-compliant cloud-based Box Secure Storage file managed by the University of Minnesota. Study participants in the think-aloud play were instructed in advance to remove their name from the Zoom screen and were provided with their study number to use as their username for the app so that their identity was not visible at any point during the think-aloud play. All data collection and storage plans were approved by the University of Minnesota Health Information Privacy and Compliance Office as part of the institutional review board review and approval process. Participants were compensated with a US $50 gift card for participation in the think-aloud play interview and a US $25 gift card for 1-week app review and survey completion.

Mixed Methods

To test the usability, usefulness, and satisfaction of the GLAm app, we followed a mixed methods approach, first conducting a qualitative think-aloud play interview session to examine usability and to identify and revise any problematic components of the app. This was followed by a separate quantitative survey study to describe acceptability and satisfaction with the app. Both studies are described in detail below, and integration of the components is illustrated in Multimedia Appendix 2.

Qualitative Study

Initial usability testing was performed using think-aloud play through videoconferencing (Zoom). After receiving standardized verbal instructions, the participant shared their screen and voiced their thoughts while interacting with the GLAm app. The think-aloud play was observed by the study’s principal investigator (DT) and research assistant (LJW). Study staff did not interact with the participants during think-aloud play except for a prerecorded prompt (“keep talking”) if the participant was silent for 15 seconds. App components identified as problematic by the first group of think-aloud play participants (n=5) were changed before testing of the last 4 participants and the quantitative study: (1) correction of a misspelled word in 1 quiz answer explanation; (2) rewording of a quiz question for better clarity; and (3) reorganization of the opening screen to make avatar creation more visible. Sessions were video recorded and auto-transcribed using videoconferencing technology. Transcripts were reviewed and corrected, and nonverbal observations were added. Data were analyzed using directed content analysis to identify major themes of ease of use, usefulness, and content satisfaction. A member of the study team (LJW) read and coded transcripts, with a second senior team member (KTS) participating in the coding. The team reviewed quotes, codes, interpretations, and themes that ultimately evolved into thematic labels, concepts, and relationships. Once thematic saturation was achieved, thematic patterns were evaluated for similarities and differences that additionally refined coding and identified the final themes.

Quantitative Study

Participants interacted with the app independently for 1 week and then completed a survey on acceptability and satisfaction. Acceptability was assessed using the validated Technology Acceptance Model [13], with questions adapted to the use of an app. Satisfaction was measured using the validated Computer System Usability Questionnaire [14], adapted to the use of an app. Surveys were completed on the internet using REDCap [9]. Mean (SD) scores (minimum score 1 and maximum score 7) are reported.

Results

Qualitative Study

Overview

The demographics of the 9 participants are detailed in Tables 1 and 2. All identified as non-Hispanic, Latina, or Latino and (8/9) 89% identified as White. All regions of the country were represented. All were cisgender people, and 8 had at least some secondary education. The mean think-aloud play time was 14.9 (SD 5.2) minutes. We identified 3 major themes.

Theme 1: Ease of Use—Educational Content Must Be Concise and Easily Digestible

Several participants preferred time-efficient and easy-to-comprehend educational content. Many participants enjoyed learning through the question-and-answer format of the game, and conciseness was favored (Table 3; participant 1). One participant expressed frustration with the quiz format (Table 3; participant 3), and frequently skipped reading informative explanations. Participants were satisfied with visual learning methods (Table 3; participants 1, 2, and 7). The Pap test video was 2.5 minutes long, and 2 participants did not watch it completely, expressing, “I know the gist of how that goes” and “I think I’ve seen enough of that video, I kind of get the concept.” (participants 1 and 6).

Theme 2: Usefulness—Content That Eases Anxiety Around Screening Is the Most Helpful

A total of 5 participants mentioned negative feelings about screening. The screening was described as “scary,” “painful,” and “uncomfortable” (participants 4-6). One participant admitted she had been avoiding the Pap test “like the plague” (participant 4). Participants also expressed confusion and anxiety about interpreting screening recommendations and results: “[Receiving results can be] super confusing, super scary, especially if you have never received an abnormal Pap test” (participant 5).

Several participants found the Pap test video to be a helpful resource for reducing anxiety (Table 3; participants 4 and 5). One participant suggested adding more content to ease anxiety around screening: “I think something that would be nice to be included is that there are different size speculums that you can even request at an OB/GYN appointment...comfort is so important...including information about that would be really nice” (participant 5).

Theme 3: Satisfaction—Gamification Was the Main Driver of User Satisfaction

Participants enjoyed the gamified aspects of the app, such as the avatar, trophies, and app response to correct or incorrect quiz answers. Participants often cheered if they got a lesson question correct. All participants who created an avatar mentioned something positive about this feature (Table 3; participant 5). Participants found the animal avatars to be cute and better at providing anonymity than human avatars or profile pictures. Seven participants responded positively to receiving trophies (Table 3; participants 6 and 7). Only 1 participant expressed a negative opinion: “I’m going to be honest that I personally don’t care about trophies...it really doesn’t make a difference to me at all” (participant 9).

Quantitative Study

A total of 24 people (including 9 think-aloud participants) enrolled in the quantitative study, but 1 person did not submit responses to any survey questions and was omitted from data analysis (Tables 1 and 2). Summary demographics include the 9 individuals who participated in both the qualitative and quantitative studies. Most participants identified as White and cisgender people; 1 participant identified as nonbinary gender; and all census race categories were represented except American Indian and Alaskan Native. Participants were highly educated (Tables 1 and 2).

The mean perceived ease of use score was 6.17 (SD 0.27), indicating the app was easy to use (Table 4). The mean perceived usefulness score was 4.94 (SD 0.22), suggesting the app would be moderately useful in increasing adherence to cervical cancer screening guidelines. Overall satisfaction with the app was high, with a mean satisfaction score of 6.12 (SD 0.40).

Discussion

Principal Results

Study participants were satisfied with the GLAm app, the education provided, and the gamified components of the app. The app was perceived as easy to use. The perceived usefulness of the app to increase cervical cancer screening uptake was more moderate, but the qualitative study results showed that it was useful for decreasing anxiety and fear associated with the Pap test. Notably, the SMS text message reminder for screening was not tested in this study. The brevity of app components was reinforced by participants who skipped reading educational explanations or truncated the 2.5-minute videos.

Comparisons With Previous Work

Similar to the results from our study, multiple studies of electronic game–based learning have shown high appeal and satisfaction with the game format [15-17]. While many reviews of mHealth studies to increase preventative health behavior have shown positive results, these studies combine multiple interventions, predominantly SMS text messaging, telephone calls, and social media messaging, with a smaller number of studies evaluating mHealth apps [7,18]. The gamified Fight HPV app comprises puzzles demonstrating complex scientific concepts such as how the immune system and various health behaviors impact the risk of human papilloma virus (HPV) infection and cancer [19]. A retrospective cohort study showed that downloading the app was associated with a greater than 2-fold increase in cervical cancer screening compared to age- and cancer screening history-matched controls [19]. This demonstrates the potential effectiveness of an app to increase cervical cancer screening but does not directly address barriers such as fear of the screening process itself, which was addressed in this app through text explanations, pictures, and videos.

Strengths and Limitations

Usability and satisfaction were evaluated using both qualitative and quantitative methodologies using validated survey questions. However, while scores on the Technology Acceptance Model predict usage, there is no score threshold to indicate “usefulness,” limiting the interpretation of results. Limitations also include selection bias inherent to the social media recruitment strategy [20,21]: parameters (age, geographic region, budget, and keywords) for promoted messages are entered into the social media platform, and a proprietary algorithm is used to target users who see the platform. Users self-select how they respond to the advertisement, which in our study resulted in a predominantly White and highly educated population; the app needs to be further tested in populations who historically have the lowest screening coverage. Last, the efficacy of the app to increase cervical cancer screening uptake was not tested in this study, and distribution of an effective app remains a challenge. During the qualitative portion of the study, 1 participant suggested the distribution of the app to adolescents through school programs.

Future Directions

To address the challenge of distribution of a health app, we decided to test the effectiveness of the app to increase preventative health behaviors within a childhood cancer survivorship clinic. The survivorship clinic was selected as this provided a way to distribute the app in a health care setting in which participants were not already presenting for the target preventative care. A revised version of the app using the same format and visual features tested for usability and satisfaction was created to encourage HPV vaccination among survivors of childhood cancer. A change in focus from cervical cancer screening to HPV vaccination was chosen to target the greatest number of participants in the survivorship clinic (upper age limit of 27 years; all genders). This is also an important aspect of survivorship care which is often not emphasized [22,23]. The efficacy of the app in increasing HPV vaccination initiation and completion is currently being tested in a randomized clinical trial.