This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.
HIV incidence among young adult men who have sex with men (MSM), particularly among black and Latino men, continues to rise. As such, continued HIV prevention interventions for young MSM of color are of utmost importance. Male Youth Pursuing Empowerment, Education and Prevention around Sexuality (MyPEEPS) Mobile is a comprehensive HIV prevention and sexual health education smartphone app initially created to promote sexual health and HIV prevention among adolescent sexual minority young men aged 13 to 18 years.
The objective of this study was to critically appraise the acceptability and usability of MyPEEPS Mobile for young adult MSM aged 19 to 25 years.
Study participants used the mobile app, completed usability questionnaires and in-depth interviews, and reported their experience using the app. Analysis of interview data was guided by the Unified Theory of Acceptance and Use of Technology (UTAUT) to better understand the usability and acceptability of this intervention for young adults. Interview data were coded using the following constructs from the UTAUT model: performance expectancy, effort expectancy, and social influence.
A total of 20 young adult MSM (n=10 in Chicago, Illinois, and n=10 in New York, New York) were enrolled in the study. Participants reported that MyPEEPS Mobile was free of functional problems (Health Information Technology Usability Evaluation Scale scores and Post-Study System Usability Questionnaire scores consistent with high usability), easy to use, and useful, with an engaging approach that increased acceptability, including the use of avatars and animation, and inclusive representation of the diverse identities by race and ethnicity, gender identity, and sexual orientation. Recommended areas for improving MyPEEPS Mobile for the target demographic included more adult-oriented graphics, advanced educational content, scenarios for youth with more sexual experience, and search function to increase accessibility of key content.
Overall, young adult MSM aged 19 to 25 years described the MyPEEPS Mobile as educational, informative, and usable for their sexual health education and HIV prevention needs, and they provided actionable recommendations to optimize its use and applicability for this age group.
Of the 38,739 new HIV diagnoses in the United States and dependent areas, young men, aged 13 to 24 years, accounted for 18% of new diagnoses of HIV in 2017 [
Sexual risk factors among YMSM identified in the literature include low rates of HIV testing; high rates of sexually transmitted infections (STIs), particularly among youth of color aged 20 to 24 years; substance use; as well as low levels of consistent condom and PrEP use [
According to the US 2020 National HIV/AIDS Strategy, public health initiatives should focus on increasing HIV and STI testing, linkage to care, universal viral suppression, and increasing PrEP awareness and access [
In consideration of the 2020 National HIV/AIDS Strategy, several culturally tailored mobile health (mHealth) technology interventions have been designed in areas of primary HIV prevention (eg, myDEx,
The
MyPEEPS Mobile is a mobile, responsive Website that is viewable on small screens and usable with touch screens. MyPEEPS Mobile provides educational information about STIs and HIV for YMSM, builds skills for condom use, and raises awareness of minority stress. The app content is guided by 4 peeps: Tommy, Philip, Nico, and Artemio, or composite characters/avatars, who relay content through comics, animation, and scenarios delivered through 21 brief activities in four sequential modules (see
In this study, we sought to assess the usability of MyPEEPS Mobile among an older age group with increased risk for HIV infection. To do so, we conducted in-depth interviews about the app content with young adult MSM aged 19 to 25 years.
This study was approved by the Institutional Review Board (IRB) at Columbia University Medical Center, which served as the single IRB of record. Data were collected in New York, New York, from August 2018 to October 2018 and in Chicago, Illinois, from December 2018 to February 2019. An a priori sample size of approximately 20 was estimated to provide saturation of acceptability and usability themes [
Participants met the study staff at each study site for a single visit, lasting approximately 3 to 4 hours. The interviews were conducted in private conference rooms at the Lurie Children’s Hospital and the Columbia University School of Nursing. After initial informed consent and completion of a computer-assisted self-interview, including demographic and behavioral questions, participants were instructed to complete intervention activities in the MyPEEPS Mobile app. While using MyPEEPS Mobile, participants recorded notes on their perceptions of app design and content. After completing the modules, participants completed the following usability questionnaires: (1) Health Information Technology Usability Evaluation Scale (Health-ITUES) [
The Unified Theory of Acceptance and Use of Technology (UTAUT) model was originally developed as a conceptual framework to explain individuals’ intention to adopt and use technological innovations. In this study, we draw from this theory to describe the applicability and likelihood of use of the MyPEEPS Mobile app among young adult MSM aged 19 to 25 years [
Performance expectancy is defined as the degree to which an individual believes that using the technology will help them attain gains in the outcome of interest, that is, work performance in the original conceptualization, and herein, health protective behavior. Effort expectancy is the degree of ease associated with use of the technology. Finally, social influence is the extent to which an individual perceives that important others believe they should use the technology. These constructs are theorized to drive behavioral intention, which leads to use behavior. That is, if users expect use of the technology to improve the outcome of interest, find it easy to use, and perceive that important others believe they should use it, use of the technology will follow. Several constructs that stem from organizational contexts used in the original model (ie, facilitating conditions and extrinsic motivation) were eliminated in this study, as they were not applicable because use of the app does not depend on an organizational structure to drive use.
In addition to the primary constructs, the UTAUT model also includes moderators that are theorized to impact the pathway from primary constructs’ technology use, based on individual characteristics. For this study, we sought to identify potential contextual factors that might also impact the pathway to use, based on analysis of interview data.
The qualitative analysis process consisted of two rounds of coding, including primary, open coding of emergent themes and secondary, thematic coding of those themes onto UTAUT constructs [
Initial content analyses via line-by-line coding, a form of open coding, was performed by the 2 coders independently to identify emergent codes [
The coders met after completion of secondary analyses to discuss the fit of their codes and any variances in the application to the UTAUT model. Cognitive mapping of secondary codes (development of a graphic map that represents the relationships between concepts) was then performed in NVivo to facilitate discussion, integration, and agreement for the development of a preliminary version of the adapted model. Moreover, 3 of the 4 macro-constructs were agreed upon and utilized in the adapted model to explain the applicability and likelihood of use of MyPEEPS among young adult MSM, including (1) performance expectancy, (2) effort expectancy, and (3) social influence. Several constructs of the UTAUT model (ie, facilitating conditions, extrinsic motivation, etc) were not apparent in the interview data, and thus, they were not applied in the adapted model. To facilitate credibility and reliability, the study team, comprising the principal investigator, primary coders, and the study coordinator, then convened for consensus coding of UTAUT-related content. Applicability of constructs and codes were discussed with any discrepancies among codes negotiated and reached by majority [
Adapted Unified Theory of Acceptance and Use of Technology model.
The study sample comprised 20 YMSM, aged 19 to 25 years, recruited from New York (n=10) and Chicago (n=10), with a median age of 22.5 years. Regarding sexual orientation, 90% (18/20) identified as gay, 5% (1/20) identified as bisexual, and 5% (1/20) identified as pansexual. In terms of race/ethnicity, 50% (10/20) identified as white, 35% (7/20) black/African American, 10% (2/20) multiracial, and 5% (1/20) other (Puerto Rican). Of these participants, 45% (9/20) had completed college, 25% (5/20) had completed some college, 20% (4/20) had completed high school, 5% (1/20) was currently enrolled in high school, and 5% (1/20) had completed a master’s degree.
Overall, participants rated the app as highly usable on the usability instruments. On the PSSUQ, all subscales, including system quality, information quality, and interface quality, fell below 2 on average, indicating high usability (see
Post-Study System Usability Questionnaire scores (N=20).
PSSUQa constructs | Scoreb, mean (SD) |
System quality | 1.49 (0.15) |
Information quality | 1.98 (0.42) |
Interface quality | 1.48 (0.10) |
aPSSUQ: Post-Study System Usability Questionnaire.
bScores range from 1=strongly agree to 7=strongly disagree.
Health Information Technology Usability Evaluation Scale scores.
Health-ITUESa constructs | Scoreb, mean (SD) |
System impact | 4.37 (0.24) |
Perceived usefulness | 4.29 (0.16) |
Perceived ease of use | 4.67 (0.08) |
User control | 4.38 (0.28) |
aHealth-ITUES: Health Information Technology Usability Evaluation Scale.
bScores range from 1=strongly disagree to 5=strongly agree.
Qualitative findings are arranged by the constructs of the UTAUT model. Quotes from study participants are presented below followed by their city of residence and their age in years.
Participants described their intention to use the app within their everyday lives. In particular, one participant said:
I think it’s applicable to anyone’s life who is sexually active. Obviously, the messaging was catered to a gay demographic. So, identifying as a gay male, I did find it applicable to my life; more specifically, my sex life. So, yeah, I did think that the information that was shared is something that I can use in my everyday life.
Performance expectancy and plans to use the MyPEEPS app were further operationalized through the three subconstructs of the UTAUT model: (1) perceived usefulness, (2) outcome expectations, and (3) relative advantage [
I feel like even I learned something new today, so we could all learn something new.
Participants expressed a foreseeable usefulness of MyPEEPS for YMSM who live in different geographic regions (ie, outside of urban settings):
Obviously, in more conservative areas you still have like very strong conservative values on that. Like, the portion of sex ed that dealt with gay sex, was like five minutes, you know. So, I think it would be applicable for people who don’t have the same type of sex education because there’s still plenty of places that their sex education is still abstinence first, which is just not reality, you know.
On the other hand, participants who had more educational or life experience perceived the app as less useful. For example, one user noted:
For me it felt redundant because it didn’t change mine [bottom line] at all, but maybe for someone who is younger or has less sexual experience or education it could still...it could be helpful for them.
Another participant commented that the app was less useful to him because of his educational background:
Coming from a student background, especially in a school of public health, this is an activity that we had done before, so it was kind of a little bit repetitive for me just because I have seen these questions before, and I know what we’re doing here.
When probed by the interviewer regarding usefulness of the app, one end user commented:
So, I guess it’s important to clarify – what is the purpose of the app? Is it to be a one-time resource, or is it to be a go-back-to-it resource?
Me and my lover have to plan. I know I will be looking at the app. When I get home, I am going to tell him all about it.
Furthermore, use of culturally representative avatars and animation surpassed participants’ expectations relative to expected traditional sexual health didactic content. One participant stated:
I really like “Testing with Tommy” where it literally...you go through what it’s like to go to an HIV clinic and go through that experience. Like, you take a number, and there are a bunch of people there. It’s like it’s not a big deal.
One of the major expectations “not met” by participants was the availability and accessibility of sexual health resources. A representative comment about this was as follows:
I really wish that all the references and everything could have been provided somewhere versus going back inside the modules.
Although sexual health–related external references and hotlines were provided in activities throughout the app, this did not meet end user’s expectation of having a single “go-to” resource tool, which was a content recommendation.
I think it’s also nice because it’s very educational and kind of starts easy, like you don’t have to have any prior knowledge, like any sexual education, before. The app really provides you with a lot of baseline sexual education, which is really important.
Acknowledging the individual variability in sexual experience, end users found the app to be useful for those who have less sexual experience or knowledge:
I thought it did a great job about...talking about the bottom line, like standing by the things that are important to you. I could see as someone who didn’t have much information about sexual education and maybe didn’t know what the risks of different sexual activities were they would get more information and that would change their bottom line.
Effort expectancy is defined as the degree of ease associated with the use of MyPEEPS, including
I felt the flow of the app was really easy to follow and set up in a very nice way.
Most end users found the overall interface of MyPEEPS to be intuitive, approachable, and easy to use:
Well it’s pretty straightforward and easy to use. The logo is cute.
The MyPEEPS activity map was designed as an urban city street, reflecting participant’s environment. Overall, participants at these study sites related to the map design. Reflecting on the ease of navigation, one participant noted:
I loved the platform of scrolling through the street. I thought that was really cool because you can see what’s still to come. While they’re only brief little titles you can still see what’s going on, you can see there is going to be a point in which you’re going to see the experience of a clinic and things of that sort that I enjoyed. It was, of course, really easy to go back and click on other things if I did want to go back.
On the other hand, one user noted:
I didn’t like how it was so linear.
Furthermore, although there were very few negative comments about the gamification aspects of the app, most end users found activity 18 (
With 18 the thing started shaking so I couldn’t read the thing properly. I think they should change that. If you ever have it shake, don’t have it shake too much.
App activity with the screen shaking to simulate being under the influence of drugs or alcohol.
Social influence was applied in this study as the degree to which an individual perceives that important others believe that they would benefit from the use of MyPEEPS app [
Me and my lover have to plan. I know I will be looking at the app. When I get home, I am going to tell him all about it.
Conversely, other participants noted that they would not share this app or content with their partner(s) or friend(s) because of embarrassment, implied stigma, or discrimination:
I feel like for friends I would be okay telling them. If it’s like a partner I might be embarrassed. It seems a little bit like I don’t know what I am doing so I have to use this app to help support me make all my decisions. For friends I feel like I’d be more open to share that with them.
Social factors are the end user’s perspective of their subjective culture (ie, YMSM) and physical environment [
Participants’ reported use of the app was influenced by their physical and social environment and sense of privacy while completing app activities. Privacy was a salient concern because of the sensitivity of sexual health content and associated concern for being “outed”:
Also, the video was really blunt and direct, and so if I was not listening to that without headphones...it was like not safe for work kind of thing. And if my parents were around, I would feel super uncomfortable. Yeah. And just like making sure that I have privacy. I mean, this stuff is pretty blunt. And I mean, that’s good, but at the same time being aware of who’s around is kind of important.
Most end users found the applicability of the content depending on age of the user. A representative quote is as follows:
Yes, and I think specifically younger adults would benefit from this. If I were a college student still in my freshman year, this would be perfect for some sort of training. I think it would be really, effective for young people who are going out into the world, like 18/19 years old...It gives you strategies. I think that a lot of young people probably don’t get the sex education they need until college, and not everyone goes to college.
Similarly, some participants commented that the scenarios are most relevant for those still living with parents or caregivers:
It wouldn’t really be as relevant if I'm in my mid-twenties...because you are in different environments. You may not be living with your parents...and you're navigating different other social circles, and networking in other communities. Some of the scenarios could be transferred over.
Another participant reflected:
I think maybe for the younger crowd; the visual style can maybe be updated, to be a little more grown-up.
Some end users commented about the app’s graphics being tailored to a younger audience, using descriptors such as “juvenile,” “cartoonish,” and “elementary.” One participant commented on the following for improving the interface for young adults aged 19 to 25 years:
It’s just tricky, because I appreciated the creativity there, but it did feel a little elementary.
Participants discussed the variability of gender expression among YMSM, which was reflected in the study sample, including participants who described themselves as nonbinary, two-spirit, masculine of center, and feminine of center. One participant noted:
Something I also appreciated about the app was it was representative of a lot of different identities that’s you could have, and it also talked about the intersection of identities, like gender identity and sexual orientation and race and how all those can play together and affect someone’s experience.
Thus, the match of key characteristics in the MyPEEPS Mobile app is likely to resonate with participants.
In addition, several participants noted the desire for a narrative or an avatar who was in the process of “coming out,” amid the stages of disclosing one’s sexuality and/or gender. For instance:
How would you address coming out to parents, or addressing to your friends that I think I might be into guys or girls or look, I have these feelings, like how to navigate those effective communicating strategies.
Most participants related to language used in the delivery of content:
Very sex positive, for sure, using vocabulary that we use; And this is all really very accessibly worded...if that’s a word...simply worded, simply put.
In contrast, several participants did not relate to the use of vernacular language:
I felt like, at that moment, some of the vocabulary I thought was intrusive.
Overall, participants found the language used by avatars during quizzes and in case scenarios to be appropriate.
As mHealth interventions become increasingly available for consumers, it is critical to ensure that mobile technologies are designed and targeted to meet end users’ needs [
Overall, end users of this study found the mobile app to be highly usable, as indicated through the survey data (Health-ITUES and PSSUQ), with no major bugs or functional problems reported and no issues with flow of activities. However, the qualitative analyses, based on the UTAUT framework [
The basic sexual health information in the app was deemed useful overall, with limitations for those with preexisting sexual health knowledge or sexual experience. The app was perceived to be most useful among those with limited sexual health knowledge and experience. To optimize content for a broader group of young adult MSM, more advanced educational content and social scenarios may need to be added.
Users reported the relative advantage of the MyPEEPS app over standard sexual health approaches because of the use of avatars and animation to aid in the understanding and absorption of content. The use of the avatars also provided a relative advantage, providing material that is salient and relatable to YMSM.
Regarding the UTAUT construct of effort expectancy, the app menu, which was depicted as an urban city street in which participants advanced along activities sequentially, moving horizontally along the street scene, was largely perceived as both easy to navigate and interesting. Although there were no functional problems with app activities, one activity in which the screen shakes to simulate intoxication was perceived as distracting, with the related text difficult to read, limiting its effectiveness.
Social influence encompassed subconstructs including social factors and subjective norms. Although many participants lauded the “sex positive” nature of the app, basic educational information, and frank language about sexual behavior, they found that these may result in discomfort on the part of the user to complete the activities in front of others because of the stigma associated with the content, reflecting the subjective norms of those around them. Although users are encouraged to use the app in private, this was reinforced given these comments.
The participants were universally enthusiastic about communicating and recognizing different gender presentations and identities in the app and the recognition of intersecting multiple identities, including gender, race/ethnicity, and sexual orientation. Participants recommended a case scenario of an avatar who is in the process of “coming out” disclosure of one’s gender and/or sexuality. Although “coming out” is a common thread in the community, the process of disclosing one’s sexual and gender identity is dependent on one’s circumstance and social context and intersecting social identities [
Although there are privacy and security concerns with the use of mobile technology for storing personal health information [
Our study presents some limitations. First, the purposive sample of young adult MSM from urban settings may not generalize to suburban or rural settings and individuals therein. In addition, this study did not include transgender men (assigned female at birth) who have sex with men, and therefore, findings cannot be generalized to that group.
Critical to the uptake and use of eHealth interventions is the rigor in which they are appraised before implementation [
Screenshots from the Male Youth Pursuing Empowerment, Education and Prevention Around Sexuality Activities app.
Health Information Technology Usability Evaluation Scale
Institutional Review Board
mobile health
men who have sex with men
Male Youth Pursuing Empowerment, Education and Prevention around Sexuality
pre-exposure prophylaxis
Post-Study System Usability Questionnaire
sexually transmitted infection
Unified Theory of Acceptance and Use of Technology
young men who have sex with men
Research reported in this publication was supported by National Institute of Minority and Health Disparities of the National Institutes of Health under award number U01MD011279 and the National Institute of Nursing Research under award number K24NR018621. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. BG was supported by the NIMH Center Grant (P30-MH43520; PI: Robert H Remien, PhD) and NIMH training grant (T32-MH19139 Behavioral Sciences Research in HIV Infection; PI: Theodorus GM Sandfort, PhD).
None declared.