This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2491) HIV/AIDS/STI Prevention and Care (882) mHealth for Treatment Adherence (404) Web-based and Mobile Health Interventions (3358) Formative Studies and eHealth/mHealth Development (527) mHealth for Wellness, Behavior Change and Prevention (2922) Engagement with and Adherence to Digital Health Interventions, Law of Attrition (505) User-centred Design Case Studies (602)

Published on in Vol 5, No 8 (2021): August

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/26861, first published .
A Mobile Health App (WYZ) for Engagement in Care and Antiretroviral Therapy Adherence Among Youth and Young Adults Living With HIV: Single-Arm Pilot Intervention Study

A Mobile Health App (WYZ) for Engagement in Care and Antiretroviral Therapy Adherence Among Youth and Young Adults Living With HIV: Single-Arm Pilot Intervention Study

A Mobile Health App (WYZ) for Engagement in Care and Antiretroviral Therapy Adherence Among Youth and Young Adults Living With HIV: Single-Arm Pilot Intervention Study

Authors of this article:

Parya Saberi1 Author Orcid Image ;   Nadra E Lisha2 Author Orcid Image ;   Xavier A Erguera1 Author Orcid Image ;   Estie Sid Hudes1 Author Orcid Image ;   Mallory O Johnson1 Author Orcid Image ;   Theodore Ruel3 Author Orcid Image ;   Torsten B Neilands1 Author Orcid Image
Article Authors Cited by (10) Tweetations (26) Metrics

    Original Paper

    1Department of Medicine, University of California, San Francisco, San Francisco, CA, United States

    2Center for Tobacco Control Research and Education, University of California, San Francisco, San Francisco, CA, United States

    3Department of Pediatrics, University of California, San Francisco, San Francisco, CA, United States

    Corresponding Author:

    Parya Saberi, PharmD, MAS

    Department of Medicine

    University of California, San Francisco

    550 16th street

    San Francisco, CA, 94143

    United States

    Phone: 1 415 502 1000 ext 17171

    Email: Parya.Saberi@ucsf.edu


    Background: Youth are globally recognized as being vulnerable to HIV. Younger age has been correlated with worse health outcomes. Mobile health (mHealth) interventions have the potential to interact with youth where they are, using a device they already access.

    Objective: Using predefined benchmarks, we sought to evaluate the feasibility and acceptability of WYZ, an mHealth app, for improved engagement in care and antiretroviral therapy (ART) adherence among youth and young adults living with HIV. WYZ was designed and developed with input from youth and young adults living with HIV using a human-centered design approach and was based on the information, motivation, and behavioral skills framework to address common barriers to care and ART adherence among youth and young adults living with HIV.

    Methods: We recruited youth and young adults living with HIV (18-29 years old) from the San Francisco Bay Area to take part in a 6-month pilot trial. Their participation included completing baseline and exit surveys, and participating in seven phone check-ins about their use of WYZ.

    Results: Youth and young adults living with HIV (N=79) reported high levels of feasibility and acceptability with WYZ use. We met predefined benchmarks for recruitment (79/84, 94%), mean logins per week (5.3), tracking ART adherence (5442/9393, 57.9%), posting chat topics per week (4.8), and app crashes reported per week (0.24). The ease of app download, install, and setup, and comfort with security, privacy, and anonymity were highly rated (all over 91%). Additionally, participants reported high satisfaction for a research project that was remotely conducted. Participants used the app for shorter timeframes compared to the predefined benchmark.

    Conclusions: We noted high feasibility and acceptability with WYZ. Further research to examine the efficacy of WYZ will enable youth and young adults living with HIV and their providers to make informed decisions when using, recommending, and prescribing the app for improved engagement in HIV care and ART adherence.

    Trial Registration: ClinicalTrials.gov NCT03587857; https://clinicaltrials.gov/ct2/show/NCT03587857

    JMIR Form Res 2021;5(8):e26861

    doi:10.2196/26861

    Keywords

    youth living with HIV; mobile health; mobile app; engagement in care; antiretroviral therapy adherence; pilot 


    In the United States, youth and young adults carry a significant burden of HIV. Youth and young adults living with HIV experience disparities at all steps of the HIV care continuum, including higher HIV incidence, lower linkage and retention in care, suboptimal antiretroviral therapy (ART) adherence, and lower virologic suppression [1-6]. The consequences of continued disparities include poor health outcomes, development and transmission of drug-resistant viruses, a future generation of adults who are more susceptible to developing AIDS, and further widening of these health disparities. Youth and young adults living with HIV experience many individual, structural (eg, transition to adult health care, inexperience with medical systems, and lack of insurance), social (eg, poverty, unstable housing, food insecurity, social isolation, and stigma), and biological (cognitive developmental stages) challenges that impact their abilities to access and adhere to oral ART [2,3,7-9]. However, there are few effective and tailored interventions that address ART adherence and engagement in HIV care for youth and young adults living with HIV.

    In the United States, over 96% of youth and young adults living with HIV own smartphones [10], over two-thirds have downloaded mobile health (mHealth) apps [11], and over 90% are social media users [12]. The nearly ubiquitous access to and use of smartphones represents a powerful platform for the delivery of mHealth interventions to this population. Additionally, given the reduction in transportation costs, time constraints, potential stigma associated with participation in in-person HIV research [13], and missing data, mHealth technology can surmount common barriers, increasing the reach and generalizability of findings. Several mHealth apps are in various stages of development for people living with HIV [14-17], as we have previously summarized [18]. However, despite technology-based behavioral interventions showing promise in older adults living with HIV [19], few interventions have shown efficacy in addressing the unique aspects of youth developmental phases, youth culture, and gravitation of youth toward the use of technology [20]. In this study, we pilot tested an mHealth app to address barriers to engagement in care among youth and young adults living with HIV.


    Study Design and Sample

    From July 2019 to May 2020, we conducted a 6-month single-arm pilot study to evaluate the feasibility and acceptability of an mHealth app, named WYZ (pronounced “wise”), to address barriers to engagement in HIV care among individuals aged 18 to 29 years living with HIV in the San Francisco Bay Area [18]. WYZ was designed and developed using a human-centered design (HCD) approach [21-24]; formative research with youth and young adults living with HIV [18,20,25-27]; the information, motivation, and behavioral skills (IMB) [28-30] framework; and mHealth designers and developers from the University of California, San Francisco (UCSF) School of Medicine Technology team (SOM Tech). HCD focuses on creating approaches and delivering solutions to problems based on efforts to understand the specific needs and perspectives of the users. Therefore, HCD seeks to gain insights into the needs of the beneficiaries of an innovation, and creates approaches and delivers solutions to meet their needs.

    Details of WYZ design and development, as well as the pilot study protocol, have previously been published [18]. In short, WYZ contains three main features, My Health, My Team, and My Community. My Health allows users to keep track of their ART medication information, visualize their adherence and laboratory data, and understand their health; My Team provides community resources and facilitates communication with health care team members; and My Community allows for social support from peers through anonymous and moderated discussion forums and allows users to stay up-to-date on health-related news. These features were developed with guidance from youth and young adults living with HIV and further refined through focus groups with youth and young adults living with HIV and iterative field testing with our Youth Advisory Panel (YAP), and were chosen to address specific barriers to ART adherence and engagement in HIV care (eg, social isolation and lack of community support).

    WYZ design, development, and technological support were provided by UCSF’s SOM Tech. To ensure Health Insurance Portability and Accountability Act (HIPAA) compliance, we used Salesforce as the backend service and for storing sensitive data in a secure cloud-based database. Data about app usage were collected using Flurry (a mobile analytics tool) and Salesforce analytics. To enhance the security and privacy of WYZ, we used a two-step authentication process for downloading, password protection (with each log in), aliases, deletion of all communications over 30 days old, and remote revocation of app access in case of theft, loss, or misuse.

    Participants were recruited using various strategies, including flyers at clinics and community-based organizations, emails to clinicians at clinics serving youth and young adults living with HIV, peer referral, and contacting prior study participants who had consented to being notified of future research. Information about the study was also disseminated through the YAP.

    Individuals aged 18 to 29 years living with HIV, who lived or received medical care in the San Francisco Bay Area, spoke English, and had access to an Android or iOS smartphone, were included. Those with any evidence of severe cognitive impairment or active psychosis that impeded their ability to provide informed consent were excluded. To confirm an individual’s age and HIV serostatus, the potential participant text messaged a photo identification showing their date of birth and either a clinician’s letter of HIV diagnosis, a copy of laboratory test results (for HIV antibody or HIV viral load), or their ART medication vial. These photos were sent via text message to an encrypted and secure study phone for verification by study staff.

    All study activities, including recruitment, screening, enrollment, study assessments, provision of incentives, and exit interviews, were conducted remotely using text message, telephone, email, and videoconference. Participants received a check-in at weeks 1, 2, and 4, followed by monthly check-ins, and up to US $215 for completion of all study activities. All procedures were reviewed and approved by the UCSF Institutional Review Board with a requirement for electronic consent. At baseline and 6 months, participants completed study assessments using a Qualtrics survey.

    Measures

    Demographics

    Demographic data, including date of birth, sex at birth, sexual identity, race/ethnicity, perceived financial security, and work status (full time, part time, or not working), were collected.

    Feasibility Metrics

    Feasibility metrics were collected using Flurry and Salesforce analytics. Metrics were based on predefined thresholds [18], including how many people were recruited for the study, mean logins to the app, mean minutes in the app, and use of specific features in the app.

    Acceptability Metrics

    Acceptability metrics were collected using a Qualtrics survey administered during the last study visit at 6 months. The survey included questions related to satisfaction with WYZ, ease of WYZ use, and satisfaction with the study. Additionally, we asked participants about WYZ acceptability using the System Usability Scale (SUS), with scores ranging from 0 to 100 and scores >68 being considered above average [31,32]. A threshold of 70% or greater satisfaction on all questions was used to determine acceptability.

    HIV and Psychosocial Outcomes

    HIV and psychosocial outcomes were measured at baseline and 6 months. These included self-reported HIV viral load (detectable or undetectable) [33], self-reported ART adherence [34], depression (Patient Health Questionnaire-9 [PHQ-9]) [35], resilience [36], social provisions [37], social isolation (Patient-Reported Outcomes Measurement Information System [PROMIS]) [38], health care empowerment [39], and unmet subsistence needs and instrumental support [40].

    Data Analysis

    Descriptive statistics of the baseline demographics of WYZ study participants were calculated. Next, we examined descriptive statistics for feasibility metrics and compared them to predefined benchmarks. We then calculated frequencies for all of our acceptability metrics measured at the exit survey. Lastly, frequencies for HIV and psychosocial outcomes were calculated at baseline and the exit survey (6 months). For these data evaluated at both baseline and 6 months, we compared data from those who were retained in the study until 6 months and the entire group to examine divergent results. Given that this was a pilot study with limited statistical power, based on guidance from the National Institutes of Health and literature regarding wide confidence intervals and instability of effect sizes from pilot studies, tests of statistical significance and the efficacy of the intervention to compare HIV clinical outcomes preintervention and postintervention were not evaluated [41-44]. All analyses were completed using SAS 9.2 (SAS Institute).


    Demographics

    Demographics of the study participants are presented in Table 1. At baseline, there were 79 participants (mean age 26.9 years, SD 2.9 years). Among the 79 participants, 69 (87%) identified as male, 48 (61%) identified as gay, 33 (43%) identified as Latino, and 16 (21%) identified as Black. Although nearly 57% (45/79) of participants were working, financial insecurity was relatively common, as 62% (49/79) of participants noted “barely getting by” or “not getting by” on the money they have.

    Feasibility metrics are presented in Table 2. Of the 92 individuals who were screened, 84 were eligible. Of these 84 individuals, 79 (94%) consented to participate in the study, and 69 (87%) of those who enrolled completed the exit survey at 6 months. All predefined benchmarks were met (Table 2), except for mean minutes in the app per week (benchmark=15 min/week, actual=8.7 min/week). The mean number of logins per week was 5.3 (SD 5.6). In My Health, ART adherence tracking was conducted in 57.9% (5442/9393) of the inquiries. Moreover, the mean number of postings of chat topics on the My Community chat per person per week was 4.8 (range, 1-42), and the number of reported app crashes was less than once per week (0.24).

    Table 1. Baseline demographics of the study participants.
    DemographicValuea (N=79)
    Age (years), mean (SD)26.9 (2.9)
    Sex at birth, n (%)

    		Male69 (87)

    		Female9 (11)

    		Decline to answer1 (1)
    Sexual identity, n (%)

    		Gay48 (61)

    		Heterosexual8 (10)

    		Other23 (16)
    Race/ethnicity, n (%)

    		Asian3 (4)

    		Black16 (21)

    		Latino33 (43)

    		Other13 (17)

    		White11 (15)
    Financial security, n (%)

    		I have enough money to live comfortably24 (30)

    		I can barely get by on the money I have40 (51)

    		I cannot get by on the money I have9 (11)

    		Decline to answer6 (8)
    Work status, n (%)

    		Full time28 (35)

    		Part time17 (22)

    		Not working28 (36)

    		Other4 (5)

    		Decline to answer2 (3)

    aMissing n ranged from 0 to 6 for each item.

    Table 2. Feasibility metrics, prespecified threshold for each metric, and actual outcome.
    Metric and thresholdActual value
    General

    		Screened, N92

    		Eligible (N=92), n (%)84 (91)

    		Recruited (N=84) (threshold: ≥70%, target N=80 [ie, ≥56]), n (%)79 (94)

    		Logins (threshold: 1 login/week), mean (SD)5.3 (5.6)

    		Minutes in the app (threshold: 15 min/week), mean (SD)8.7 (5.0)
    My Health

    		ARTa adherence tracking (threshold: ≥3 times/week [ie, ≥43%]), mean/week58%
    My Community

    		Post chat topic (threshold: 1 chat topic/person/week), mean number/person/week (range)4.8 (1-42)
    Administrative

    		Reported app crashes (threshold: <1 app crash/week), mean number/week0.24

    aART: antiretroviral therapy.

    Acceptability metrics are presented in Table 3. Among the 69 participants who completed the study, 77% (n=53) rated their overall experience with the app as excellent to very good, 91% (n= 63) reported the app to be extremely to somewhat easy to download and install, and 96% (n=66) reported that WYZ setup was extremely to somewhat easy. All participants reported being extremely to somewhat comfortable with the security, privacy, and anonymity of WYZ. Moreover, approximately 83% (n=57) stated that they would be extremely to somewhat likely to continue to use WYZ and 94% (n=64) were extremely to somewhat likely to participate in a similar study in the future. Furthermore, 86% (n=59) of participants rated their overall experience with participation in the WYZ study as excellent to very good and 90% (n=62) reported excellent to very good experience with participating in a completely remotely conducted study. The mean SUS score was 75.6, which is considered to be well above average.

    HIV and psychosocial metrics are presented in Table 4. At baseline and 6 months, 9% (7/79) and 4% (3/69) of participants, respectively, reported a detectable HIV viral load. During this time, self-reported ART adherence was unchanged. From baseline to 6 months, participants reporting mild depressive symptoms decreased by 9% (30/79, 38% to 19/66, 29%). Moreover, the mean social isolation score decreased by 12.1 points. Overall, we did not note divergent patterns with regard to the HIV and psychosocial metrics between baseline data from the entire sample (N=79) and those who were retained until 6 months (N=69).

    Table 3. Acceptability metrics exit survey findings.
    MetricsValuea (N=69), n (%)
    How would you rate your overall experience with the WYZ app?

    		Excellent to very good53 (77)

    		Good0 (0)

    		Fair2 (3)

    		Poor to very poor14 (20)
    How easy or difficult was it to download and install WYZ on your phone?

    		Extremely to somewhat easy63 (91)

    		Extremely to somewhat difficult6 (9)
    How easy or difficult was it to setup WYZ (ie, create a personal passcode, set med reminders, enter your pharmacy information, etc)?

    		Extremely to somewhat easy66 (96)

    		Extremely to somewhat difficult3 (4)
    How satisfied were you with the visual design of WYZ?

    		Extremely to somewhat satisfied56 (81)

    		Extremely to somewhat unsatisfied12 (19)
    Overall, how helpful was the My Health module in supporting your ability to take your medications as prescribed?

    		Extremely to very helpful54 (81)

    		Moderately helpful7 (10)

    		A little to not at all helpful6 (9)
    How satisfied were you with medication and refill reminders?

    		Extremely to somewhat satisfied50 (78)

    		Extremely to somewhat unsatisfied14 (22)
    How satisfied were you with the adherence calendar?

    		Extremely to somewhat satisfied57 (88)

    		Extremely to somewhat unsatisfied8 (12)
    How satisfied were you with the lab visualization tools (ie, CD4 graph and viral load graphs)?

    		Extremely to somewhat satisfied41 (82)

    		Extremely to somewhat unsatisfied9 (18)
    Overall, how helpful was the My Team module in supporting your ability to seek and connect to support and services that you need?

    		Extremely to very helpful40 (62)

    		Moderately helpful18 (28)

    		A little to not at all helpful7 (11)
    How satisfied were you with the listing of community resources (ie, list of local resources and organizations in your area)?

    		Extremely to somewhat satisfied64 (100)

    		Extremely to somewhat unsatisfied0 (0)
    How satisfied were you with the secure messaging feature with your health care team (ie, being able to send a message to your health care team through WYZ)?

    		Extremely to somewhat satisfied45 (82)

    		Extremely to somewhat unsatisfied10 (18)
    How would you rate your comfort level with asking your health care providers to support your participation in the study by agreeing to be added to My Team?

    		Extremely to somewhat comfortable51 (88)

    		Extremely to somewhat uncomfortable7 (12)
    Overall, how helpful was the My Community module in supporting your ability to feel connected to other youth living with HIV?

    		Extremely to very helpful34 (52)

    		Moderately helpful20 (30)

    		A little to not at all helpful12 (15)
    How satisfied were you with the news feature within WYZ (ie, being able to see the latest HIV and health news in WYZ)?

    		Extremely to somewhat satisfied52 (84)

    		Extremely to somewhat unsatisfied10 (16)
    How satisfied were you with the private calendar (ie, being able to add appointments and community events within WYZ)?

    		Extremely to somewhat satisfied44 (70)

    		Extremely to somewhat unsatisfied19 (30)
    Overall, how helpful was the private calendar in supporting your ability to keep your health appointments?

    		Extremely to very helpful35 (56)

    		Moderately helpful14 (20)

    		A little to not at all helpful13 (23)
    What is your comfort level with the security, privacy, and anonymity provided by WYZ?

    		Extremely to somewhat comfortable67 (100)

    		Extremely to somewhat uncomfortable0 (0)
    How likely are you to continue to use WYZ after your participation in the study ends?

    		Extremely to somewhat likely57 (83)

    		Extremely to somewhat unlikely12(17)
    In the future, how likely would you be to participate in a similar study where you are asked to use a mobile health app like WYZ on a regular basis?

    		Extremely to somewhat likely64 (94)

    		Extremely to somewhat unlikely4 (6)
    How would you rate your overall experience with participation in the WYZ study?

    		Excellent to very good59 (86)

    		Good10 (15)

    		Fair0 (0)

    		Poor to very poor0 (0)
    How would you rate your experience participating in a study where everything was conducted remotely (ie, you did not have to come into a clinic or office to complete study activities)?

    		Excellent to very good62 (91)

    		Good5 (7)

    		Fair1 (1)

    		Poor to very poor0 (0)
    How would you rate your experience with having to schedule and complete regular check-ins over the phone with a study coordinator?

    		Excellent to very good56 (82)

    		Good10 (15)

    		Fair2 (3)

    		Poor to very poor0 (0)
    How would you rate your experience with receiving compensation for your study participation using a reloadable debit card (ie, ClinCard)?

    		Excellent to very good60 (87)

    		Good6 (9)

    		Fair3 (4)

    		Poor to very poor0 (0)
    How easy or difficult was it to remember to use WYZ regularly for the 6 months of your study participation?

    		Extremely to somewhat easy53 (77)

    		Extremely to somewhat difficult16 (23)
    How helpful was the communication with study staff?

    		Extremely to very helpful66 (96)

    		Moderately helpful2 (3)

    		A little to not at all helpful1 (1)

    aMissing n ranged from 0 to 14 for each item.

    Table 4. HIV and psychosocial outcomes at baseline and 6 months.
    OutcomeBaseline (N=79)Baseline (N=69)a6 months (N=69)
    HIV viral load, n (%)



    		Detectable7 (9)6 (9)3 (4)

    		Undetectable70 (89)59 (88)60 (87)

    		Do not know2 (3)2 (3)2 (3)

    		Decline to answer0 (0)0 (0)4 (6)
    ARTb adherence, mean (SD)85.0 (16.6)86.5 (15.3)85.0 (16.9)
    Depression (PHQ-9c), n (%)

    		No to minimal depression (0-4)22 (28)17 (25)26 (39)

    		Mild depression (5-9)30 (38)26 (38)19 (29)

    		Moderate depression (10-14)12 (15)10 (15)12 (18)

    		Severe to moderately severe depression (15-27)15 (19)14 (21)9 (14)
    Resilience, mean (SD)3.6 (0.7)3.5 (0.8)3.6 (0.7)
    Social support, mean (SD)37.9 (6.0)23.1 (5.3)37.7 (7.6)
    Social isolation, mean (SD)50.0 (10.0)35.4 (12.3)37.9 (7.0)
    Health care empowerment, mean (SD)17.4 (3.1)18.6 (3.3)17.4 (3.3)
    Relationship with health care provider, mean (SD)1.4 (0.6)1.4 (0.6)1.40 (0.6)
    Unmet subsistence needs and instrumental support score, mean (SD)0.61 (1.2)0.65 (1.2)0.67 (1.2)
    Unmet subsistence needs and instrumental support, n (%)



    		054 (70)46 (70)37 (65)

    		111 (14)9 (14)11 (19)

    		26 (8)4 (6)3 (5)

    		≥37 (8)7 (11)6 (11)

    aBaseline data for participants who were retained until the end of the study (6 months).

    bART: antiretroviral therapy.

    cPHQ-9: Patient Health Questionnaire-9.


    Principal Findings

    The use of WYZ was highly feasible and acceptable among youth and young adults living with HIV in the San Francisco Bay Area. We met predefined benchmarks for recruitment, mean logins per week, tracking ART adherence, posting chat topics, and app crashes reported. The ease of app download, installation, and setup, and the overall comfort with security, privacy, and anonymity were highly rated. Additionally, participants reported high satisfaction for a research project that was remotely conducted. These findings demonstrate high potential for uptake and app functionality, indicating a promising role for WYZ as an intervention for engagement in HIV care and ART adherence among youth and young adults living with HIV.

    Participants used the app for shorter timeframes than were predefined; however, our predefined benchmark may have been an overestimate. Additionally, due to the ability to log ART adherence using out-of-app notifications, some of the interactions with WYZ were not captured in the analytical tools used. In the next phase of this study, we will ask participants to further elaborate about app use during exit qualitative interviews.

    Small changes in self-reported HIV and psychosocial metrics from baseline to 6 months highlight the limitations of pilot studies, in which examination of the intervention’s “preliminary impact” is not meaningful due to wide confidence intervals [41-44]. However, we noted improvements in the social isolation score, which, along with the high level of activity in the My Community Chat section, underscore the importance of this feature and deserve further evaluation in future research.

    There are currently few mHealth apps in the early stages of development and pilot testing for enhanced engagement in HIV care, ART adherence, and communication with health care teams for people living with HIV [14-17]. The limitations of some of these mHealth apps include lack of specification of a theoretical framework, limited feasibility and acceptability metrics with no predefined benchmarks, small sample size (N<30), wide age range (≥18 years), and availability for either iOS or Android (not both). We have previously summarized these studies [18]. In developing and pilot testing WYZ, we have addressed these limitations.

    In this pilot study, we were able to recruit a diverse group of participants with regard to race/ethnicity; however, participants were mainly gay cis-gender men. The other limitations of our study include a single-arm design (ie, no control group) and a relatively small convenience sample of participants from the San Francisco Bay Area who had access to a smartphone and most of who had an undetectable HIV viral load; therefore, study findings may not be generalizable to other populations. The loss to follow-up was approximately 13%, which is lower than estimates among youth and young adults living with HIV in the HIV Research Network (20%) [45] and in other studies in this population (up to 55%) [46]. We believe that the relatively low loss to follow-up may have been due to the fact that this research was conducted completely remotely, which allowed for flexibility for participation. Since the completion of this pilot study, we have resolved all minor bugs and smartphone compatibility challenges. Additionally, we are updating My Health for those who may use long-acting injectables in the near future and the My Team resources section based on user geolocation.

    Conclusion

    Youth and young adults living with HIV represent a population that is disproportionately impacted by HIV and requires tailored youth-friendly interventions. There is a dearth of technology-based interventions that address the changing needs of youth and young adults living with HIV. In future research, we will examine the efficacy and effectiveness of WYZ in improving engagement in HIV care and ART adherence among a larger sample of youth and young adults living with HIV taking into account findings from this study. Given the speed of technological advancement and the need for evidence-based solutions for improved HIV health outcomes among youth and young adults living with HIV, we believe that more funding should be allocated to technology-based interventions to move the National Institutes of Health’s Behavioral and Social Sciences Research agenda forward.

    Acknowledgments

    The authors’ work was supported by the National Institute of Mental Health (award number R34MH114604). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Authors' Contributions

    PS conceptualized the study and received grant funding. PS and XAE collected the data. PS, ESH, and NEL ran the analyses. PS and NEL wrote the first draft of the study. TBN guided the design study and analysis plan. TR and MOJ guided grant funding and study design. All authors read and approved the final manuscript.

    Conflicts of Interest

    None declared.

    1. Bekker L, Johnson L, Wallace M, Hosek S. Building our youth for the future. J Int AIDS Soc 2015;18(2 Suppl 1):20027 [FREE Full text] [CrossRef] [Medline]
    2. Kahana SY, Jenkins RA, Bruce D, Fernandez MI, Hightow-Weidman LB, Bauermeister JA, Adolescent Medicine Trials Network for HIV/AIDS Interventions. Structural Determinants of Antiretroviral Therapy Use, HIV Care Attendance, and Viral Suppression among Adolescents and Young Adults Living with HIV. PLoS One 2016 Apr 1;11(4):e0151106 [FREE Full text] [CrossRef] [Medline]
    3. Zanoni BC, Mayer KH. The adolescent and young adult HIV cascade of care in the United States: exaggerated health disparities. AIDS Patient Care STDS 2014 Mar;28(3):128-135 [FREE Full text] [CrossRef] [Medline]
    4. HIV and Youth. Centers for Disease Control and Prevention.   URL: https://www.cdc.gov/hiv/group/age/youth/index.html [accessed 2021-08-16]
    5. Lally M, van den Berg J, Westfall A, Rudy B, Hosek S, Fortenberry J, Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN). HIV Continuum of Care for Youth in the United States. J Acquir Immune Defic Syndr 2018 Jan 01;77(1):110-117 [FREE Full text] [CrossRef] [Medline]
    6. Wood SM, Lowenthal E, Lee S, Ratcliffe SJ, Dowshen N. Longitudinal Viral Suppression Among a Cohort of Adolescents and Young Adults with Behaviorally Acquired Human Immunodeficiency Virus. AIDS Patient Care STDS 2017 Sep;31(9):377-383 [FREE Full text] [CrossRef] [Medline]
    7. Dobroszycki J, Lee P, Romo DL, Rosenberg MG, Wiznia A, Abadi J. Antiretroviral therapy in HIV-infected adolescents: clinical and pharmacologic challenges. Expert Rev Clin Pharmacol 2017 May 14;10(5):509-516. [CrossRef] [Medline]
    8. Rotheram-Borus MJ, Lee S, Swendeman D. Getting to Zero HIV Among Youth: Moving Beyond Medical Sites. JAMA Pediatr 2018 Dec 01;172(12):1117-1118 [FREE Full text] [CrossRef] [Medline]
    9. Reeder C, Neilands TB, Palar K, Saberi P. Food Insecurity and Unmet Needs Among Youth and Young Adults Living With HIV in the San Francisco Bay Area. J Adolesc Health 2019 Aug;65(2):262-266 [FREE Full text] [CrossRef] [Medline]
    10. Mobile Fact Sheet. Pew Research Center. 2021.   URL: https://www.pewresearch.org/internet/fact-sheet/mobile/ [accessed 2020-02-25]
    11. Rideout V, Fox S. Digital Health Practices, Social Media Use, and Mental Well-Being Among Teens and Young Adults in the U.S. Hopelab and Well Being Trust. 2018.   URL: https:/​/assets.​hopelab.org/​wp-content/​uploads/​2020/​08/​a-national-survey-by-hopelab-and-well-being-trust-2018.​pdf [accessed 2020-12-07]
    12. Social Media Fact Sheet. Pew Research Center. 2021.   URL: https://www.pewresearch.org/internet/fact-sheet/social-media/ [accessed 2020-03-24]
    13. DiClemente R, Ruiz M, Sales J. Barriers to adolescents' participation in HIV biomedical prevention research. J Acquir Immune Defic Syndr 2010 Jul;54 Suppl 1:S12-S17 [FREE Full text] [CrossRef] [Medline]
    14. Escobar-Viera C, Zhou Z, Morano JP, Lucero R, Lieb S, McIntosh S, et al. The Florida Mobile Health Adherence Project for People Living With HIV (FL-mAPP): Longitudinal Assessment of Feasibility, Acceptability, and Clinical Outcomes. JMIR Mhealth Uhealth 2020 Jan 08;8(1):e14557 [FREE Full text] [CrossRef] [Medline]
    15. Flickinger TE, Ingersoll K, Swoger S, Grabowski M, Dillingham R. Secure Messaging Through PositiveLinks: Examination of Electronic Communication in a Clinic-Affiliated Smartphone App for Patients Living with HIV. Telemed J E Health 2020 Mar 01;26(3):359-364 [FREE Full text] [CrossRef] [Medline]
    16. Whiteley L, Brown LK, Mena L, Craker L, Arnold T. Enhancing health among youth living with HIV using an iPhone game. AIDS Care 2018 Jan 09;30(sup4):21-33 [FREE Full text] [CrossRef] [Medline]
    17. Hightow-Weidman L, Muessig K, Knudtson K, Srivatsa M, Lawrence E, LeGrand S, et al. A Gamified Smartphone App to Support Engagement in Care and Medication Adherence for HIV-Positive Young Men Who Have Sex With Men (AllyQuest): Development and Pilot Study. JMIR Public Health Surveill 2018 Apr 30;4(2):e34 [FREE Full text] [CrossRef] [Medline]
    18. Erguera XA, Johnson MO, Neilands TB, Ruel T, Berrean B, Thomas S, et al. WYZ: a pilot study protocol for designing and developing a mobile health application for engagement in HIV care and medication adherence in youth and young adults living with HIV. BMJ Open 2019 May 05;9(5):e030473 [FREE Full text] [CrossRef] [Medline]
    19. Kempf M, Huang C, Savage R, Safren SA. Technology-Delivered Mental Health Interventions for People Living with HIV/AIDS (PLWHA): a Review of Recent Advances. Curr HIV/AIDS Rep 2015 Dec 9;12(4):472-480 [FREE Full text] [CrossRef] [Medline]
    20. Saberi P, Ming K, Dawson-Rose C. What does it mean to be youth-friendly? Results from qualitative interviews with health care providers and clinic staff serving youth and young adults living with HIV. Adolesc Health Med Ther 2018;9:65-75 [FREE Full text] [CrossRef] [Medline]
    21. Norman D, Draper S. User Centered System Design: New Perspectives on Human-computer Interaction. Boca Raton, FL: CRC Press; 1986.
    22. LeRouge C, Wickramasinghe N. A review of user-centered design for diabetes-related consumer health informatics technologies. J Diabetes Sci Technol 2013 Jul 01;7(4):1039-1056 [FREE Full text] [CrossRef] [Medline]
    23. Matheson GO, Pacione C, Shultz RK, Klügl M. Leveraging human-centered design in chronic disease prevention. Am J Prev Med 2015 Apr;48(4):472-479. [CrossRef] [Medline]
    24. Kachirskaia I, Mate K, Neuwirth E. Human-Centered Design and Performance Improvement: Better Together. NEJM Catalyst. 2018.   URL: https://catalyst.nejm.org/doi/full/10.1056/CAT.18.0144 [accessed 2019-03-14]
    25. Saberi P, Yuan P, John M, Sheon N, Johnson MO. A pilot study to engage and counsel HIV-positive African American youth via telehealth technology. AIDS Patient Care STDS 2013 Sep;27(9):529-532 [FREE Full text] [CrossRef] [Medline]
    26. Saberi P, Mayer K, Vittinghoff E, Naar-King S, Adolescent Medicine Trials Network for HIV/AIDS Interventions. Correlation between use of antiretroviral adherence devices by HIV-infected youth and plasma HIV RNA and self-reported adherence. AIDS Behav 2015 Jan 31;19(1):93-103 [FREE Full text] [CrossRef] [Medline]
    27. Saberi P, Siedle-Khan R, Sheon N, Lightfoot M. The Use of Mobile Health Applications Among Youth and Young Adults Living with HIV: Focus Group Findings. AIDS Patient Care STDS 2016 Jun;30(6):254-260 [FREE Full text] [CrossRef] [Medline]
    28. Fisher JD, Fisher WA, Misovich SJ, Kimble DL, Malloy TE. Changing AIDS risk behavior: effects of an intervention emphasizing AIDS risk reduction information, motivation, and behavioral skills in a college student population. Health Psychol 1996 Mar;15(2):114-123. [CrossRef] [Medline]
    29. Amico KR, Toro-Alfonso J, Fisher JD. An empirical test of the information, motivation and behavioral skills model of antiretroviral therapy adherence. AIDS Care 2005 Aug 20;17(6):661-673. [CrossRef] [Medline]
    30. Fisher JD, Amico KR, Fisher WA, Harman JJ. The information-motivation-behavioral skills model of antiretroviral adherence and its applications. Curr HIV/AIDS Rep 2008 Nov 25;5(4):193-203. [CrossRef] [Medline]
    31. Lewis JR. The System Usability Scale: Past, Present, and Future. International Journal of Human–Computer Interaction 2018 Mar 30;34(7):577-590. [CrossRef]
    32. System Usability Scale (SUS). Usability.gov.   URL: http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html [accessed 2021-03-31]
    33. Kalichman SC, Rompa D, Cage M. Reliability and validity of self-reported CD4 lymphocyte count and viral load test results in people living with HIV/AIDS. Int J STD AIDS 2000 Sep 25;11(9):579-585. [CrossRef] [Medline]
    34. Wilson IB, Lee Y, Michaud J, Fowler FJ, Rogers WH. Validation of a New Three-Item Self-Report Measure for Medication Adherence. AIDS Behav 2016 Nov 20;20(11):2700-2708 [FREE Full text] [CrossRef] [Medline]
    35. Fine TH, Contractor AA, Tamburrino M, Elhai JD, Prescott MR, Cohen GH, et al. Validation of the telephone-administered PHQ-9 against the in-person administered SCID-I major depression module. J Affect Disord 2013 Sep 25;150(3):1001-1007. [CrossRef] [Medline]
    36. Smith BW, Dalen J, Wiggins K, Tooley E, Christopher P, Bernard J. The brief resilience scale: assessing the ability to bounce back. Int J Behav Med 2008 Sep;15(3):194-200. [CrossRef] [Medline]
    37. Mancini JA, Blieszner R. Social provisions in adulthood: concept and measurement in close relationships. J Gerontol 1992 Jan 01;47(1):P14-P20. [CrossRef] [Medline]
    38. Cella D, Riley W, Stone A, Rothrock N, Reeve B, Yount S, PROMIS Cooperative Group. The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. J Clin Epidemiol 2010 Nov;63(11):1179-1194 [FREE Full text] [CrossRef] [Medline]
    39. Johnson MO, Rose CD, Dilworth SE, Neilands TB. Advances in the conceptualization and measurement of Health Care Empowerment: development and validation of the Health Care Empowerment inventory. PLoS One 2012 Sep 19;7(9):e45692 [FREE Full text] [CrossRef] [Medline]
    40. Chen WT, Wantland D, Reid P, Corless IB, Eller LS, Iipinge S, et al. Engagement with Health Care Providers Affects Self- Efficacy, Self-Esteem, Medication Adherence and Quality of Life in People Living with HIV. J AIDS Clin Res 2013 Nov 01;4(11):256 [FREE Full text] [CrossRef] [Medline]
    41. Leon AC, Davis LL, Kraemer HC. The role and interpretation of pilot studies in clinical research. J Psychiatr Res 2011 May;45(5):626-629 [FREE Full text] [CrossRef] [Medline]
    42. Whitehead AL, Sully BG, Campbell MJ. Pilot and feasibility studies: is there a difference from each other and from a randomised controlled trial? Contemp Clin Trials 2014 May;38(1):130-133. [CrossRef] [Medline]
    43. Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA. Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiatry 2006 May 01;63(5):484-489. [CrossRef] [Medline]
    44. Eldridge SM, Lancaster GA, Campbell MJ, Thabane L, Hopewell S, Coleman CL, et al. Defining Feasibility and Pilot Studies in Preparation for Randomised Controlled Trials: Development of a Conceptual Framework. PLoS One 2016 Mar 15;11(3):e0150205 [FREE Full text] [CrossRef] [Medline]
    45. Agwu AL, Lee L, Fleishman JA, Voss C, Yehia BR, Althoff KN, et al. Aging and loss to follow-up among youth living with human immunodeficiency virus in the HIV Research Network. J Adolesc Health 2015 Mar;56(3):345-351 [FREE Full text] [CrossRef] [Medline]
    46. Farmer C, Yehia BR, Fleishman JA, Rutstein R, Mathews WC, Nijhawan A, HIV Research Network. Factors Associated With Retention Among Non-Perinatally HIV-Infected Youth in the HIV Research Network. J Pediatric Infect Dis Soc 2016 Mar 19;5(1):39-46 [FREE Full text] [CrossRef] [Medline]

    ‎
    ART: antiretroviral therapy
    HCD: human-centered design
    mHealth: mobile health
    SUS: System Usability Scale
    UCSF: University of California, San Francisco
    YAP: Youth Advisory Panel

    Edited by G Eysenbach; submitted 31.12.20; peer-reviewed by JM Simoni, M Medich, P Athilingam; comments to author 02.03.21; revised version received 07.03.21; accepted 26.07.21; published 31.08.21

    Copyright

    ©Parya Saberi, Nadra E Lisha, Xavier A Erguera, Estie Sid Hudes, Mallory O Johnson, Theodore Ruel, Torsten B Neilands. Originally published in JMIR Formative Research (https://formative.jmir.org), 31.08.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.