The Delphi method is based on the convergence of expert opinion after multiple rounds of data collection (eg, surveys with Likert scale items) with the final round occurring when agreement emerges among experts [23]. The multiple rounds of data collection and analysis in the Delphi method allow for elimination of survey items that experts agree are unimportant and refinement of the remaining items [23]. The final round leaves a consensus list of recommendations experts agree are crucial to consider. This study used a modified Delphi method [24] to reach consensus on expert recommendations for AYA HIV mHealth interventions. The modified Delphi method used in this study maintained the multiple rounds of data collection, but it omitted the expert-to-expert face-to-face discussion that would occur following data collection due to difficulties in varying schedules and time zones as well as to limit experts from influencing each other. In the absence of the expert discussion, participants had the option to leave text comments in the survey that were later considered when designing the next survey data collection round.

Deidentified survey Likert responses were exported from REDCap for analysis in Excel (Microsoft Corporation). Scores for each survey responses were determined based on the mean importance rating from not important (1) to very important (4) and were then categorized as retained, flagged, or dropped. Retained survey responses were those with mean scores between 3 (important) and 4 (very important), and automatically appeared on the next iteration of the survey. Flagged survey responses had a score between 2 (somewhat important) and 3 (important). For flagged survey responses, the SDs were calculated. The survey responses with an SD>1 were dropped, and the other flagged survey responses were revised based on the experts’ comments. Finally, survey responses with a score ≤2 (not at all important, 1, to somewhat important, 2) were dropped.

As part of the REDCap survey, expert comments were collected in textboxes, which provided additional information on why they chose a particular survey response, and were reviewed during analysis [34]. Comments were compiled in Word (Microsoft Word) under their survey item, and responses with similar explanations/questions were grouped together for review. Comments were used to inform revisions following round 1 and were especially helpful for reviewing the flagged items, considering where divergence reflected disagreement among experts or unclear wording. Comments were also used to assist in assessing consensus among experts. The revised recommendation list presented to experts were those assessed as “important” (3) to “very important” (4) following the round 2 survey.

This study received an exempt determination from the Children’s Hospital of Philadelphia’s Institutional Review Board. The beginning of each survey included an overview of the study and consent language where moving to the next page of the survey indicated willingness to participate in the study.

Survey responses were received from 23 of the 35 (66%) invited adolescent HIV/mHealth experts for round 1 (see Figure 1) identified using reports from HIV mHealth intervention networks and recently funded HIV mHealth projects and programs. Participating experts represented 20 research organizations across the United States. More than half were principal investigators (n=15) and the remaining experts included various roles on a study team, including project directors (n=3), evaluators (n=2), project coordinators (n=2), and other project staff (n=1). Experts who completed the first survey were reinvited to participate in round 2 with a high response rate of 21 of 23 (91%) experts.

The modified Delphi method allowed for the rapid collection of data including the opinions of more than 20 AYA-focused mHealth HIV intervention experts with varied experience across the United States. Following two rounds of data collection, an original list of 35 items were modified into 24 recommendations on the design, evaluation, and implementation of HIV prevention and care mHealth interventions for AYAs. These consensus items provide guidelines for future development, evaluation, and implementation of mHealth interventions as well as for prevention scientists to support the health of AYAs at risk for acquiring or living with HIV. At each stage of AYA-focused mHealth HIV research, these considerations can guide decision-making to optimize efficacy, acceptability, and accessibility of mHealth technology to work toward ending the HIV epidemic.

The three items that experts scored as most important were related to intervention design: ensuring that intervention content is relevant to the needs of the specific youth population; having a multidisciplinary team including experts in HIV, adolescent development, software programming, user interface design, and youth; and using an iterative process (incorporating feedback and refining) in developing or adapting the intervention. These design recommendations highlight the potential value of youth involvement and using community-engaged research principles (eg, listening to your community stakeholders throughout the research process to build trust and improve health outcomes that outlast the study duration) [35,36]. These processes can ensure a strong foundation for the intervention as well as for intervention engagement and sustainability.

This study also identified that intrinsic elements of the design are critical for accessibility and privacy. The accessibility of mHealth interventions may fluctuate depending on the type of platform (eg, iOS, Android, or web applications), device’s operating system version, and adolescent’s access to Wi-Fi and data plan. Privacy and confidentiality of HIV-related mHealth interventions has been documented as very important to YLH [13,37,38], and AYA HIV experts in this study agreed. Preventing unwanted disclosures in fear of the stigma associated with HIV may stop adolescents from downloading or using mHealth interventions that do not protect their privacy and confidentiality [37]. Password-protected accounts, discrete app names, and limiting the access to one’s health information are ways to improve privacy and confidentiality [37,38].

The key considerations for the evaluation of mHealth interventions identified included determining what intervention components were the most effective and advantageous data collection strategies. The round 1 survey analysis found experts scored recommendations on individual data collection methods such as self-report, paradata, and EMRs as less important than selecting the right one for a specific project. This suggests that a one-size-fits-all approach to mHealth intervention evaluations may not be appropriate and should rather be tailored to fit each project’s goals and evolving technology. A clear understanding of the intervention’s desired outcomes may help to determine the most appropriate data collection methods.

In regard to real-world implementation of interventions, experts scored seeking youth input about strategies to improve engagement highest, indicating youth input would be the most important factor. Using a multidisciplinary team, including AYA input, from the start of intervention development was regarded as most important among experts. Again, the importance of including AYA input throughout the entirety of the project was recognized as crucial to reach and engage the youth audience the intervention was designed to help. Other highlights from the implementation recommendations primary focus on technology limitations such as high costs and frequent updates to platforms, operating systems, or devices. Understanding and taking into account the challenges facing youth most affected by HIV may help plan for a more successful implementation. For example, multiple forms of contact information, like emails or social media handles, may be helpful to collect since mobile phone disruptions are common in this group [39]. Additionally, staff time for technology maintenance and initiation were highlighted as important aspects to consider for intervention implementation for clinical care teams and community health organizations.

While not hosting a face-to-face discussion among experts allowed us to avoid scheduling challenges and reduced participant time burden, it remains a potential limitation to this study [23]. The option for experts to leave comments served as a discussion forum that allowed us to receive every expert’s opinion without influence from other experts. Another limitation was the potential for bias among experts invited to participate. Additionally, demographic, employment responsibilities, or years of experience were not collected. For this reason, researchers who occupied diverse roles on research teams were intentionally invited as experts to reduce this bias and gain a more complete understanding of different perspectives of the research design, evaluation, and implementation process.

This study is among the first to propose expert recommendations on the development, evaluation, and implementation of mHealth interventions for HIV prevention among AYAs. With a clear focus on the role of youth in all aspects of the process, these expert opinions may not only help move forward the quality of technology-based research for adolescent HIV prevention but also ensure that successful interventions will be disseminated broadly for the most impact.