All data were collected as part of the 3-arm RCT evaluating HMP 2.0. The 3 arms were the tailored information-only control (attention control), researcher-created HMP network (intervention arm 1), and peer-referral network (intervention arm 2). Details of the original study design and intervention are described in the published protocol [9].

Participants were recruited between July 2020 and September 2022 through 4 primary sources: paid ads on social media platforms (eg, Facebook, Instagram, X [formerly Twitter]); paid ads on partner-seeking apps (eg, Jack’d, Scruff, Grindr); study contacts (ie, individuals who had previously participated in research conducted by the study team or affiliated projects and had consented to be recontacted about future studies through institutional email listservs); and other sources that included in-person recruitment, peer referrals, community outreach, and postings on LGBTQ+ forums. Ads on social media platforms and partner-seeking apps appeared in users’ feeds, inboxes, or app interfaces and directed interested individuals to an online Qualtrics screener to verify eligibility.

To be eligible for the study, individuals had to be between the ages of 15 and 29 years (inclusive) at the time of screening, identify as Black or African American and/or Latino or Hispanic (except participants referred by those in Intervention arm 2), reside in the United States, and speak and read English. They were also required to have regular access to a smartphone and either report anal intercourse in the past 6 months or have been recruited from a partner-seeking app. The study included individuals assigned male sex at birth, with no restrictions on their current gender identity.

Interested individuals completed an online screening survey [10], and those who met all study eligibility criteria were emailed a unique link to the study’s informed consent form. Upon providing consent, participants completed a 30 to 50-minute baseline computer-assisted self-interview (CASI) survey.

To maintain data integrity, a thorough fraud detection process was implemented that involved reviewing IP addresses, geolocation data, and comparing responses between the screening and baseline surveys. Additionally, participant data were cross-checked with the study’s participant database to detect duplicate or fraudulent entries. This process effectively identified and removed duplicate or fraudulent entries.

Participants who passed the validation checks were randomized into 1 of the 3 study arms using a computer-generated blocked randomization procedure stratified by HIV status. They were then provided a link to download the study app from their respective app store (ie, Apple’s App Store or Google Play). To support retention across the 12-month study period, participants received regular reminders via email, text messages, and in-app notifications. These efforts aimed to maximize completion rates for follow-up surveys conducted at 3, 6, 9, and 12 months.

Participants could receive up to US $490 in Amazon gift card incentives for completing study surveys and other study-related activities. The breakdown of incentives includes up to US $280 for completing the study surveys, up to US $140 for completing study test kits, up to US $20 for successfully referring participants to the study (intervention arm 2 only), and US $50 for completing an in-depth interview, if selected.

The HMP 2.0 RCT was approved by the University of Pennsylvania institutional review board (IRB; protocol #829805) and registered on ClinicalTrials.gov (NCT03678181). All participants provided informed consent prior to their participation, with individuals aged 15 to 17 years granted a waiver of parental consent to protect their privacy as sexual minority youth. To ensure confidentiality, all collected data were deidentified prior to analysis.

A total of 750 participants were enrolled in the study between July 2020 and September 2022. The majority (n=445, 59.3%) were aged 25 to 29 years, followed by 236 (31.5%) participants aged 20 to 24 years and 69 (9.2%) participants aged 15 to 19 years. In terms of ethnicity, 318 (42.4%) participants identified as Hispanic, while 432 (57.6%) participants identified as non-Hispanic. Regarding race, 609 (81.2%) participants identified as non-White or multiracial.

Among study participants, 230 (30.7%) reported living with HIV, while 520 (69.3%) were HIV-negative. Most participants (n=666, 88.8%) identified as cisgender men, while 84 (11.2%) participants were categorized as gender diverse, including transgender women (n=16), nonbinary individuals (n=64), and other gender identities (n=4).

Participants were recruited through 4 primary sources. The largest proportion (n=394, 52.5%) were enrolled via partner-seeking apps. Social media platforms accounted for 202 (26.9%) participants, followed by prior study contacts, which contributed 118 participants (15.7%). The remaining 36 participants (4.8%) were recruited through other sources, including peer referrals, in-person outreach, and LGBTQ+ forums. Participant demographics and recruitment sources are summarized in Table 1.

Survey completion varied across participants. The majority (n=514, 68.5%) completed all 4 follow-up surveys. An additional 84 (11.2%) participants completed 3 surveys, meeting the high retention threshold. The remaining 152 (20.3%) participants were classified as low retention, including 36 (4.8%) who completed 2 surveys, 36 (4.8%) who completed 1 survey, and 80 (10.7%) who did not complete any follow-up surveys.

A significant association was observed between recruitment source and participant retention (χ²₃=36.62, N=750; P<.001). Retention rates varied across recruitment sources. Retention rates were the highest among participants recruited via social media and study contacts, while those recruited from partner-seeking apps and other sources had lower retention rates.

Among participants recruited via social media, 184 (91.1%) participants met the high retention criterion (completing at least 3 follow-up surveys; Figure 1). Similarly, 103 (87.3%) participants recruited via study contacts were highly retained. In contrast, participants recruited via partner-seeking apps had a retention rate of 288 (73.1%) participants, while those recruited via other sources had a retention rate of 23 (63.9%) participants.

Fisher exact tests revealed significant differences in retention between social media and other recruitment sources (Table 2). Retention was significantly higher among participants recruited via social media compared to those recruited via partner-seeking apps (OR 3.76, 95% CI 2.21‐6.41; P<.001) and other sources (OR 5.78, 95% CI 2.51‐13.31; P<.001). Retention did not significantly differ between social media and study contacts (OR 1.49, 95% CI 0.72‐3.08; P=.34). Participants recruited via study contacts also had significantly higher retention than those recruited via partner-seeking apps (OR 2.53, 95% CI 1.41‐4.54; P=.001) and other sources (OR 3.88, 95% CI 1.63‐9.26; P=.003). Retention did not significantly differ between partner-seeking apps and other sources (OR 1.54, 95% CI 0.75‐3.14; P=.25).

To evaluate whether these differences were independent of demographic variation across recruitment sources, a multivariable logistic regression was conducted adjusting for age, HIV status, race, ethnicity, and gender identity (N=750; pseudo R²=0.092; Akaike information criterion [AIC]=706.6). After adjustment, participants recruited via social media (aOR 2.80, 95% CI 1.51‐5.18; P=.001) and study contacts (aOR 2.11, 95% CI 1.14‐3.92; P=.02) remained significantly associated with higher odds of retention compared to those recruited via partner-seeking apps. Other sources did not differ significantly from partner-seeking apps after adjustment (aOR 0.61, 95% CI 0.28‐1.31; P=.21). Full model results are presented in Table 3.

Among covariates, HIV-positive status was independently associated with lower odds of retention (aOR 0.64, 95% CI 0.42‐0.96; P=.03). Gender-diverse participants also had significantly lower odds of retention compared to cisgender men (aOR 0.38, 95% CI 0.23‐0.63; P<.001). Age, race, and ethnicity were not independently associated with retention after adjustment.

Adjusted pairwise comparisons across all 4 recruitment source pairs are presented in Table 4. Other sources had significantly lower adjusted odds of retention compared to both social media (aOR 0.22, 95% CI 0.09‐0.54; P=.001) and study contacts (aOR 0.29, 95% CI 0.12‐0.72; P=.007). Social media and study contacts did not differ significantly from each other after adjustment (aOR 0.76, 95% CI 0.35‐1.61; P=.47).

Using a strict definition (completion of all 4 surveys; n=514, 68.5% of the participants), retention rates were 83.2% (168/202) for social media, 72.9% (86/118) for study contacts, 61.2% (241/394) for partner-seeking apps, and 52.8% (19/36) for other sources. The overall χ² remained significant (χ²₃=35.15, N=750; P<.001). Social media remained significantly associated with higher odds of retention compared to partner-seeking apps (aOR 2.79, 95% CI 1.70‐4.57; P<.001). The advantage for study contacts was attenuated and no longer reached significance (aOR 1.58, 95% CI 0.97‐2.55; P=.07).

The baseline model achieved an accuracy of 77.3%. It correctly classified 97.4% (114/117) of the participants with high retention but only 6.1% (2/33) of the participants with low retention. The resulting F₁-score for low-retention participants was 0.11, reflecting limited precision and recall for this group.

To improve sensitivity in classifying participants with low retention, the SMOTE-adjusted model was trained on a dataset with an artificially balanced distribution of retention groups. This model achieved a lower overall accuracy of 64.7%. However, recall for low-retention participants increased from 6.1% (2/33) to 63.6% (21/33), and the F₁-score for this group improved from 0.11 to 0.44. Recall for high-retention participants decreased from 97.4% (114/117) to 65.0% (76/117). Full performance metrics for both models are presented in Table 6.

Beyond overall model performance, logistic regression coefficients were examined to assess the relative contribution of each predictor relative to its reference group (Table 7). These coefficients represent the change in the log-odds of high retention when moving from the reference category to the comparison category for each predictor variable, holding all other variables constant. Because all predictors in the model are categorical, their coefficients should be interpreted as the relative likelihood of high retention compared to the reference group. Recruitment source remained the strongest predictor, with social media (β=1.45) and study contacts (β=1.29) showing the largest positive associations with retention compared to partner-seeking apps. These findings were directionally consistent with the adjusted inferential analysis. HIV-positive status (β=−.37) and gender-diverse identity (β=−0.75) showed negative associations with retention, also consistent with the inferential model. The coefficient for other sources (β=0.74) was directionally inconsistent with the adjusted regression, likely reflecting instability in the small subsample available for model training (n≈29 after the 80/20 split); the adjusted logistic regression is the more reliable estimate for that comparison.

Our findings demonstrate that recruitment source is associated with participant retention in online health interventions for YBLMT. By evaluating how different platforms correspond to retention outcomes, we highlight the need for strategic recruitment planning that goes beyond sample diversity to consider sustained participation over time.

In line with prior research [21,22], our primary analysis found that retention rates differed significantly across recruitment sources. Participants recruited via social media and prior study contacts maintained the highest retention (184/202, 91.1% and, 103/118, 87.3%, respectively). This may reflect platform norms: social media users often engage in longitudinal interactions (such as browsing, commenting, and following content over time), which may translate more effectively to long-term study participation [21]. The higher retention observed among prior study contacts likely reflects preexisting institutional trust and an established rapport with the research team [23,24]. Individuals who have previously participated in research and opted into a recontact listserv may possess a higher baseline motivation and a more favorable view of the research process, significantly increasing their likelihood of completing longitudinal follow-up.

In contrast, the lower retention (288/394, 73.1%) observed among participants recruited via partner-seeking apps may reflect the “low-friction” and highly goal-oriented nature of those platforms that may not align with sustained engagement [25]. Users may engage with a recruitment ad out of momentary curiosity or during a brief window of app usage without the baseline commitment or institutional trust found in social media networks or prior research listservs. This suggests that while apps are excellent for reaching high-risk populations, they may require more intensive “onboarding” to bridge the gap between initial click and long-term retention. Community-based recruitment (comprising in-person outreach, peer referrals, and LGBTQ+ forum postings) was associated with the lowest adjusted retention across all 4 sources, which may reflect greater structural or logistical barriers to sustained digital study participation among community-recruited participants and warrants further investigation.

Retention differences may also reflect the distinct participant characteristics associated with each recruitment source, as demonstrated in our secondary analysis. Social media recruitment yielded a younger, predominantly Hispanic, and HIV-negative cohort, whereas partner-seeking apps recruited older, non-White, and HIV-positive participants. Although retention rates were lower among participants recruited via partner-seeking apps, continued recruitment through these platforms is warranted, given that they reach large populations living with or vulnerable to HIV. Allocating additional resources for follow-up efforts, such as tailoring reminder strategies to participants’ recruitment platforms, may support retention in these groups. By contrast, the high retention observed among prior study contacts suggests that formalized participant referral programs are a useful strategy for sustaining retention in future studies. These patterns highlight the intersectional nature of recruitment dynamics and the importance of adjusting for potential confounders when estimating the independent association between recruitment source and retention.

Participants recruited via partner-seeking apps represented a cohort that was significantly older, more racially diverse, and more likely to be living with HIV compared to those from social media. These overlapping identities and statuses may correlate with unique structural demands (such as navigating health care systems for HIV care or differing digital engagement habits) that likely influenced their lower retention rates.

After adjusting for age, HIV status, race, ethnicity, and gender identity, social media and study contacts remained significantly associated with higher retention compared to partner-seeking apps, indicating that the retention advantage observed in these channels is not fully explained by the demographic differences in who they recruit. Beyond recruitment source, 2 demographic factors were independently associated with lower retention after adjustment. HIV-positive participants had significantly lower adjusted odds of retention (aOR 0.64), consistent with evidence that individuals managing HIV face competing health demands, stigma-related burdens, and structural barriers that may limit sustained engagement with digital health programs [1]. Gender-diverse participants had substantially lower adjusted odds of retention (aOR 0.38), a finding that persisted across the full multivariable model. Transgender and gender-nonconforming individuals experience elevated rates of housing instability, employment discrimination, and mistrust of health and research institutions [1], any of which may affect their ability to maintain participation over a 12-month study period. These findings suggest that retention support efforts should not only be tailored to recruitment platforms but also prioritize outreach to HIV-positive and gender-diverse participants, who face independently elevated risks of dropout regardless of how they were enrolled.

A sensitivity analysis using a stricter retention definition (ie, completion of all 4 follow-up surveys) was directionally consistent with the primary findings. The social media advantage over partner-seeking apps remained significant under this stricter threshold, confirming that this association is not an artifact of the chosen retention cutoff. The study contacts advantage attenuated and no longer reached significance when full completion was required, suggesting that prior study participants reengage readily but may face the same real-world barriers to sustained long-term participation as other groups. This distinction is worth considering in future studies that rely heavily on prior participant networks for recruitment.

To complement the inferential analysis, we developed a predictive logistic regression model to explore whether demographic and recruitment variables could classify retention outcomes—a hypothesis-generating exercise motivated by the potential of ML approaches to surface patterns in behavioral health data [17,18]. The baseline model achieved high accuracy overall (77.3%) but classified only 6.1% (2/33) of low-retention participants correctly, reflecting the limitations of standard logistic regression under class imbalance. Because the dataset was heavily skewed toward high-retention participants, the model’s estimates for predictors of disengagement were unreliable without correction.

To address this, we applied SMOTE to synthetically balance the training dataset. The SMOTE-adjusted model achieved an accuracy of 64.7%, with low-retention recall improving from 6.1% (2/33) to 63.6% (21/33) and the F₁-score for that group improving from 0.11 to 0.44, at the cost of a reduction in overall accuracy. Importantly, the goal of applying SMOTE was not to predict which individual participants would drop out, but to strengthen the model’s ability to detect group-level patterns associated with lower retention. The direction of the SMOTE model coefficients was largely consistent with the adjusted inferential analysis: social media and study contacts showed the strongest positive associations with retention, while HIV-positive status and gender-diverse identity were negatively associated. The coefficient for community-based recruitment diverged from the adjusted regression estimate, likely reflecting instability given the small subsample available for model training (n≈29 after the 80/20 split); the adjusted logistic regression provides the more reliable estimate for that comparison.

Our findings suggest that retention strategies should be tailored based on the recruitment platform’s digital “friction.” For participants recruited through highly goal-oriented channels such as partner-seeking apps, informatics-based interventions could incorporate targeted engagement features to mimic the rapport seen in established research networks. Specific digital “nudges” might include personalized push notifications tailored to the participant’s study milestone, milestone-based incentives, and gamified progress indicators that visually reward longitudinal participation. Additionally, incorporating peer interaction prompts or “community boards” may help build the institutional trust and social connectivity that characterized our high-retention study contact group. These design strategies offer a scalable way to optimize both participant diversity and sustained study engagement.

This study has several limitations. First, the race variable was collected using a multiselect item, which precluded disaggregation beyond a White versus non-White or multiracial binary classification. Given that the study specifically targets Black and Latino YBLMT, future analyses should use survey instruments that capture racial subgroup membership in a format amenable to group-level comparisons, enabling more granular examination of retention differences within the non-White population.

Although age was not independently associated with retention in the primary adjusted regression, the youngest cohort (15‐19 y, n=69) showed relatively high apparent retention, and we recognize that age may nevertheless serve as a proxy for competing life responsibilities. Factors such as full-time employment, independent housing logistics, and other adult responsibilities (which correlate with older age in this demographic) may serve as barriers to longitudinal engagement compared to younger participants who may have more discretionary time. Since these specific socioeconomic status indicators were not collected in the parent trial, we cannot definitively separate the impact of life stage from economic stability. Future research should prioritize the inclusion of comprehensive socioeconomic status metrics to better understand these overlapping influences on study persistence. Additionally, reliance on self-reported data and survey completion introduces the possibility of selection or response bias. Beyond these concerns, our study did not assess individual-level factors such as motivation, competing life responsibilities, or platform usability. Integrating qualitative data or engagement analytics could provide deeper insights into these factors.

Because this analysis was conducted using data from a parent RCT, the study was not originally powered to detect differences in retention across recruitment sources. The findings should therefore be interpreted as exploratory and hypothesis-generating. Although multiple digital platforms were used for recruitment (eg, Facebook, Instagram, Grindr, Jack’d, Scruff), the sample size within individual platforms was insufficient to conduct reliable platform-specific comparisons. Future research examining cross-platform comparisons may be warranted [26], as it may elucidate how differences in platform design, user demographics, and interaction norms influence both recruitment efficiency and longitudinal engagement in digital health interventions. Such analyses could help researchers more strategically allocate recruitment resources and tailor retention strategies to the behavioral patterns and expectations associated with specific digital environments.

Our logistic regression model successfully identified key predictors of retention, including recruitment source, age, race, gender identity, and HIV status. However, its ability to classify low-retention participants was limited, pointing to the need for refinement of predictive approaches. A sensitivity analysis using a stricter retention definition (completion of all 4 surveys) was conducted and found to be directionally consistent with the primary findings, though the study contacts advantage attenuated under this threshold.

Finally, we used SMOTE to address class imbalance in retention outcomes. While this method improved recall for low-retention participants from 6.1% (2/33) to 63.6% (21/33) and increased the F₁-score from 0.11 to 0.44, it reduced overall accuracy from 77.3% to 64.7%. Other resampling or model-weighting techniques may yield different trade-offs and should be explored in future work.

Recruitment source was associated with participant retention in this online health intervention, highlighting its importance in digital research planning. Some platforms correspond to higher retention, while others enhance sample diversity or reach populations at increased HIV risk. As digital advertising costs rise, researchers must weigh recruitment efficiency against retention feasibility. By refining recruitment strategies and retention frameworks, future online HIV interventions can be optimized to better sustain participation among populations most impacted by the epidemic.