This study is presented in 4 phases. Prevention messages that were developed and pilot-tested in 2018 [24,25] served as the foundation for message review for the YWG (phase 1). The original messages revised in this study (approximately 300 intervention and control messages in total) were developed using an iterative process that also used the experience and expertise of a YWG, content experts, and the study team (phase 2). In 2024, messages were then further refined and revised using a newly formed YWG and a substance use prevention expert (phase 3), and the process incorporated the use of an AI tool for additional message generation (phase 4).

In the 2018 pilot study, the team used an intervention mapping approach [26] to ensure that the message content reflected the cognitive, behavioral, and environmental factors related to HIV prevention among youth experiencing homelessness [27,28]. The study team conducted a literature review, assessed resources, and developed messages guided by social cognitive theory [29] and known HIV behavior change communication strategies [30]. The team convened 5 meetings with the YWG to review the initial slate of messages, create new messages, and suggest linguistic translations to ensure the messages were motivating, relatable, and culturally sensitive to the needs of youth experiencing homelessness. In total, 28 individuals participated in 1 or more meetings of the YWG, and the meeting size for each session was 10 participants. Participants completed a brief demographic survey on paper. All YWG meetings were held onsite (at the shelter or drop-in centers) immediately after recruitment. Each meeting was audio-recorded, and an observer took field notes.

The study team started with messages tested in the 2018 pilot study to expand upon and refine them prior to presenting them to the 2024 YWG. Collectively, the investigative team has extensive experience developing theory-driven prevention messages for youth experiencing homelessness and has conducted several JITAI studies focused on behavioral science, adolescent health, youth experiencing homelessness, substance use, health communication, and personalized message–based prevention interventions. The study team that interacted with the YWG to develop messages included 2 research nurses, 2 research assistants, a research coordinator with a Master of Public Health degree, and an HIV specialist nurse practitioner, all of whom have at least 5 years of experience working with the target population. The team also included 2 undergraduate students within the participants’ age range who were pursuing degrees in nursing and nutrition.

In 2024, we held 1 meeting to orient the YWG to the study and review study procedures, followed by 4 YWG meetings to re-evaluate the salience and acceptability of the pilot-tested prevention messages and to assess the need to add additional topics. Two meetings were held at a drop-in center, and 2 meetings were held at a shelter. In total, 24 individuals participated in 1 or more meetings of YWG, and meeting sizes ranged from 3 to 15 participants. Each participant completed a short demographic survey using REDCap (Research Electronic Data Capture) hosted at the University of Texas Health Science Center [31,32], and group facilitators outlined the specific meeting objectives, which were set by the study team and revised with YWG input. Each meeting was audio-recorded, and an observer took comprehensive field notes. Each meeting started with an icebreaker and a general orientation to the agenda, as well as an explanation of how these messages would be received in the future (via an app tailored to individual responses). All messages were programmed into REDCap, which allowed participants to individually rate the messages on an iPad using a 4-point Likert scale: 1=“Like a lot,” 2=“Like a little,” 3=“Dislike a little,” and 4=“Dislike a lot.” Every message was rated by at least 2 members of the YWG, and messages with an average rating of ≤2 (ie, liked a lot or a little) were retained. Messages rated 3 to 4 (ie, disliked a little or a lot) were reviewed for patterns and ultimately discarded. Messages with mixed ratings of 2.1 to 2.9 (liked and disliked) were rerated by a different member of the YWG to determine whether to retain or discard the message. Time for discussion was included in each meeting to gather additional feedback on specific messages. Participants provided feedback on the salience and acceptability of the messages and described their understanding of the message to determine if their interpretation matched the message’s intent. Messages were then rank-ordered within content categories to identify the most liked messages.

The most highly rated messages, which scored an average of 1 (liked a lot), were extrapolated and used to generate additional, similar message content modeled after prompts from another study [33] using the free, web-based Microsoft Copilot (version GPT-4 Turbo). At the time of the YWG meetings, Microsoft Copilot was the only AI tool approved for use in research at the university. The AI prompt used was: “Create (number of messages needed) messages similar to these about (topic):” followed by 4 highly rated messages with at least 1 information-based, 1 motivation-based, and 1 behavior-based message.

For example, “Create 20 messages about stress similar to these:”

The pilot study team developed 187 original intervention messages in 2018 after an extensive literature review and assessment of existing resources related to HIV prevention and substance use, including websites, social media accounts, and educational materials from local and national organizations. The pilot YWG met 5 times to rate and revise the messages developed by the study team and to write new messages that were relevant and acceptable. Demographic characteristics for the 2018 YWG are provided in Table 1.

Youth expressed that some risk reduction messages, especially those related to drug use, were perceived as endorsing the behavior and often recommended risk avoidance alternatives. For example, messages such as “Before you drink or use drugs, be sure you are in a safe place” or “If you inject drugs, use only new, sterile needles each time you inject” yielded alternatives such as “Don’t do drugs” and “Injecting drugs isn’t safe.” This sentiment was also expressed in sexual contexts, with 1 participant sharing, “I say you shouldn’t have sex before marriage. That would eliminate a lot of this.”

Others endorsed drugs and/or alcohol, particularly marijuana, writing that “marijuana is inspirational” and “sometimes weed is good for you.” One participant proposed replacing the message “You seem stressed right now. Call and talk to someone” with “Smoke a blunt.”

A key takeaway was that motivational messages were important to this group, and concise messages were preferred. One participant described the message “Repeat this to yourself: ‘I am strong, I can get through this without using alcohol’” as an “outstanding message.” Examples of message ideas included “Strap it up,” “Better safe than sorry,” “Be careful who you surround yourself with,” “Think before you act,” “Better days are to come!” and “Down with dope and up with hope.”

After compiling the information gathered from the YWG, 386 intervention messages were developed across 7 content categories (Table 2): sex urge (n=94, 24.3%), drug and alcohol urge (n=100, 25.9%), stress (n=27, 7%), drug use (n=37, 9.6%), recent sexual activity (n=51, 13.2%), recent sexual assault (n=8, 2.1%), and general motivational messages that would be provided when no risk is detected in the JITAI (n=69, 17.9%). These messages were pilot-tested and found to reduce drug use and sexual urge in the intervention group and to reduce stress in both groups [24].

Messages were then divided among the study team, who removed outdated local resources or language and segmented broader categories into smaller groups to respond to recent behaviors or currently occurring urges to use drugs, consume alcohol, or have sex, which are antecedents of HIV risk. The study team rotated messages across categories twice to review, edit, and create additional messages. Finally, the investigative team reviewed, edited, and suggested additional categories not covered in the original messages but that were endorsed by youth experiencing homelessness, such as injection drug use. This process resulted in a total of 888 messages across 10 categories (Table 1): sex urge (n=100, 11.3%), drug urge (n=103, 11.6%), alcohol urge (n=141, 15.9%), stress (n=109, 12.3%), recent drug use (n=101, 11.4%), injected drugs (n=10, 1.1%), recent sexual activity (n=100, 11.3%), used alcohol (n=106, 11.9%), recent sexual assault (n=14, 1.6%), and motivational messages provided when no urge or risk is detected (n=104, 11.7%).

Four YWG meetings were held to specifically gather feedback on the messages and assess their likability. Demographic characteristics for the 2024 YWG are provided in Table 1. Due to the low frequency of injection drug use among youth experiencing homelessness in the region across several past studies [24] and the sensitive nature of sexual assault, messages in these 2 categories were not reviewed by the YWG (but were reviewed by the study team). Demographic characteristics for the YWG are provided in Table 1. To work within the study timeline, the study team focused on having the YWG score messages, with the goal of eliminating the least liked messages.

The YWG reviewed 864 messages across 3 meetings. Of these, 231 (26.7%) messages with scores of 2.1 to 2.9 (liked and disliked) were rerated by another member of the YWG to determine whether to retain or discard the message. Ultimately, 93% (n=804) of messages had an average score of ≤2 (liked a lot or a little) and were categorized as acceptable for the intervention. The YWG disliked 7% (n=60) of messages, which were discarded.

The highest-rated messages in each category, scoring 1 (liked a lot by all YWG raters), were reviewed, and 4 sample messages per category were chosen to prompt the AI tool, Copilot.

Table 3 lists all AI prompts. The output (n=146) was reviewed for redundancy, and 70 (47.9%) messages were chosen for testing among the YWG. The low-scoring (“disliked”) messages were replaced with the 70 messages generated by Copilot, which were tested in a fourth YWG meeting for understandability and salience. All 70 messages were liked by the majority of the participants (n=69, 99%), with a mean score of 1.29 (SD 0.33).

Overall, the YWG rated 934 messages: 864 (92.5%) messages from the 2018 pilot and those generated by the study team in 2024, and 70 (7.5%) messages generated by Copilot. In total, 886 (94.9%) messages were programmed into the Insight mHealth Platform (University of Oklahoma Health Sciences Center) to test the efficacy of MY-RIDE on HIV prevention and substance use reduction (Table 2). Messages were programmed so that the highest-scoring, or most liked, messages were delivered first, with users receiving alternating message types (information-, behavioral-, or motivation-oriented).