A single-group cohort study was used to examine the feasibility of assessing economic and sexual risk behaviors using weekly text message surveys. Participants were invited to respond to a text message survey sent to their cell phone every Monday at 9 AM for 5 weeks.

Potential participants were recruited onsite from 2 community-based organizations (CBOs) providing emergency and supportive residential services to young adults in Baltimore, Maryland. A recruitment flyer was posted in the main building of both CBOs. Designated CBO staff introduced potential participants to the study team on scheduled visit days. Study eligibility was determined using a paper-based screening questionnaire that was administered by a trained research assistant. Individuals were eligible to participate if, at the time of enrollment, they were African American, aged 18-24 years, living in Baltimore, experiencing homelessness within the last 12 months (ie, defined as reporting any episode in which a person lacked a regular or adequate nighttime residence, such as a hotel/motel, vehicle, shelter, or friend’s home, and was living primarily on their own, apart from a parent or guardian), unemployed or underemployed (≤10 hours per week), out-of-school, reporting one or more episodes of unprotected sex in the last 12 months, and having a cell phone that could send and receive text messages. Eligible participants were then introduced to the study, and informed consent was obtained.

As part of the enrollment process, we invited participants to register their cell phone number with the text message survey app. Participants sent a text message with the word join to the study phone number to register. Each person then received a brief orientation regarding the survey’s content, timing, and payment incentive (US $20 in cash for answering 4 out of 5 weekly surveys). We also provided snacks and beverages. In the presence of a trained research assistant, the participant also completed a mock but identical version of the 14-question text message survey on his or her cell phone. This was done to confirm readability and understanding of the text message questions and prompts and to clarify any points of confusion. Participants were also advised on how to opt out of the survey by sending a text with the word leave at any time. As a final orientation step, participants were provided an informational sheet and advised on how to increase privacy during the study period and avoid unintended loss of confidentiality, such as activating cell phone passwords, deleting all text message surveys, responding only to the study’s phone number, and answering in a quiet and private space. Participants were also informed of the study’s security protocol that included separating cell phone numbers from identifying information; selecting a platform, such as TextIt.In, that did not require handing over participants’ names, addresses, or other identifying information to a mobile database software company; anonymizing phone numbers with a random code at the end of the study to render numbers invalid for future use; and using encrypted and compliant channels of TextIt.In.

We used TextIt.In to create, send, and receive text messages from participants. TextIt.In is an online service for building text messaging apps using a visual and interactive flow. The text message survey was powered by Twilio, a cloud communications platform, using a study-sponsored phone number. We then developed an online logic tree to order how survey questions would be texted to the participants. Figure 1 shows an excerpt of the branch logic used in the question tree. Each of the 14 questions was sent sequentially and in the same order as a single text message after the prior question had been answered. To facilitate responsiveness and data quality, the text messaging app included automated reminders and quality check prompts. Participants had 24 hours to complete each weekly text message survey. One automated text message reminder was sent to participants who did not initiate responding to the survey or to those who started but did not complete the survey within the first 24 hours. Reminder text messages included the name of the study, the payment incentive, and a reminder to respond to the survey within the next 24 hours. In addition, participants who responded with ineligible words or numbers outside of preset ranges received a text message query asking them to re-enter a valid response. All completed surveys generated 1 automatic text message that thanked the participant for his or her time.

Data were collected in August and September 2017. Participants received the same 14 questions as text messages each week, regardless of their responses to the week’s prior text message survey. Table 1 lists the questions used in the survey of this formative study. The set of questions was adapted from previous studies of economic empowerment and costs associated with HIV preventive and treatment services [42-51] and included questions developed by the study team, specifically for African American young adults experiencing homelessness and unemployment [52-54]. All questions were in English and reviewed by the study team, piloted with young adults, and revised as necessary prior to utilization. The questions referred to the last 7 days, equivalent to the prior week. Eligible responses were yes/no or number of units (ie, dollars, episodes, and people). There were 7 economic questions relating to involvement in any type of paid work, the amount of cash earned from a job, the amount of cash earned from one’s own business, the amount of cash deposited into a savings account, the occurrence of loss of housing, the occurrence of requesting for cash to meet living expenses, and the amount of cash spent on any HIV preventive services or products (ie, condom or lubricant purchases, insurance copays for HIV testing or antiviral medications, and travel expenses to HIV educational sessions). An additional 7 sexual behavioral questions inquired about the number of sex partners, engagement in sex while high or drunk, frequency of condomless sex, utilization of other noncondom HIV preventive methods, frequency of sex exchange, discussion of HIV testing with sex partners, and receipt of HIV testing.

The primary feasibility measures of this study were: number of participants who responded to the survey in a given week, number of questions to which a participant responded in a given week, and number of hours from sending a survey to participants to receiving their response in a given week. We calculated the number, mean, and proportion of participants who responded to each question in each of the weekly surveys over the 5-week study period. A participant was categorized as responding to the question if he or she provided a valid response such as yes/no, a numerical response, a free-form text, or a skip response to proceed to the next question. A participant was categorized as responding to the survey if he or she provided a valid response to at least one question of the 14-question survey. Ever responders were defined as enrolled participants who responded to at least one text message survey over the course of the study period. Nonresponders were defined as enrolled participants who did not return a text message response to any of the text message surveys over the course of the study period. We considered the study to be feasible if we were able to identify and recruit >15 eligible participants within the study period. The study was also considered feasible if a mean question response rate of 70% or more among ever responders was achieved and if the mean response time was 24 hours or less. Feasibility was based on ever responders to account for any initial run-off of participants who signed up for the study but did not participate once the text message survey was initiated.

As part of the study’s process evaluation, 1 discussion group with 5 responders was used to obtain feedback. A recruitment flyer was posted in the main building of both CBOs, and all participants, including nonresponders, received a text message regarding the day, time, and location of the focus group discussion. The group discussion was moderated by the study PI who used a focus group discussion guide. Participants were asked to describe what they liked or disliked about the text message survey, what they considered to be barriers and facilitators to responding, and what changes they would recommend regarding the survey design (ie, questions, timing, and frequency), including suggestions for additional information or questions they would have liked to receive. Participant responses were documented using memo field notes that were expanded upon immediately after the discussion.

To analyze the results of the text message survey, we first created a database in Excel that included: a cell phone number for each participant; a participant unique study ID; demographic data relating to participants’ age, gender, education level, years living in Baltimore, number of hours worked per week, and number of children living in and out of the household, the date and time of study enrollment, the date and time of all outgoing and/or incoming messages, and the numerical, textual, or free-from text message response to each of the 14 text message questions each week. Secondly, we calculated the study’s primary feasibility measures, as listed above. Third, we calculated the frequencies of sexual and economic behaviors per the specific responses for each weekly question. Finally, lessons learned from the study’s process evaluation were analyzed over 5 implementation domains: acceptability, enrollment and registration, responsiveness, data quality, and data access. This process involved a close reading of the study’s field notes, coding lessons learned by each implementation domain, and discussing findings with the study team.

This study received ethics approval from the Johns Hopkins Bloomberg School of Public Health institutional review board (IRB#00007563).

The dataset analyzed during this study is available in the Mendeley repository [55].

A total of 17 participants were enrolled in the study, accounting for 1 of the study’s 3 feasibility criteria. Table 2 describes the sample’s demographic characteristics. All participants (n=17/17, 100%) were African American (per inclusion criteria), living in Baltimore, and recruited from 1 of the 2 community organizations providing support to young adults experiencing homelessness. The mean age was 21.2 years (SD 2.1). 53% (n=9/17) of participants were female and 47% (n=8/17) of the participants were male. About half of participants (n=9/17, 53%) had not received a high school diploma or equivalent. None were currently enrolled in school. The majority of participants were unemployed (n=13/17, 76%), and 24% (n=4/17) were working part-time. 29% (n=5/17) were parents with children living in or outside of their household.

Table 3 describes additional feasibility measures of the study. 65% (n=11/17) of participants responded to at least 1 text message survey over the 5-week study period compared with 35% (n=6/17) of participants who never responded. The majority (n=7/11, 64%) of ever responders were young women. The majority (n=4/6, 67%) of never responders were young men (Table 2). Among those who ever responded, an average of 7.6 participants responded to the text message survey in any given week (69% response rate; Table 3). Response rates among ever responders ranged from 64% to 82% across the 5-week study period, representing 62.7% of all survey questions in all 5 weeks. When participants responded in a given week, they also answered the majority of the 14 survey questions. The mean number of answered questions for responders in a given week was 12.6 (SD 2.7; 90% of all questions), ranging from 72% to 100% of all questions. This met the study’s feasibility criteria of an average weekly question response rate of 70% or more among ever responders. An average of 6.4 participants (84%) answered all 14 text message survey questions in a given week, ranging from 57% to 100%. Participants responded on average 8.7 hours (SD 10.3) after receiving the survey. In week 1, participants responded the fastest with an average of 1.7 hours (SD 2.2). The slowest mean time to response was 12.6 hours (SD 13.2) in week 3. This met the study’s feasibility criteria of a mean response time of <24 hours.

Table 4 presents the number of responders per question per week among ever-responding participants. The questions at the beginning of the survey had the highest response rates. Response rates were comparable across all questions in weeks 2 and 5 but tapered at the end of the survey in weeks 1, 3, and 4. In week 5, 1 participant answered the first question of the survey but omitted answering any further questions. Participants were most responsive to questions about employment, condomless sex, and discussions with sex partners. Nonresponse was highest for questions relating to sex exchange and money spent on HIV prevention products or services. The text messaging app successfully sent and received 1289 text messages with few errors (0.1%), indicating relative efficiency and reliability (Table 3).

Weekly economic and sexual behaviors reported by the participants are shown in Table 5. Employment rates remained low, ranging from 14% to 43% over the study period. Mean earnings from employment by others or from the participant’s own business ranged from US $37 (SD 62.8) to US $146 (SD 220.3) per week and from US $7 (SD 18.9) to US $55 (SD 107.2) per week, respectively (Table 5). Participants experiencing housing instability decreased from 43% to 0% over the course of the study period, as did the proportion of those requesting money from others to cover living expenses (57%-0%). For most weeks, no money was spent on HIV prevention services or products, such as condoms, HIV testing, lubricants, or antiviral medications. For 3 of the 5 weeks, approximately 14% of participants reported having sex while high or drunk at least once in the past week. Condomless sex was a common risk behavior, with 14%-75% of participants reporting condomless sex at least once in a given week. There were no reports of sex exchange for money, food, or housing. Using other noncondom prevention methods was also low (11% in week 3). Participants were more likely to respond yes to the last two sexual behaviors of the survey, which were about discussing HIV testing with any of their sex partners (43%-67%) and receiving an HIV test in the past week (14%-43%). Participants used the skip response infrequently and only during weeks 1 and 2. When used, skip was most common for questions relating to sex exchange and money spent on HIV prevention services or products (Table 5). The dataset analyzed during this study is publicly available [55].

Table 6 summarizes the successes, challenges, and lessons learned in using text message surveys in this population. Key successes included participant acceptability, willingness to respond to the survey, confirming readability and functionality using a mock text message survey at enrollment, having moderately high responsiveness, and building in quality checks. Implementation challenges were low responses to questions perceived as sensitive or stigmatizing, technological delays, and the time required for restructuring text message data for analysis. Additional feedback from responders in a post-study discussion was that having the text message surveys arrive weekly and at the same time was helpful, as participants were always on their cell phones and available to respond quickly and conveniently. This, along with receiving cash payments, was viewed as a positive outcome. However, the reported weaknesses were that, for some, receiving the same set of questions each week was repetitive and may have contributed to response fatigue. Participants also requested whether informational text messages such as job announcements or sexual health tips could be provided as a reward for responding to each week’s survey. The study team’s observations while implementing the text message survey was that reducing text message wording, including using response prompts (eg, reply with: yes/no, US $ dollars, and # of times), and reminders were important to facilitating participation.

The goal of this study was to examine the feasibility of a relatively new mode of data collection using text message surveys in a high HIV prevalence urban and ethnic minority setting. We found that using weekly automated text message surveys with short assessments was feasible with vulnerable young adults. Data collection with this population can be challenging, given the unpredictability of young adults’ schedules and the uncertainty regarding their interests in research participation. However, the majority of invited participants completed the survey and were receptive to answering the study’s text message questions. To our knowledge, this is one of the first studies in an urban setting to use text message surveys to assess economic and sexual risk behaviors in economically-vulnerable African American young adults, who also had little experience responding to text messages for research purposes.

The study’s experience is informative with regard to 3 research areas: acceptability of text message surveys, survey responsiveness, and implications for future studies relating to efficiency and data quality. First, in the context of acceptability, several factors may have contributed to the study’s generally positive reception. This study’s recruitment process began with an introduction to the study to young adults in the presence of their peers at the CBO center. Therefore, potential participants had an opportunity to enjoy snacks, ask questions, and determine their own interests, including the interests of their peers, in participating in the study. The study team also explained the cash incentives being used to compensate individuals who completed the 5-week test cycle. Although eligible young adults appeared to be motivated by cash payments, other potential drivers to participation may have been the perceived benefit of participating in a study on HIV prevention with friends, including being prompted to think about or discuss HIV. Another driver to participation may have been interests in using text messages as a new means of income generation, since nearly all young adults had a cell phone and were underemployed. The study also invited each participant to try a mock text message survey on their cell phone in a private location at the CBO. This enabled them to see what they would be receiving each week and to confirm their capacity to respond. It was our experience that participants found responding to relatively sensitive text message questions on economic and sexual behaviors acceptable, given the readability of the questions, the short completion time required (about 3 to 4 minutes), the anonymity of their cell phone, the ability to use phone passwords for additional privacy, and the option to delete all text messages.

A second area of consideration is responsiveness to the text message survey. Among all participants, the response rate of 65% was moderately high, representing nearly two-thirds of participants. For those individuals who responded to at least one text message survey, response rates were even higher in a given week, ranging from 64% to 82%, with participants answering about 90% of all questions. This study’s findings included higher response rates than similar text message survey studies, including 3 studies assessing substance use and sexual risk behaviors with US young adults aged 18 to 25 years (49% response rate) [2], medication adherence in HIV-negative transgender men and women (39% response rate) [3], and medication adherence from caregivers of HIV-infected children in Uganda (24% response rate) [56]. On the other hand, we have similar response rates as 2 additional studies assessing drug adherence using text message surveys with lesbian, gay, bisexual, and transgender young adults in the United States living with HIV (61% response rate) [5] and assessing quality of life via text message surveys with older patients with rheumatoid arthritis (69% response rate) [14]. Although higher responsiveness may be needed if text messages are the sole form of evaluation, such engagement by predominantly financially- and residentially-unstable young adults is encouraging. In addition, the average time to response ranged from 2 to 12 hours, representing a relatively rapid response period compared with mail-in or online surveys that may experience several days or weeks between distribution and response. Potential contributing factors may have been that the study’s in-person orientation process allowed participants to feel prepared in knowing what type of questions would be asked and how long it would take to respond. Our sequentially sending each question only after the prior question had been answered further allowed participants to track and respond to questions at their own pace. Being female also appeared to aid responsiveness, although more research is needed to understand whether and why this may be the case.

To that end, it is important to consider the nonresponsiveness observed in this study. The reasons for nonresponse may have been poor network coverage, having one’s cell phone lost/stolen, or lacking sufficient charge or cellular credit. The potential to earn cash payments spurred interest in many participants. However, for some, this enthusiasm may have waned over time. Nonresponsiveness may also have resulted from experiencing negative emotions when thinking about financial hardships or prior sex partners. It is possible that some participants simply wanted a break from the study but chose not to use the opt-out commands (leave or stop). Future studies should assess reasons for nonresponse, as this could increase the number of participants providing study data. Increasing incentives, reducing the number of questions, or reducing the frequency of text message surveys may also improve responsiveness. Paying participants more frequently rather than at the end of the study and requesting alternative forms of contact (ie, email and social media) to reconnect with nonresponders may additionally be helpful. Greater engagement might also be achieved if the text message surveys are concurrently embedded within an intervention or other in-person contact targeting the outcomes of the text assessment.

A final important area relates to implications for future research. Our text message surveys showed some promise as a measurement tool in behavioral research. Having repeated measures each week from participants provides stronger statistical power and enables trialists to better characterize fluctuations over time. Our text message survey was configured to query again any invalid or out-of-range responses to maximize data quality over time. Once the survey flow was automated and launched, there was minimal maintenance. However, despite having data available from the moment participants responded, the process of restructuring and aggregating weekly data files was time-consuming and resulted in the team generally viewing and analyzing data at the end of the study, rather than on a weekly basis. Improvements in exporting and coding the messaging app’s data would increase efficiency. Finally, given that text messaging technology is constantly evolving as are young adults’ cell phone behaviors, including the option to respond to surveys via instant messaging or other text messaging apps may improve participation.

The study’s small sample size was a limiting factor, as the study was not designed to determine efficacy or estimate prevalence of economic and sexual risk behaviors. Rather, the study aimed to conduct a rapid test cycle using a small group of young adults with a preliminary goal of assessing instrument feasibility for a larger intervention trial. We additionally selected young adults who had a working cell phone, were literate, and were receiving residential services at the participating CBOs. Although the participants were vulnerable in other ways, such a sampling strategy could mean that the findings are not generalizable to more disconnected young adults. In addition, although not using an interviewer to administer surveys may have increased participants’ responsiveness to sensitive questions, the use of self-administered assessments could have reduced data quality. Finally, although weekly assessments provided more frequent contact than traditional pre-post study designs, asking participants to recall economic and sexual behaviors in the last 7 days rather than the day before via daily surveys may have been challenging for some participants. Despite these limitations, the study was successful in monitoring behaviors over time. This study’s findings provide support for using text message surveys to collect data in future behavioral trials.

Text messages offer the potential to better evaluate HIV behavioral interventions using repeated longitudinal measures at lower cost and research burden. However, they have been underutilized in US minority settings. We found that using weekly automated text message surveys with short assessments was feasible with vulnerable young adults. Additional research should focus on maintaining high responsiveness, improving the efficiency of data analysis, and ensuring data quality.