Virtual Lab is a survey platform used to recruit and survey participants within a social media platform, and is owned and run independently of social media companies [20]. To utilize what Virtual Lab has to offer, specifically within Facebook, the first step was to create a Facebook business account. To prevent potential biases, the account was given a general name, “Digital Health Research.” Under this account, we created a Facebook page, “Digital Media Experiment,” to host advertisements for recruitment purposes. In order to demonstrate the credibility of the page, we posted relevant content and acquired likes. We used this Facebook Page to run recruitment advertisements during August 5-12, 2021. The recruitment advertisements were shown to our target population of people aged 18-24 years and located in the United States. The two recruitment advertisements used in this feasibility study (Figure 1) were designed using the 99designs website. The advertisements used the text “Take a 15 minute survey, get paid $10.” After participants clicked on the study’s advertisement, they were sent a message via Facebook Messenger inviting them to participate in the study.

Under the same business account, we created another Facebook page named “Consumer Consciousness,” which was solely used to run the target advertisements on the enrolled participants’ Facebook Newsfeeds during August 20-26, 2021. We created this second Facebook Page with a different name to prevent biases.

Privacy measures include encryption of data that are stored and in transmission. Before beginning the pre-exposure survey, participants were sent messages regarding the topic of the survey, compensation, privacy measures, and contact information if they had any questions. Following those messages, the participants were sent a message asking for consent for their participation in the study and if they would like to continue. All studies were reviewed and approved by the institutional review board at George Washington University (NCR202837).

The pre- and postexposure surveys were completed using a survey platform called Typeform, which is supported by Virtual Lab. After designing the surveys in Typeform, they were linked to our Facebook business account and pages using the Virtual Lab interface. Typeform used our “Digital Media Experiment” page to send automated messages through Facebook Messenger, similar to a chatbot, to participants who clicked on the recruitment advertisements. Through Typeform, we were able to create logic jumps based off participants’ responses and confirm that they met the recruitment criteria. If participants did not meet the eligibility criteria, they were thanked for their participation and the survey ended. If the participants met the eligibility criteria, they were allowed to continue the survey.

After participating in the pre-exposure survey administered by the “Digital Media Experiment” page, respondents randomized to be exposed were shown the target advertisement via the “Consumer Consciousness” page in their Facebook Newsfeeds. All participants were invited to take the postexposure survey a week after completion of the pre-exposure survey. The postexposure survey was also administered by the “Digital Media Experiment” page via Facebook Messenger.

The pre-exposure survey consisted of questions to determine demographic characteristics and e-cigarette status. Questions determining demographics included age, gender, sexual orientation, combined race and ethnicity, and perceived financial status. e-Cigarette use status was determined through 2 questions: an ever-use question, “Have you ever tried using any e-cigarette/vape (even 1 or 2 puffs)?” with response of “yes” or “no.” For those who responded with “yes,” there was a current-use question, “During the past 30 days, on how many days did you use an e-cigarette (even 1 or 2 puffs)?” where they could respond with 0-30 days. Respondents were classified as never users if they responded with “no” to the ever-use question, as past users if they responded with “yes” to the ever-use question and 0 to the current-use question, and as current users if they responded with “yes” to the ever-use question and >1 days to the current-use question.

Intentions to use e-cigarettes was determined by asking the question, “Do you think you will use an e-cigarette (even 1 or 2 puffs) in the next year?” Answer choices were “definitely not,” “probably not,” “probably yes,” and “definitely yes,” where “definitely not” was coded as no intentions to use and all other choices were coded as having intentions to use. Intentions to quit using e-cigarettes was only asked of current users: “Are you seriously thinking about quitting e-cigarettes/vapes for good?” The responses were dichotomized where “No, I am not thinking about quitting” was coded as no quitting intentions and the following were coded as having intentions to quit: “Yes, but not within the year,” “Yes, within the year,” “Yes, within the next 6 months,” “Yes, within the next 30 days,” and “I’ve already quit.”

The post-survey included questions for e-cigarette use status and intentions to use and quit, as listed above, as well as questions about the target advertisement they were exposed to between taking the pre- and postexposure surveys. These questions were asked in the pilot study to ensure feasibility for the proceeding full study. Participants were first shown an advertisement they were exposed to and asked if they can see and hear the video. If they responded with “no,” they were directed to the end of the survey. Those who responded with “yes” were asked how many times they have seen the advertisement, receptivity questions on what they thought of the advertisement, and about actions they would take after seeing the advertisement. This series of questions on the target advertisement were then repeated for the second target advertisement.

A series of descriptive analyses were conducted to examine the feasibility of recruitment methods and the diversity of samples. Survey data were compared between complete cases and participants lost to follow-up and among different e-cigarette use status groups. Complete cases were defined as participants who completed the survey through the last question in the postexposure survey. Loss to follow-up was defined as participants having been lost at any point before the last question in the postexposure survey.

In analysis 1, we compared sample demographics and use status between complete cases and participants lost to follow-up. For continuous measures, means and SDs were obtained and t tests were conducted to compare differences between groups. For categorical measures, frequencies and proportions were obtained and Pearson chi-square tests were conducted. The analysis included the number of missing participants to observe the difference between complete cases and participants lost to follow-up.

In analysis 2, we compared sample demographics with the US national survey data. Race and ethnicity were compared by sex between the US population and complete cases collected in this study. The US national demographics for the population aged 18-24 years were retrieved from the US Census Bureau as of 2019 (the most recently available year). We conducted chi-square tests to evaluate the difference of proportions of these demographics between the US population and the study sample. Analyses were conducted using Stata SE 15 (StataCorp) and Excel (Microsoft Inc).

To our knowledge, this is the first study to examine the utility of Virtual Lab within the real-life setting of Facebook as a viable platform for media campaign awareness studies. The results of this pilot study provide support for continued use of the Virtual Lab platform to recruit participants and obtain data from a nationwide sample.

We found that recruitment yielded a diverse sample, consistent with other convenience samples [10,21], and it was carried out cost-effectively and quickly. Within 1 week of recruitment and 1 week of conducting the study, 375 respondents participated in the study with a total of 234 to complete the study. Including the US $5 Amazon e-gift cards for each survey, the total cost for incentives per respondent was approximately US $10 [22]. There were additional costs for executing the study and recruitment, which vary on the basis of project goals and sample size; expenses remained lower than those on other survey platforms. Overall, this is a low-cost data collection option with great potential for high reach and customized sampling by respondent characteristics via longitudinal panel research on a social media platform.

Most demographic characteristics were similar between those who completed the study and those who were lost to follow-up, indicating that the platform is not causing biases in recruitment or testing to result in certain groups dropping out of the study at greater rates than others. Chi-square analysis showed a difference in gender between complete cases and participants lost to follow-up. This was also observed when comparing the samples to the US census data. However, a resulting sample with a greater proportion of females than males is commonly seen in surveys using a convenience sample [10,21]. Although the sample lost to follow-up shows a higher rate of current use status than that of the complete cases sample, the latter consists of a significant amount of past and current e-cigarette users, comprising over 50% of the sample. e-Cigarette use status in the completion sample is also similar to nationwide prevalence numbers, deeming the sample to still be valuable and representative.

Although the results support the feasibility of the Virtual Lab platform, there are some limitations to this study. First, a large proportion (37.6%) of the eligible starting sample of the pilot study was lost to follow-up. Efforts to bolster retention are being made for the proceeding study, including increasing compensation for participation and speed of distribution of compensation. Second, this study only examined the feasibility of Virtual Lab on Facebook. Thus, the feasibility of Virtual Lab on other social media platforms, such as Instagram, will require future research once it becomes available. Third, there were limitations in the ways to ask questions within Facebook Messenger using the Typeform platform. One of these limitations includes the inability to select more than one response for a question, such as identification of race and ethnicity. In order to address this limitation, we programmed the survey to repeat questions where multiple-choice responses would normally be available until the participant indicated that they had selected all relevant responses. Another limitation we encountered when converting the Typeform survey to Facebook Messenger was the inability to boldface words. 

Digital health is a growing field, but there has been relatively little research using social media platforms to recruit participants, deliver interventions, and collect data. The ability to follow up with participants over time and collect data in a low-cost, rapid, and relatively low-burden manner offers tremendous potential for social media health research and interventions. It can lead to more effective campaign interventions that aim to improve youth and young adults’ health behaviors, such as substance use prevention. This study suggests that such intervention studies are feasible and may be a valuable tool for researchers. Future studies should include randomized controlled trials in real-world settings on Facebook and other social media, provide social media stimuli aimed at health behavior change, and follow participants over time to evaluate outcomes.

The development and use of a platform that allows for experimentation within social media platforms is essential for the progress of mass media campaign evaluation research. Virtual Lab, a new cost-effective platform that allows for customized recruitment and longitudinal follow-up of participants and execution of survey research on Facebook, has shown to be feasible for media campaign awareness studies. Importantly, with the use of Virtual Lab, research can result in more accurate data to inform health campaigns and their dissemination.