Recruitment of Participants for a 3D Virtual Supermarket: Cross-sectional Observational Study

Introduction

Supermarkets are an important point-of-purchase setting [1] particularly applicable in studies targeting diet as a risk factor for noncommunicable diseases. Examples of promising strategies to improve population diets in supermarket settings include pricing and nudging strategies. Studies have shown that pricing (eg, price promotions on healthier products) and nudging (eg, prominent placement of healthier products) strategies can be effective in increasing purchases of healthy foods [2-5]. While pricing strategies can be seen as “harder” approaches, nudges can be seen as “softer,” as nudges are less intrusive and simply alter the choice environment to make the healthy choice the easier choice, without removing the unhealthy choice [3]. Despite their promise, investigating the effectiveness of pricing (especially increasing the price of unhealthy foods) and nudging strategies in real supermarkets is costly (eg, purchases of materials or compensation of the supermarket for loss of revenue) and time consuming for researchers (eg, recruitment of participants, collecting receipts, imputing purchasing data).

Virtual supermarket environments may offer a practical and affordable means of testing the efficacy of different pricing and nudging strategies before they are implemented in real-world settings. Virtual supermarket environments include online web shops for grocery shopping and 3D virtual supermarkets. The 3D virtual supermarket imitates a real-life supermarket by duplicating the layout and using 3D products. Once a virtual supermarket is constructed, researchers can easily manipulate the supermarket environment by adjusting prices, and adding nudges such as posters or frames around products. A number of 3D virtual supermarkets have been developed to date [6-9]. Overall, previous studies of 3D virtual supermarkets have indicated that they are a valid tool for investigating the effect of pricing strategies on food purchases [6], also when compared to real-life purchases [8]. As such, 3D virtual supermarkets appear to be a valid alternative to real-world supermarkets as an environment in which the efficacy of nudging and pricing strategies can be studied.

Recruitment of a sufficiently large sample that adequately represents the target population can be difficult [10]. Reporting on the effectiveness of recruitment strategies facilitates improvements in the design and methods of future studies. The effectiveness of recruitment strategies depends on several factors including the study design, setting, study population, and the use of incentives [11]. Despite the advantages of 3D virtual supermarkets, it may be more difficult to recruit participants for online studies compared to interventions in real-world settings, as participants need to be adept at using technology and need to have access to a smartphone or computer with internet access [12,13]. Difficulty with recruitment can lead to longer recruitment times, increased use of resources, and reduced sample size and power. Additionally, it may be more difficult to retain participants as compared to studies in real-world settings due to the lack of personal contact [11,13] for example, which may lead to selection bias and loss of statistical power [12]. Furthermore, particularly when using a within-subject study design where participants are asked to conduct multiple rounds of shopping over a specified period, long waits between these shopping trips might lead to diminished interest, as the novelty of the online intervention decreases, and increased frustration, resulting in additional attrition. While some degree of attrition is largely inevitable, excessive attrition reduces statistical power, increases bias, and leads to lower generalizability of results [13].

Evidence suggests that most intervention studies, that is, experimental studies online or in the real world in which investigators assign the exposure(s) to participants, use print advertising such as flyers, posters, and newspaper advertisements to recruit potential participants [14,15]. Challenges related to the recruitment and retention of participants in online studies have led to the use of alternative recruitment strategies that rely on internet advertising and social media [13]. These innovative recruitment strategies are attractive due to their potential to reach a larger number of people, apparent cost-effectiveness, and ability to reach populations that are considered hard-to-reach (eg, young adults and adults with a lower educational level) [13,15]. Despite the growing popularity of recruitment via social media, data on the effectiveness of this strategy in the context of online studies are limited [15].

Online studies have reported on the use of several recruitment strategies [6,11]. However, as far as we are aware, no studies investigating the effectiveness of social media as a recruitment strategy have been conducted in the Netherlands to date [13]. The aim of this study was to describe cost, time, and retention rates with regard to different recruitment strategies (including innovative and traditional recruitment strategies), and the sociodemographic characteristics of participants recruited via these different strategies.

Methods

Study Overview

This study is part of the “Sustainable Prevention of Cardiometabolic Risk through Nudging Health Behaviors” (Supreme Nudge) project [16]. Data presented in this paper describe the cost, time, and retention rates associated with different recruitment strategies from a larger study investigating the efficacy of nudging and pricing strategies on food purchasing behavior in a virtual supermarket and effect modification by socioeconomic position (SEP). Results of this trial are reported elsewhere [17] and additional details about the study aims and design are presented in Multimedia Appendix 1. The study design and procedures for this virtual supermarket study were approved by the Medical Ethics Review Committee of VU University Medical Centre (OHRP: IRB00002911), and all participants provided informed consent.

Inclusion Criteria

Inclusion criteria were that participants had to be 18 years or older, were able to communicate in Dutch, had access to a computer with internet, had a valid email address, and regularly did the grocery shopping for their household. This study aimed to include an approximately equal number of lower and higher SEP adults determined using the proxy educational level. Given the known difficulties associated with recruiting low SEP individuals combined with the fact that only approximately 28% of the Dutch population is considered to have a low educational level [18], we included individuals with both low and medium educational level in the lower SEP group. Adults were considered low or medium SEP if their highest obtained educational level was primary education, intermediate vocational education, or higher secondary education. As shopping was done for the household, only 1 person per household was allowed to participate.

Recruitment of Participants

According to the sample size calculation, at least 300 participants were needed to find a statistically significant difference in one of the main outcomes of the trial (vegetable purchases) between the control condition and experimental conditions (not yet accounting for possible attrition). Details regarding the sample size calculation can be found in Multimedia Appendix 1. Both traditional and more novel recruitment strategies were used to recruit participants. The traditional recruitment methods included advertising via flyers. The more novel recruitment strategy included using social media advertising, which has become an increasingly popular approach [11]. Flyers were distributed directly to participants on the street, at local events, and in real-world supermarkets. The flyers contained information on the inclusion criteria, activities within the study, and the reward for completing the study (a guaranteed incentive of €25 [~US $30]). Distribution of the flyers took place in October 2018. Flyers were also distributed around the University campus in October 2018. Approximately 500 flyers were printed at €0.35 (US $0.42) per flyer. Flyers were also delivered to addresses in low-income neighborhoods via postal services and by the study team. Low-income neighborhoods (ie, those with an average household income per resident under the median Dutch household income) were selected in order to increase participation rates of lower SEP individuals [19,20]. The social media campaign consisted of pay-per-click Facebook and Instagram campaigns and ran from mid-September to mid-December 2018. A professional was hired to set up the Facebook and Instagram campaigns. Campaigns were separated for low and high SEP target groups. Using existing and nondisclosed Facebook algorithms, the campaigns were adapted automatically based on what worked best for each target group. The target groups of the 2 Facebook campaigns were adjusted according to the characteristics of participants included in the study at a particular point in time. For example, if too few men had been recruited for the study after a few weeks, the Facebook campaign was adjusted to only include men in order to increase the recruitment of men. In addition, a Twitter post was created using the Supreme Nudge account (with over 250 followers at that time). Participants recruited from Facebook, Instagram, and Twitter were considered to be recruited using “social media strategies.” Although the researchers did not actively encourage participants to recruit others (eg, there was no incentive for participants to recruit others for the study), participants were also recruited by word-of-mouth at no cost to the researchers. A log to track the costs of the different recruitment strategies was kept by the study team. Furthermore, a log tracking the development of recruitment material (eg, posters and Facebook campaign) by the researchers and the distribution of posters by the researchers was kept. We intended to recruit participants between September and December 2018. If insufficient participants completed the study within this period (N≤300), recruitment would have continued in January 2019.

Study Procedure

The social media campaigns and the study flyers directed potential participants to a registration website where more information about the study was provided and visitors could be redirected to a Survalyzer questionnaire for informed consent by entering their email address. Potential participants received an email with a link to the baseline questionnaire, which included questions regarding their sociodemographic characteristics and shopping habits. Inclusion criteria were assessed using the baseline questionnaire. If participants met the inclusion criteria, they received a link to the virtual supermarket and were asked to download the virtual supermarket to their computer and conduct a trial shop in which they needed to find 5 specific products from a grocery shopping list. Participants that successfully retrieved at least four out of five products were included in the study (Multimedia Appendix 1). Participants were then asked to shop 5 times in the virtual supermarket over the course of 5 consecutive weeks. During the first virtual shopping trip, participants were asked the following: “Imagine that you only have herbs at home and you decide to do the shopping for the entire household (people for whom you normally do the grocery shopping for) for one week. You receive a budget from us. You buy all your daily meals, snacks and drinks for the entire week (toiletries and alcohol are not for sale in this supermarket). The budget is only a guideline; it is possible to spend a little less or a little more.” For the subsequent 4 shopping trips, this prompt was updated to include the information that their usual supermarket was now closed and, as such, that they had to do their shopping in a new supermarket. Participants received guaranteed incentives for completing weekly shops: after participants completed their first shopping trip, they received a €5 (~US $6.05) gift voucher and after completing all 5 rounds, participants received an additional €20 (~US $24) gift voucher.

Participant Characteristics

When assessing the eligibility of participants through a questionnaire, participants also answered questions regarding their age (years), sex (male or female), height (meters), weight (kilograms), household size (number of children and adults in the household), household net monthly income (ranging from <€1700 [~US $1815] to >€5000 [~US $6053]), highest educational level attained (primary school, secondary school, vocational education, or higher education), employment status (full-time employed, part-time employed, housewife/man, receiving benefits, retired, student, and other), responsibility for household shopping (fully responsible, mostly responsible, partly responsible, and someone else is responsible), frequency of household shopping (less than once a week, once a week, twice a week, three times a week, and more often), weekly budget for food shopping (<€25 [~US $30], €26-€50 [~US $31-US $61], €51-€100 [~US $62-US $121], €101-€150 [~US $122-US $182], €201-€250 [~US $243-US $303], €251-€300 [~US $303-US $363], and >€300 [>US $363]), and location of usual food shopping (at the market, in the supermarket, in small local shops, in organic food shops and other). After completing the final round of shopping, participants were also asked 8 questions regarding their experience of the virtual supermarket. Examples of prompts were “The program was easy to understand” and “The products I purchased in the virtual supermarket resemble my regular food purchases.” These items have been used in previous studies to assess participants’ experience of other virtual supermarkets [6,7]. Answering options were 5-point Likert scales ranging from “strongly disagree” to “strongly agree.” Results regarding participants’ experience of the virtual supermarket is presented in Multimedia Appendix 2.

Outcome Measures

We collected data on participant characteristics and recruitment method to describe the type of participants that were recruited and retained using the different recruitment strategies. Furthermore, data on the costs associated with the different recruitment strategies were collected.

Analyses

Overall Recruitment and Retention of Participants

We report descriptive statistics on the overall number of participants recruited and retained using the different recruitment strategies.

Recruitment Cost, Time, and Retention Rates According to Recruitment Strategy

Descriptive statistics on the number of participants recruited and retained using the different recruitment strategies and the costs associated with these strategies are reported. The cost per recruitment strategy was calculated by dividing the total amount spent on a recruitment strategy by the number of participants recruited via the corresponding strategy. This was also done for the time researchers spent on each recruitment strategy.

Participant Characteristics

Differences in participant characteristics between those who signed up for the study and met the inclusion criteria, participants that successfully conducted a training shop, participants that completed the study (ie, carried out all 5 rounds of shopping), and study noncompleters were inspected visually and formally tested. Differences between completers and noncompleters and differences in population characteristics between the different recruitment strategies were assessed using a one-way ANOVA for continuous outcome variables (ie, BMI and age) and the Pearson chi-square test in the case of categorical outcome variables (eg, educational level and income). Non-normally distributed continuous outcome variables were log transformed. Analyses were conducted in STATA version 14.1 (StataCorp) and a P-value of .05 was used to indicate statistical significance.

Results

Overall Recruitment and Retention of Participants

Participants were recruited between September and December 2018. Figure 1 shows the flow of participants over the trial period. Regarding the recruitment campaigns, 3427 people clicked on the advertisement and were directed to the registration website. Initially, the campaign was much more likely to reach women (often aged 45 and older), after which the campaign was adapted to reach more men. This resulted in the campaign mostly reaching men under the age of 25. Around 17,500 people received a study flyer in their mailbox and 450 people received a study flyer directly from the study team. After being recruited by the different strategies and registering via the website, 809 people provided informed consent and were eligible for participation. Of those, 455 successfully conducted the training shop and were included in the study. A total of 346 participants completed the study and 318 participants generated usable data for all 5 rounds of shopping (ie, generated data that could be linked back to the participant and in which the login code corresponded to the participants’ assigned budget). Of all participants who successfully conducted the training shop, 76.0% (346/455) completed the study.

‎View this figure

Recruitment Cost, Time, and Retention Rates According to Recruitment Strategy

In Table 1, the costs of various recruitment strategies and the number of participants recruited using the different recruitment strategies are displayed. Over half (n=426) of the 809 adults who signed up for the study were recruited using the social media campaigns (mostly Facebook) and 29.7% (n=240) were recruited using flyers distributed to home addresses. Social media campaigns resulted in the highest absolute registration, inclusion in the study, and study completion. Distribution of flyers by the research team was the most expensive strategy, while the social media campaigns were the least expensive paid strategy in terms of the cost per person after completion of the study. Regarding the Facebook campaign, the cost per click was estimated to be €0.14 (US $0.17). The unpaid recruitment strategy “word-of-mouth” required no time investment by the research team and involved no recruitment costs. Regarding paid recruitment strategies, social media campaigns were the most time-efficient and flyers distributed directly by the study team were the least time-efficient for the researchers (8 minutes per study completer compared to 100 minutes per study completer, respectively). While the costs of flyers distributed by the team were similar to flyers distributed to home addresses (ie, €21 [US $25.52] and €20 [US $24.21], respectively), the time investment for flyers distributed directly by the study team was much higher compared to flyers distributed to home addresses (ie, 100 minutes and 25 minutes, respectively). The highest study completion rate was achieved with the “word-of-mouth” recruitment strategy (from 6.8% [55/809] of registered participants to 11.6% [40/346] of study completers). These results are also confirmed when calculating the percentage of participants that were included and completed the study compared to those that registered for each recruitment strategy (Table 2). In total, 72.7% (40/55) of participants recruited via word-of-mouth completed the study, compared to 39.9% (170/426), 44.2% (106/240), 54% (27/50), and 3.4% (3/38) of participants recruited via social media campaigns, flyers distributed to home addresses, flyers distributed directly by the study team, and unknown recruitment strategies, respectively (Table 2).

Participant Characteristics

Characteristics of participants who signed up for the study and met the inclusion criteria, participants that successfully conducted a training shop, participants that completed the study (ie, carried out all 5 rounds of shopping), and study noncompleters are presented in Table 3. The median age of participants included in the study was 31 (SD 25.0) and the majority of participants were female. Most participants included in the study had a medium educational level and a monthly household net income below €1700 (~US $2057). Study completers were somewhat younger than study noncompleters (median age of 31.0 compared to 37.0, respectively), but this difference was not statistically significant (P=.21). Study noncompleters were statistically significantly more often overweight (P=.01) and had older computers (P≤.001) compared to study completers. No other large observable differences in participant characteristics were found between study completers and noncompleters. For study completers, the average time in days between participants’ first shop and last shop was 38.1 (SD 13.1).

Participant characteristics by recruitment strategy can be found in Table 4. The participant characteristics age, overweight status, employment situation, and the percentage of households with at least two adults differed statistically significantly by recruitment strategy (Table 4). For example, the average age of participants recruited via social media was lower, and a larger proportion of overweight or obese participants were recruited via flyers distributed to home addresses.

Discussion

Principal Findings

This study found that the recruitment strategy word-of-mouth involved zero costs, required no time effort on the part of the researchers, and yielded the highest study-completion rate. Of all paid recruitment strategies, the least expensive strategy was social media campaigns. Social media campaigns also yielded the highest absolute registration and completion rates. Sociodemographic characteristics such as age, sex, and overweight status varied with the recruitment strategy.

Effective recruitment approaches are those that lead to the creation of a representative and large enough sample of study participants [21]. The combination of different recruitment strategies resulted in the recruitment of a relatively diverse study population in the space of 3 months. Social media campaigns were the most cost-efficient paid recruitment strategy employed and word-of-mouth was free, required no time effort on the part of researchers, and yielded in the highest retention rates. These results are comparable to previous studies carried out among the general population that report on the effectiveness and costs of recruitment via social media campaigns and other more traditional recruitment strategies [11,15,22]. For example, a study by Frandsen et al [15] found that social media drew more interest and was more cost effective than traditional methods such as flyering at baseline. Also, a systematic review investigating the effectiveness of Facebook as a recruitment strategy found reduced costs, shorter recruitment periods, better representation, and improved participant selection compared to traditional recruitment methods [22]. Surprisingly, a comparable study investigating the effectiveness of online methods to recruit participants for a virtual supermarket study found Facebook advertisements to be less successful as a recruitment strategy than was anticipated [6]. The use of a guaranteed incentive in this study and other studies that have successfully used Facebook to recruit participants (eg, [13]) may provide an explanation for the difference in findings between this study and the aforementioned study. A guaranteed incentive is likely to attract more people than no incentive or a prize lottery, for example. Future studies investigating the efficacy of social media campaigns for the recruitment of participants could investigate the role of incentives alongside this strategy.

Participants recruited via social media were less likely to complete the study compared to those recruited by flyers and word-of-mouth. In this study, word-of-mouth was found to be surprisingly effective; 10.1% (46/455) of the study population was recruited via word-of-mouth without the researchers actively encouraging participants to recruit peers. A disadvantage associated with recruitment via word-of-mouth, or via the exclusive use of a single recruitment strategy in general, is that it may yield a demographically skewed study population [15]. Contrary to the previous research finding that only age varied by recruitment strategy [11], we found that other demographic variables such as household composition, overweight status, sex, age, and employment situation all varied by recruitment strategy. Overall, our results suggest that it is important to use several different recruitment strategies if the aim is to include a diverse population (eg, younger and older adults with low and high SEP) in a study. Similarly, a systematic review investigating strategies for the successful recruitment of young adults to healthy lifestyle programs found that single recruitment strategies are less effective than mixed strategies, as fewer participants were recruited and higher attrition rates were reported when using a single recruitment strategy exclusively [14]. Nevertheless, despite using several recruitment strategies and targeting the social media campaigns to people with specific characteristics (eg, SEP, age, and sex), the recruitment strategies did not result in a sample that perfectly represented the target population. Instead, the study included a slightly younger population with more females and more highly educated participants. Differences in sociodemographic characteristics between this study sample and the average Dutch population may have been caused by the inclusion criteria of the study such as being the primary shopper for the household (leading to inclusion of more female participants) and the type of recruitment strategy used (eg, younger people may be more likely to be recruited via Facebook).

The current study results also suggest that recruitment strategies directly involving people (ie, active recruitment strategies using word-of-mouth or flyers distributed by the study team) lead to higher retention rates compared to recruitment strategies that do not involve personal contact (ie, passive recruitment strategies using social media campaigns and flyers distributed to homes). By contrast, the reach of social media campaigns and flyers sent to homes was much larger compared to the other recruitment strategies used. Moreover, social media campaigns can be used to target certain groups that are underrepresented in the study sample [22]. As such, neither active nor passive recruitment strategies are necessarily superior to the other [14]. Rather, it appears to be important to use a combination of both strategies, as active recruitment methods enhance recruitment and retention rates, but also require the most resources. However, despite the higher attrition rates associated with recruitment by means of passive strategies, these strategies do seem to have a wider reach and require only limited resources (especially when using social media) as compared to active recruitment strategies.

Strengths and limitations

A strength of this study is the use of different recruitment strategies (eg, Facebook and flyers), which led to the creation of a diverse study population in a relatively short period. Furthermore, a relatively high completion rate of 76.0% (346/455) was found; this is particularly interesting in light of the fact that participants were asked to conduct 5 rounds of shopping over the course of 5 consecutive weeks. A limitation of this study is the limited generalizability of the results. While this study successfully recruited a relatively representative sample using traditional and novel recruitment strategies within the specified timeframe, the same might not apply to different studies in different settings. For example, we do not know whether our recruitment efforts were successful because of the methods used, because of the type of study (virtual supermarket study) that participants signed up for, or because of the guaranteed incentive of €25 (US $30). Another limitation of this study is that the study population was self-selecting, which could have led to the creation of a nonrepresentative study population (eg, due to the inclusion of more highly motivated adults). This type of bias may be inherent to this type of research in a community-based setting in which participants, by definition, need to sign up for a study themselves rather than be recruited by a physician, for example. This self-selection bias could, for example, be quantified by comparing the sociodemographic characteristics of the study sample with the sociodemographic characteristics of adults who received the study flyers.

Conclusion

Regarding paid recruitment strategies, social media campaigns, particularly via Facebook, were more cost-effective than other more traditional methods. The unpaid recruitment strategy “word-of-mouth” yielded the highest study completion rate and required the least amount of time and effort on the part of the researchers. Employing only 1 recruitment strategy may lead to the creation of a demographically skewed sample.