This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.
Virtual supermarkets offer a practical and affordable setting to test the efficacy of different pricing and nudging strategies before they are implemented in the real world. Despite the advantages of using virtual supermarkets for this purpose, conducting studies in online settings is challenging with regard to recruitment and retention of sufficient and suitable participants.
To describe cost, time, and retention with regard to participants recruited using various strategies and potential sociodemographic differences between participants recruited via different strategies.
This cross-sectional study used data from a randomized controlled trial in which 455 Dutch adults with low and high educational levels were invited to shop 5 times in a 3D virtual supermarket. Participants were recruited via social media and flyers. A log that tracked the costs of and time spent on the different recruitment strategies was kept by the study team. Outcome measures included the cost of recruitment strategies, the time investment by researchers, and recruitment and attrition rates of participants in the study.
The median age of study completers was 31.0 (IQR 25.0) and 157 out of 346 study completers (45.4%) were highly educated. Out of the 455 included participants, 235 (51.6%) were recruited via social media campaigns, 131 (28.8%) via home-delivered flyers, 38 (8.4%) via flyers directly distributed by the study team, and 46 (10.1%) via word-of-mouth. Of all paid recruitment strategies, social media campaigns were the cheapest and least time-consuming, whereas the distribution of flyers by the study team was the most expensive and time-consuming recruitment strategy. Age, sex, overweight status, employment situation, and number of adults within the household varied by recruitment strategy.
Using different recruitment strategies resulted in the efficient recruitment of a representative study sample and retention of participants was relatively high. While “word-of-mouth” was the most cost- and time-effective recruitment strategy, using only one type of recruitment strategy could result in a demographically skewed study population.
Supermarkets are an important point-of-purchase setting [
Virtual supermarket environments may offer a practical and affordable means of testing the efficacy of different pricing and nudging strategies
Recruitment of a sufficiently large sample that adequately represents the target population can be difficult [
Evidence suggests that most intervention studies, that is, experimental studies online or in the real world in which investigators assign the exposure(s) to participants, use print advertising such as flyers, posters, and newspaper advertisements to recruit potential participants [
Online studies have reported on the use of several recruitment strategies [
This study is part of the “Sustainable Prevention of Cardiometabolic Risk through Nudging Health Behaviors” (Supreme Nudge) project [
Inclusion criteria were that participants had to be 18 years or older, were able to communicate in Dutch, had access to a computer with internet, had a valid email address, and regularly did the grocery shopping for their household. This study aimed to include an approximately equal number of lower and higher SEP adults determined using the proxy educational level. Given the known difficulties associated with recruiting low SEP individuals combined with the fact that only approximately 28% of the Dutch population is considered to have a low educational level [
According to the sample size calculation, at least 300 participants were needed to find a statistically significant difference in one of the main outcomes of the trial (vegetable purchases) between the control condition and experimental conditions (not yet accounting for possible attrition). Details regarding the sample size calculation can be found in
The social media campaigns and the study flyers directed potential participants to a registration website where more information about the study was provided and visitors could be redirected to a Survalyzer questionnaire for informed consent by entering their email address. Potential participants received an email with a link to the baseline questionnaire, which included questions regarding their sociodemographic characteristics and shopping habits. Inclusion criteria were assessed using the baseline questionnaire. If participants met the inclusion criteria, they received a link to the virtual supermarket and were asked to download the virtual supermarket to their computer and conduct a trial shop in which they needed to find 5 specific products from a grocery shopping list. Participants that successfully retrieved at least four out of five products were included in the study (
When assessing the eligibility of participants through a questionnaire, participants also answered questions regarding their age (years), sex (male or female), height (meters), weight (kilograms), household size (number of children and adults in the household), household net monthly income (ranging from <€1700 [~US $1815] to >€5000 [~US $6053]), highest educational level attained (primary school, secondary school, vocational education, or higher education), employment status (full-time employed, part-time employed, housewife/man, receiving benefits, retired, student, and other), responsibility for household shopping (fully responsible, mostly responsible, partly responsible, and someone else is responsible), frequency of household shopping (less than once a week, once a week, twice a week, three times a week, and more often), weekly budget for food shopping (<€25 [~US $30], €26-€50 [~US $31-US $61], €51-€100 [~US $62-US $121], €101-€150 [~US $122-US $182], €201-€250 [~US $243-US $303], €251-€300 [~US $303-US $363], and >€300 [>US $363]), and location of usual food shopping (at the market, in the supermarket, in small local shops, in organic food shops and other). After completing the final round of shopping, participants were also asked 8 questions regarding their experience of the virtual supermarket. Examples of prompts were “The program was easy to understand” and “The products I purchased in the virtual supermarket resemble my regular food purchases.” These items have been used in previous studies to assess participants’ experience of other virtual supermarkets [
We collected data on participant characteristics and recruitment method to describe the type of participants that were recruited and retained using the different recruitment strategies. Furthermore, data on the costs associated with the different recruitment strategies were collected.
We report descriptive statistics on the overall number of participants recruited and retained using the different recruitment strategies.
Descriptive statistics on the number of participants recruited and retained using the different recruitment strategies and the costs associated with these strategies are reported. The cost per recruitment strategy was calculated by dividing the total amount spent on a recruitment strategy by the number of participants recruited via the corresponding strategy. This was also done for the time researchers spent on each recruitment strategy.
Differences in participant characteristics between those who signed up for the study and met the inclusion criteria, participants that successfully conducted a training shop, participants that completed the study (ie, carried out all 5 rounds of shopping), and study noncompleters were inspected visually and formally tested. Differences between completers and noncompleters and differences in population characteristics between the different recruitment strategies were assessed using a one-way ANOVA for continuous outcome variables (ie, BMI and age) and the Pearson chi-square test in the case of categorical outcome variables (eg, educational level and income). Non-normally distributed continuous outcome variables were log transformed. Analyses were conducted in STATA version 14.1 (StataCorp) and a
Participants were recruited between September and December 2018.
Flow chart of participant recruitment and retention.
In
The cost, time, and percentage of participants in each recruitment strategy during three phases of the study.
Recruitment type | Cost (€a) | Time (min) | Registered | Included | Completed | ||||||||
n (%) |
Cost (€) per participant | Time (min) per participant | n (%) |
Cost (€) per participant | Time (min) per participant | n (%) |
Cost (€) per participant | Time (min) per participant | |||||
Social media campaigns | 1.298 | 1440 | 426 (52.7) | 3 | 3 | 235 (51.6) | 6 | 6 | 170 (49.1) | 8 | 8 | ||
Flyers distributed to home addresses | 2.142 | 2700 | 240 (29.7) | 9 | 11 | 131 (28.8) | 16 | 21 | 106 (30.6) | 20 | 25 | ||
Flyers distributed directly by the team | 0.558 | 3000 | 50 (6.2) | 11 | 60 | 38 (8.4) | 16 | 71 | 27 (7.8) | 21 | 100 | ||
Word-of-mouth | 0 | 0 | 55 (6.8) | 0 | 0 | 46 (10.1) | 0 | 0 | 40 (11.6) | 0 | 0 | ||
Unknown | N/Ab | N/A | 38 (4.7) | N/A | N/A | 5 (1.1) | N/A | N/A | 3 (0.9) | N/A | N/A |
a€1 = Approximately US $1.2.
bN/A: not applicable.
The percentage of participants that were included in the study and completed the study compared to those that registered in each recruitment strategy.
Recruitment type | Registered, N | Included, n (%) | Completed, n (%) |
Social media campaigns | 426 | 235 (55.2) | 170 (39.9) |
Flyers distributed to home addresses | 240 | 131 (54.6) | 106 (44.2) |
Flyers distributed directly by the team | 50 | 38 (76.0) | 27 (54.0) |
Word-of-mouth | 55 | 46 (83.6) | 40 (72.7) |
Unknown | 38 | 5 (13.2) | 3 (7.9) |
Characteristics of participants who signed up for the study and met the inclusion criteria, participants that successfully conducted a training shop, participants that completed the study (ie, carried out all 5 rounds of shopping), and study noncompleters are presented in
Participant characteristics by recruitment strategy can be found in
Characteristics of the study population who completed and did not complete the study.
Characteristics | Total sample (N=809) | Sample included (N=455) | Completers (N=346) | Noncompleters (N=463) | ||||||||
Median (IQR) age, years | 35.0 (27.0) | 31.0 (25.0) | 31.0 (24.0) | 37.0 (30.0) | .21 | |||||||
Female sex, n (%) | 515 (63.7) | 284 (62.4) | 215 (62.1) | 299 (64.6) | .53 | |||||||
Mean (SD) BMIa | 25.3 (5.3) | 24.9 (4.8) | 24.9 (4.9) | 25.6 (5.5) | .05 | |||||||
Overweight status, n (%): overweight or obesea,b | 348 (43.8) | 176 (39.9) | 129 (38.9) | 219 (48.8) | .01 | |||||||
|
|
|
|
|
.07 | |||||||
|
Low educational level | 90 (11.1) | 43 (9.5) | 31 (9.0) | 59 (12.7) |
|
||||||
|
Medium educational level | 379 (46.8) | 212 (46.6) | 158 (45.7) | 221 (47.7) |
|
||||||
|
High educational level | 337 (41.7) | 200 (44.0) | 157 (45.4) | 180 (38.9) |
|
||||||
|
|
|
|
|
.32 | |||||||
|
€0-€1700e | 306 (38.3) | 172 (38.6) | 123 (36.5) | 183 (40.3) |
|
||||||
|
€1701-€2500 | 195 (24.4) | 105 (23.5) | 81 (24.0) | 114 (25.1) |
|
||||||
|
€2501-€3500 | 140 (17.5) | 84 (18.8) | 69 (20.5) | 71 (15.6) |
|
||||||
|
More than €3501 | 159 (19.9) | 90 (20.2) | 70 (20.8) | 89 (19.6) |
|
||||||
|
|
|
|
|
.12 | |||||||
|
Full time job | 183 (22.6) | 108 (23.7) | 90 (26.0) | 93 (20.5) |
|
||||||
|
Part time job | 206 (25.5) | 112 (24.6) | 84 (24.3) | 122 (26.9) |
|
||||||
|
Student | 187 (23.1) | 118 (25.9) | 85 (24.6) | 78 (17.2) |
|
||||||
|
Unemployedf | 204 (25.2) | 101 (22.2) | 78 (22.5) | 126 (27.8) |
|
||||||
|
Entrepreneur or other | 29 (3.6) | 16 (3.5) | 9 (2.6) | 20 (4.4) |
|
||||||
|
|
|
|
|
|
|||||||
|
At least two adults | 547 (67.6) | 312 (68.6) | 243 (70.2) | 314 (69.2) | .40 | ||||||
|
At least one child | 263 (32.5) | 141 (31.0) | 109 (31.5) | 154 (33.9) | .60 | ||||||
|
|
|
|
|
.56 | |||||||
|
Apple-based | 106 (13.1) | 54 (11.9) | 44 (12.7) | 62 (13.7) |
|
||||||
|
Windows-based | 495 (61.2) | 293 (64.4) | 213 (61.6) | 291 (64.1) |
|
||||||
|
Other or unknown | 93 (11.5) | 42 (9.2) | 33 (9.5) | 51 (11.2) |
|
||||||
|
Two or more computers/laptops | 115 (14.2) | 66 (14.5) | 56 (16.2) | 59 (13.0) |
|
||||||
|
|
|
|
|
<.001 | |||||||
|
Less than 3 years | 411 (50.8) | 245 (53.8) | 183 (52.9) | 228 (49.2) |
|
||||||
|
Between 3 and 6 years | 310 (38.3) | 175 (38.5) | 140 (40.5) | 170 (36.7) |
|
||||||
|
Older than 6 years | 70 (8.7) | 30 (6.6) | 20 (5.8) | 50 (10.8) |
|
||||||
|
Unknown | 18 (2.2) | 5 (1.1) | 3 (0.9) | 15 (3.2) |
|
a14 missing values.
bParticipants with a BMI higher than 25.0 were considered overweight or obese.
cLow educational level included participants with primary education, medium educational level included participants with lower or higher secondary education and high educational level included participants with tertiary education.
dNine missing values.
e€1 = Approximately US $1.2.
fIncludes those who are retired, unemployed, unable to work and/or receiving social benefits and housewives/husbands.
Characteristics of the study population for the entire sample and stratified by recruitment strategy.a
Characteristics | Total sample (N=455) | Social media (N=235) | Flyers to home addresses (N=131) | Flyers from study team (N=38) | Word-of-mouth (N=46) | |||||||||
Median (IQR) age, years | 31.0 (25.0) | 25.0 (18.0) | 46.0 (26.0) | 39.0 (25.0) | 27.0 (25.0) | .02 | ||||||||
Female sex, n (%) | 284 (62.4) | 131 (55.7) | 92 (70.2) | 26 (68.4) | 32 (69.6) | .03 | ||||||||
Mean (SD) BMIb | 24.9 (4.8) | 24.7 (4.9) | 25.4 (4.4) | 24.7 (5.4) | 24.3 (5.4) | .63 | ||||||||
Overweight status: overweight or obese statusb,c, n (%) | 176 (39.5) | 87 (38.5) | 65 (50.0) | 10 (27.0) | 11 (24.4) | .01 | ||||||||
|
|
.22 | ||||||||||||
|
Low educational level | 43 (9.5) | 19 (8.1) | 19 (14.5) | 1 (2.6) | 3 (6.5) |
|
|||||||
|
Medium educational level | 212 (46.6) | 111 (47.2) | 63 (48.1) | 16 (42.1) | 22 (47.8) |
|
|||||||
|
High educational level | 200 (44.0) | 105 (44.7) | 49 (37.4) | 21 (55.3) | 21 (45.7) |
|
|||||||
|
|
.34 | ||||||||||||
|
€0-€1700f | 172 (38.1) | 100 (42.7) | 38 (29.0) | 15 (39.5) | 17 (37.0) |
|
|||||||
|
€1701-€2500 | 105 (23.3) | 45 (19.5) | 39 (29.8) | 11 (28.9) | 8 (17.4) |
|
|||||||
|
€2501-€3500 | 84 (18.6) | 41 (17.7) | 26 (19.8) | 5 (13.2) | 11 (23.9) |
|
|||||||
|
More than €3501 | 90 (20.0) | 45 (19.5) | 28 (21.4) | 7 (18.4) | 10 (21.7) |
|
|||||||
|
|
.02 | ||||||||||||
|
Full-time job | 108 (23.7) | 50 (21.3) | 35 (26.7) | 9 (23.7) | 13 (28.3) |
|
|||||||
|
Part-time job | 112 (24.6) | 48 (20.4) | 31 (23.7) | 14 (36.8) | 18 (39.1) |
|
|||||||
|
Student | 118 (25.9) | 89 (37.9) | 9 (6.9) | 6 (15.8) | 13 (28.3) |
|
|||||||
|
Unemployedg | 101 (22.2) | 42 (17.9) | 50 (38.2) | 5 (13.2) | 2 (4.3) |
|
|||||||
|
Entrepreneur or other | 16 (3.5) | 6 (2.6) | 6 (4.6) | 4 (10.5) | 0 (0.0) |
|
|||||||
|
|
|
||||||||||||
|
At least two adults | 312 (68.6) | 152 (64.7) | 97 (74.0) | 22 (57.9) | 38 (82.6) | .01 | |||||||
|
At least one child | 141 (31.0) | 74 (31.5) | 43 (32.8) | 12 (31.6) | 12 (26.1) | .93 |
aUnknown recruitment strategy was not included in the analyses.
bNine missing values; 6 missing values for social media and 1 missing value for the other strategies.
cParticipants with a BMI higher than 25.0 were considered overweight or obese.
dLow educational level included participants with primary education, medium educational level included participants with lower or higher secondary education and high educational level included participants with tertiary education.
eFour missing values for social media.
f€1 = Approximately US $1.2.
gIncludes those who are retired, unemployed, unable to work and/or receiving social benefits and housewives/husbands.
This study found that the recruitment strategy word-of-mouth involved zero costs, required no time effort on the part of the researchers, and yielded the highest study-completion rate. Of all paid recruitment strategies, the least expensive strategy was social media campaigns. Social media campaigns also yielded the highest absolute registration and completion rates. Sociodemographic characteristics such as age, sex, and overweight status varied with the recruitment strategy.
Effective recruitment approaches are those that lead to the creation of a representative and large enough sample of study participants [
Participants recruited via social media were less likely to complete the study compared to those recruited by flyers and word-of-mouth. In this study, word-of-mouth was found to be surprisingly effective; 10.1% (46/455) of the study population was recruited via word-of-mouth without the researchers actively encouraging participants to recruit peers. A disadvantage associated with recruitment via word-of-mouth, or via the exclusive use of a single recruitment strategy in general, is that it may yield a demographically skewed study population [
The current study results also suggest that recruitment strategies directly involving people (ie, active recruitment strategies using word-of-mouth or flyers distributed by the study team) lead to higher retention rates compared to recruitment strategies that do not involve personal contact (ie, passive recruitment strategies using social media campaigns and flyers distributed to homes). By contrast, the reach of social media campaigns and flyers sent to homes was much larger compared to the other recruitment strategies used. Moreover, social media campaigns can be used to target certain groups that are underrepresented in the study sample [
A strength of this study is the use of different recruitment strategies (eg, Facebook and flyers), which led to the creation of a diverse study population in a relatively short period. Furthermore, a relatively high completion rate of 76.0% (346/455) was found; this is particularly interesting in light of the fact that participants were asked to conduct 5 rounds of shopping over the course of 5 consecutive weeks. A limitation of this study is the limited generalizability of the results. While this study successfully recruited a relatively representative sample using traditional and novel recruitment strategies within the specified timeframe, the same might not apply to different studies in different settings. For example, we do not know whether our recruitment efforts were successful because of the methods used, because of the type of study (virtual supermarket study) that participants signed up for, or because of the guaranteed incentive of
Regarding paid recruitment strategies, social media campaigns, particularly via Facebook, were more cost-effective than other more traditional methods. The unpaid recruitment strategy “word-of-mouth” yielded the highest study completion rate and required the least amount of time and effort on the part of the researchers. Employing only 1 recruitment strategy may lead to the creation of a demographically skewed sample.
Details regarding the SN VirtuMart trial.
Participant experience of the SN VirtuMart trial.
socioeconomic position
The Supreme Nudge (CVON2016–04) project is funded by The Netherlands Heart Foundation and The Netherlands Organization for Health Research and Development (ZonMw).
JH, JM, JL, WW, and JB helped design the study. NvdL constructed the virtual supermarket software. JH conducted analyses and interpreted the results. JH and JM drafted the paper. All authors reviewed and edited the manuscript.
None declared.