Background: Social media platforms have experienced unprecedented levels of growth and usage over the past decade, with Facebook hosting 2.7 billion active users worldwide, including over 200 million users in the United States. Facebook users have been underutilized and understudied by the academic community as a resource for participant recruitment.
Objective: We performed a pilot study to explore the efficacy and cost-effectiveness of Facebook advertisements for the recruitment of an online agricultural health and safety survey.
Methods: We undertook a 1-week advertising campaign utilizing the integrated, targeted advertising platform of Facebook Ads Manager with a target-spending limit of US $294. We created and posted three advertisements depicting varying levels of agricultural safety adoption leading to a brief survey on farm demographics and safety attitudes. We targeted our advertisements toward farm mothers aged 21-50 years in the United States and determined cost-effectiveness and potential biases. No participant incentive was offered.
Results: We reached 40,024 users and gathered 318 advertisement clicks. Twenty-nine participants consented to the survey with 24 completions. Including personnel costs, the cost per completed survey was US $17.42. Compared to the distribution of female producers in the United States, our advertisements were unexpectedly overrepresented in the eastern United States and were underrepresented in the western United States.
Conclusions: Facebook Ads Manager represents a potentially cost-effective and timely method to recruit participants for online health and safety research when targeting a specific population. However, social media recruitment mirrors traditional recruitment methods in its limitations, exhibiting geographic, response, and self-selection biases that need to be addressed.
Researchers are facing a range of issues in recruiting participants, including decreasing response rates and increased difficulties in reaching participants through traditional modes such as phone and paper [- ]. In turn, these challenges are associated with increased recruitment costs. As the presence of social media has increased and as our communication strategies have shifted, researchers are increasingly looking to these systems for potential recruitment to address current challenges.
Facebook is the largest social media platform with over 1.8 billion daily active users and 2.7 billion monthly active users . Of these, over 200 million users are within the United States. This high proportion of users allows Facebook advertising to reach potential research participants in a short period of time. As of 2019, up to 90% of US adults use the internet, a number that has been steadily growing [ ]. Additionally, although 24% of Americans do not have an internet connection at home, 17% of total US adults use a smartphone without home internet access, thereby increasing the feasibility of internet-based research [ ].
One of the largest barriers in recruitment is cost. Mailed survey costs are relatively evenly distributed, ranging from US $6.51 to $30.24; however, this includes high personnel costs [- ]. Telephone surveys have been observed to range from US $29 to $99 per participant, also requiring high personnel costs [ , ]. Some online health and behavioral studies reported costs as low as US $0.64 per participant, whereas others reported costs up to US $33 per participant; however, the median cost tends to be skewed toward the lower end of the scale [ - ].
Both telephone and mailed surveys require preexisting information to reach a targeted population, whereas Facebook’s integrated Ads Manager utilizes its internal user data to determine such demographics. However, this broad targeting introduces further concerns over multiple biases [, ]. Although more traditional recruitment methods have been used and evaluated for agricultural populations, often with mixed results, our review of the literature indicates that there has been little data reported on the use of Facebook or other social media platforms for recruitment [ - ]. Therefore, we undertook this pilot study to investigate the effectiveness of targeted advertisements on Facebook as a tool to recruit participants in the agricultural sector, and to assess possible self-selection and geographic biases in this sampling method.
To determine the feasibility and cost-effectiveness of Facebook advertisements for an agricultural health and safety survey, we ran an advertising campaign consisting of three separate advertisements, each with a different agricultural image. The campaign was performed on Facebook from March 9, 2020 to March 16, 2020 at a total cost of US $294, distributed evenly among the three advertisements. Images were selected based on the apparent safety adoption displayed to determine if advertisement imagery affected performance. This included low (overturned tractor), neutral (child feeding a calf), and high (woman sweeping a barn in full personal protective equipment) safety adoption (see- for examples of advertising images). These advertisements were run concurrently through the National Children’s Center for Rural and Agricultural Health and Safety (NCCRAHS) Facebook page. We investigated the cost per click, cost per participant, and potential geographical biases that arose during this campaign.
We chose the following inclusion criteria to target our advertisements: individuals currently living in the United States, identified as female, within the ages of 21-50 years, and had a “role,” as defined by Facebook, in Farming, Fishing, or Forestry (FFF; delineated by Facebook as Demographics > Work > Industries > FFF). Facebook reported 170,000 eligible users according to these criteria. This campaign directed users who clicked on the advertisement to an online, REDCap-hosted consent document outlining a novel open 10-item survey assessing farm family demographics, views on child agricultural safety, and open-ended feedback on the Facebook advertisement they were shown [, ]. This survey was devised in anticipation of a research project assessing the impacts of relevant news media on farm mothers’ knowledge, attitudes, and behavioral intentions toward childhood agricultural safety. No financial incentive was provided for participation. The survey instrument, consent form, and protocol for this study were deemed exempt from review by the Marshfield Clinic Research Institute Institutional Review Board.
We investigated the geographic distribution of advertisements by comparing Facebook’s advertisement targeting system performance to an expected distribution based on the number of female producers at the state level . We then mapped the ratio of expected to observed reach of the advertisements.
Over this 1-week period, the total reach of the advertising campaign (measured by number of views) was 40,024 and 318 users clicked on one of the advertisements, representing a click rate of 0.79% (). Of these 318 visitors, 29 consented to the survey, representing a participation rate of 9.1%, with 24 fully completing the survey for a completion rate of 83%. This yields a completed response rate of 0.06% out of the total reach and 7.5% out of the total clicks. The total cost per click for this study was US $0.92, whereas the cost per participant was US $10.14. With the 17% dropout rate (completed surveys vs consenting participants) and a total personnel cost of US $31 per hour at 4 hours, the total cost per completed survey was US $17.42.
The neutral advertisement depicting a child feeding a calf generated the most clicks with 122 out of 318 total clicks (38.4%). However, the image of a young woman sweeping the barn had the highest click rate at 0.84% ().
Advertisements were posted more often than expected in eastern states, whereas western states (excluding California, Nevada, and Utah) tended to be overrepresented compared to the distribution of female producers (). There was no observed difference in geographical distribution among the three advertising images.
|Advertisement image||Reach (N)||Click rate, n (%)|
|Overturned tractor||14,240||102 (0.72)|
|Child feeding calf||14,652||122 (0.83)|
|Woman sweeping||11,132||94 (0.84)|
This pilot project aimed to determine if Facebook Ads Manager could be a useful tool for recruiting samples of the agricultural population for online health and safety surveys. The current adoption of Facebook and other social media in the United States, as well as increasing rates of internet access, show promise for future research to be performed in part or entirely online.
We reached over 40,000 Facebook users at a cost of US $294 and generated a total cost per completed survey of US $17.42. This is comparable to other observed costs ranging between US $0.64 and $33 for online health and behavioral studies; however, we were unable to identify any published studies that utilized Facebook Ads Manager for recruiting an agricultural population such as farmers, ranchers, farm or ranch parents, or farmworkers specifically [- ]. It should be noted that as advertisements approach the total sampling frame for a study, there is the risk of reaching saturation among the target population, meaning that individuals will be shown the same advertisement multiple times. Further research is needed to determine what diminishing returns, if any, exist when recruiting larger sample sizes and showing a single advertisement multiple times to the same people.
This pilot study had several limitations. First, our study had a remarkably small final response rate of 0.06% based on total reached (N= 40,024) and of 7.5% based on total clicks (N=318). Facebook and other online methods of recruitment allow for high scalability with relatively low cost. However, with this low response rate, it is possible for other, larger samples to reach a level of saturation among the target population, resulting in advertisements being shown multiple times to potential participants.
Additionally, this study exhibited a level of geographic bias that was algorithmically determined by the Facebook Ads Manager platform. Facebook users across multiple states were shown the advertisement a disproportionately high or low amount when compared to the expected target population residing in those states. Although online studies allow for ease in reaching national and international samples, the risk of bias remains prominent, and appropriate measures should be taken to adjust or account for these discrepancies (eg, data weighting, stratification, quota sampling).
We did not offer or disburse any financial incentive for participation. Although we had 318 visits to the survey, only 29 visitors consented to participate and 24 fully completed the survey. We were also unable to accurately measure response rates and completion rates for each advertisement image. Future studies should ensure that the survey infrastructure is set up to measure and assess these potential differences. This campaign was administered via the NCCRAHS Facebook Page, which then appeared as the host organization ads, and may have influenced which respondents engaged with the advertisements, in turn affecting click rates and completion rates. The provision of compensation could help increase the participation and completion rates; however, measures should be taken to control for the additional self-selection bias such measures could introduce.
It should be noted that we do not have full control over, or knowledge of, Facebook’s mechanisms for targeted advertising. The woman sweeping advertisement had lower reach than the other images, as determined by Facebook, but had the highest click rate. Additionally, Facebook reported 170,000 women, aged 21-50 years, who have a role in FFF. The National Agricultural Statistics Service 2017 Census estimated over 500,000 female agricultural producers between the ages of 21 and 50 years in the United States; this does not include farm hands or women whose spouse works in agriculture .
We argue for additional research to make use of these relatively underutilized resources alongside sufficient measures to control for geographic and response biases, as well as misrepresentation and data validity. Future research should include data weighting, verifying consistent responses across similar questions, and “insider knowledge” questions [, ]. More in-depth studies and analyses could offer insight into additional methods for applying these methods to online survey studies and how these methods may differ for agricultural populations.
We were successful in using Facebook Ads Manager to recruit a sample of an agricultural population for an online health and safety survey in a short time frame. We observed low response rates, which were counteracted by a large advertisement reach. This method shares many limitations with traditional survey recruitment methods and requires appropriate measures to limit potential biases. Future research is needed to determine the effectiveness of Facebook Ads Manager on recruiting a large agricultural population and if this method is sustainable.
The authors would like to thank Dr. Florence Becot of the National Farm Medicine Center for her help in reviewing this manuscript. Funding was provided through the National Farm Medicine Center, the Marshfield Clinic Research Institute, and the National Children’s Center for Rural and Agricultural Health and Safety via the CDC/NIOSH (grant number U54OH009568-11-00).
All authors participated in the conception or design of the work; the acquisition, analysis, or interpretation of data for the work; drafting the work and revising it critically for important intellectual content; final approval of the version to be submitted/published; and all agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Conflicts of Interest
Preview of low safety adoption advertisement.PNG File , 709 KB
Preview of neutral safety adoption advertisement.PNG File , 813 KB
Preview of high safety adoption advertisement.PNG File , 914 KB
- Kennedy C, Hartig H. Response rates in telephone surveys have resumed their decline. Pew Research Center. 2020. URL: https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/ [accessed 2020-12-08]
- Stedman RC, Connelly NA, Heberlein TA, Decker DJ, Allred SB. The end of the (research) world as we know it? Understanding and coping with declining response rates to mail surveys. Soc Nat Resource 2019 Mar 29;32(10):1139-1154. [CrossRef]
- Czajka J, Beyler A. Declining response rates in federal surveys: trends and implications. Mathematica Policy Research Reports. 2016 Jun 15. URL: https://aspe.hhs.gov/system/files/pdf/255531/Decliningresponserates.pdf [accessed 2020-12-08]
- Facebook IQ Insights to Go. Facebook for Business. 2020. URL: https://www.facebook.com/iq/insights-to-go/ [accessed 2020-11-18]
- 10% of Americans don't use the internet. Pew Research Center Fact Tank. 2019. URL: https://www.pewresearch.org/fact-tank/2019/04/22/some-americans-dont-use-the-internet-who-are-they/ [accessed 2020-03-23]
- Demographics of internet and home broadband usage in the United States. Pew Research Center Internet & Technology. 2019. URL: https://www.pewresearch.org/internet/fact-sheet/internet-broadband/ [accessed 2020-03-23]
- Sinclair M, O'Toole J, Malawaraarachchi M, Leder K. Comparison of response rates and cost-effectiveness for a community-based survey: postal, internet and telephone modes with generic or personalised recruitment approaches. BMC Med Res Methodol 2012 Aug 31;12:132 [FREE Full text] [CrossRef] [Medline]
- Hardigan PC, Succar CT, Fleisher JM. An analysis of response rate and economic costs between mail and web-based surveys among practicing dentists: a randomized trial. J Community Health 2012 Apr;37(2):383-394. [CrossRef] [Medline]
- Williams D, Edwards W, Giambo P, Kena G. Cost effective mail survey design. 2018 Mar Presented at: Federal Committee on Statistical Methodology Research Conference; March 7-9, 2018; Washington DC URL: https://nces.ed.gov/FCSM/pdf/F4_Williams_2018FCSM.pdf
- Potoglou D. Evidence on the comparison of telephone and internet surveys for respondent recruitment. Open Transport J 2012 Mar 22;6(1):11-22 [FREE Full text] [CrossRef]
- Duncan SC, Strycker LA, Duncan TE, Chaumeton NR. Telephone recruitment of a random stratified youth sample for a physical activity study. J Sport Exerc Psychol 2004 Aug 01;18(3):353-368 [FREE Full text] [CrossRef] [Medline]
- Ramo DE, Prochaska JJ. Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. J Med Internet Res 2012 Feb 23;14(1):e28 [FREE Full text] [CrossRef] [Medline]
- Kosinski M, Matz SC, Gosling SD, Popov V, Stillwell D. Facebook as a research tool for the social sciences: Opportunities, challenges, ethical considerations, and practical guidelines. Am Psychol 2015 Sep;70(6):543-556. [CrossRef] [Medline]
- Greenlaw C, Brown-Welty S. A comparison of web-based and paper-based survey methods: testing assumptions of survey mode and response cost. Eval Rev 2009 Oct;33(5):464-480. [CrossRef] [Medline]
- Lee S, Torok M, Shand F, Chen N, McGillivray L, Burnett A, et al. Performance, cost-effectiveness, and representativeness of Facebook recruitment to suicide prevention research: online survey study. JMIR Ment Health 2020 Oct 22;7(10):e18762 [FREE Full text] [CrossRef] [Medline]
- Ads Management for Facebook. Facebook for Business. 2021. URL: https://www.facebook.com/business/tools/ads-manager [accessed 2020-03-31]
- Weigel C, Paul LA, Ferraro PJ, Messer KD. Challenges in recruiting U.S. farmers for policy‐relevant economic field experiments. Appl Econ Perspect Policy 2020 Sep 28:online ahead of print [FREE Full text] [CrossRef]
- Glas ZE, Getson JM, Gao Y, Singh AS, Eanes FR, Esman LA, et al. Effect of monetary incentives on mail survey response rates for midwestern farmers. Soc Nat Resource 2018 Nov 20;32(2):229-237. [CrossRef]
- Wallander S, Ferraro P, Higgins N. Addressing participant inattention in federal programs: a field experiment with the conservation reserve program. Am J Agric Econ 2017 Apr 19;99(4):914-931. [CrossRef]
- Buse RC. Increasing response rates in mailed questionnaires. Am J Agric Econ 1973 Aug;55(3):503-508. [CrossRef]
- Swenson A, Ploeckelman M, Salzwedel M, Weichelt B. Assessing Facebook advertising as a recruitment strategy for in-person workshops. J Agromed 2020 Dec 06;25(3):261-262. [CrossRef]
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381 [FREE Full text] [CrossRef] [Medline]
- Harris PA, Taylor R, Minor BL, Elliott V, Fernandez M, O'Neal L, REDCap Consortium. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform 2019 Jul;95:103208 [FREE Full text] [CrossRef] [Medline]
- 2017 Census of Agriculture. United States Department of Agriculture National Agricultural Statistics Service. 2017. URL: https://www.nass.usda.gov/AgCensus [accessed 2020-03-27]
- Kramer J, Rubin A, Coster W, Helmuth E, Hermos J, Rosenbloom D, et al. Strategies to address participant misrepresentation for eligibility in Web-based research. Int J Methods Psychiatr Res 2014 Mar;23(1):120-129 [FREE Full text] [CrossRef] [Medline]
- Pedersen ER, Kurz J. Using Facebook for health-related research study recruitment and program delivery. Curr Opin Psychol 2016 May;9:38-43 [FREE Full text] [CrossRef] [Medline]
|FFF: Farming, Fishing, and Forestry|
|NCCRAHS: National Children’s Center for Rural and Agricultural Health and Safety|
Edited by G Eysenbach; submitted 01.04.20; peer-reviewed by MA Bahrami, E Da Silva, I Shubina, P Kanzow; comments to author 18.09.20; revised version received 11.12.20; accepted 17.01.21; published 07.04.21Copyright
©Richard R Burke, Bryan P Weichelt, Kang Namkoong. Originally published in JMIR Formative Research (http://formative.jmir.org), 07.04.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.