JFR JMIR Form Res JMIR Formative Research 2561-326X JMIR Publications Toronto, Canada v8i1e38189 39173153 10.2196/38189 Original Paper Original Paper Recruitment in Appalachian, Rural and Older Adult Populations in an Artificial Intelligence World: Study Using Human-Mediated Follow-Up Mavragani Amaryllis Saunders Elizabeth Holm Ingrid Ben-Zeev Dror Milliken Tabitha MPH 1 https://orcid.org/0000-0002-3311-6013 Beiler Donielle BSc 1 https://orcid.org/0000-0002-4570-631X Hoffman Samantha BA 1 https://orcid.org/0000-0003-2666-7594 Olenginski Ashlee DO 2 https://orcid.org/0009-0008-4412-3172 Troiani Vanessa PhD 1
Research Institute Geisinger 100 N. Academy Ave Danville, PA, 17821 United States 1 215 681 1733 vtroiani@geisinger.edu
https://orcid.org/0000-0001-6689-8157 Research Institute Geisinger Danville, PA United States Philadelphia College of Osteopathic Medicine Philadelphia, PA United States Corresponding Author: Vanessa Troiani vtroiani@geisinger.edu 2024 22 8 2024 8 e38189 30 3 2022 25 7 2022 17 5 2024 15 6 2024 ©Tabitha Milliken, Donielle Beiler, Samantha Hoffman, Ashlee Olenginski, Vanessa Troiani. Originally published in JMIR Formative Research (https://formative.jmir.org), 22.08.2024. 2024

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

 Background

Participant recruitment in rural and hard-to-reach (HTR) populations can present unique challenges. These challenges are further exacerbated by the need for low-cost recruiting, which often leads to use of web-based recruitment methods (eg, email, social media). Despite these challenges, recruitment strategy statistics that support effective enrollment strategies for underserved and HTR populations are underreported. This study highlights how a recruitment strategy that uses email in combination with follow-up, mostly phone calls and email reminders, produced a higher-than-expected enrollment rate that includes a diversity of participants from rural, Appalachian populations in older age brackets and reports recruitment and demographic statistics within a subset of HTR populations.

 Objective

This study aims to provide evidence that a recruitment strategy that uses a combination of email, telephonic, and follow-up recruitment strategies increases recruitment rates in various HTR populations, specifically in rural, older, and Appalachian populations.

 Methods

We evaluated the overall enrollment rate of 1 recruitment arm of a larger study that aims to understand the relationship between genetics and substance use disorders. We evaluated the enrolled population’s characteristics to determine recruitment success of a combined email and follow-up recruitment strategy, and the enrollment rate of HTR populations. These characteristics included (1) enrollment rate before versus after follow-up; (2) zip code and county of enrollee to determine rural or urban and Appalachian status; (3) age to verify recruitment in all eligible age brackets; and (4) sex distribution among age brackets and rural or urban status.

 Results

The email and follow-up arm of the study had a 17.4% enrollment rate. Of the enrolled participants, 76.3% (4602/6030) lived in rural counties and 23.7% (1428/6030) lived in urban counties in Pennsylvania. In addition, of patients enrolled, 98.7% (5956/6030) were from Appalachian counties and 1.3% (76/6030) were from non-Appalachian counties. Patients from rural Appalachia made up 76.2% (4603/6030) of the total rural population. Enrolled patients represented all eligible age brackets from ages 20 to 75 years, with the 60-70 years age bracket having the most enrollees. Females made up 72.5% (4371/6030) of the enrolled population and males made up 27.5% (1659/6030) of the population.

 Conclusions

Results indicate that a web-based recruitment method with participant follow-up, such as a phone call and email follow-up, increases enrollment numbers more than web-based methods alone for rural, Appalachian, and older populations. Adding a humanizing component, such as a live person phone call, may be a key element needed to establish trust and encourage patients from underserved and rural areas to enroll in studies via web-based recruitment methods. Supporting statistics on this recruitment strategy should help researchers identify whether this strategy may be useful in future studies and HTR populations.

 telecommunication enrollment rate Northern Appalachia web-based aging recruitment rural Introduction

The success of human subject research is reliant on effective recruitment strategies. Without successful recruitment, human subject research cannot be accomplished, yet the recruitment process itself can present an obstacle for researchers [1-3]. Common barriers to patient recruitment include mistrust of research, perceived harms, cost, patient location (eg, rural area), transportation, medical illiteracy, lack of research opportunities, time, and family objection [4-7]. Given the difficulties associated with recruitment, one might anticipate an increase in research centered on successful recruitment strategies and retention statistics in specific target populations; especially hard-to-reach (HTR) populations. In research, HTR populations describe subgroups that are difficult to contact due to economical, geographical, physical, or social situations [8].

Recruitment methods commonly include flyers and other printed advertisements (eg, newspapers), but since web-based access has become commonplace, many strategies involve some form of web-based recruitment. For example, emails may be sent that include information about a study that would have typically required posted mail in the past, and posts on social media platforms (eg, Facebook) are now more widely used to reach populations that a researcher is interested in recruiting. While many recruitment strategies are used in human subjects research, data about specific types of recruitment strategies (eg, social media, email, telephone, postal, patient portal, combination) with supporting recruitment statistics and evidence of efficacy are often inconsistent, vary based on target populations, and often limited in HTR populations [1]. Many of the existing review articles regarding enrollment rates cite the need for additional literature that supports efficient recruitment methods in specific populations, especially HTR populations [2,9,10]. Web-based or digital recruitment has become popular, as roughly 6%-50% of studies use web-based methods [11], but web-based methods often fail to reach the diversity of patient populations [12]. One study that aimed to provide support to effective and equitable recruitment methods found that unsolicited recruitment emails had a low 6.1% enrollment rate and telephone calls had 37.8% enrollment [9]. Another study found that older populations and marginalized groups were often underrepresented in digital approaches, thus necessitating further research to understand recruitment barriers and successful recruitment strategies [2,9]. Using a successful recruitment strategy is even more important when targeting marginalized or HTR populations [2,8,9]. HTR populations face additional barriers that can make recruitment arduous, while at the same time, are the very populations whose participation in research is critical to fill gaps in knowledge [4]. Thus, identifying efficient recruitment strategies that result in successful enrollment of HTR and other targeted populations is important.

One specific HTR population includes individuals residing in Appalachia. Appalachia is a 205,000–square mile region that follows the Appalachian Mountains from northern Mississippi to southern New York and is home to more than 25 million people [13,14]. Appalachia is a federally designated region that is characterized by environmental and socioeconomic factors that include low education, low income, limited health literacy, lack of access to health care, and challenging rural geography [7,14]. This region has been impacted by geographical isolation and natural resource exploitation and, as such, the people of Appalachia often have an “insider versus outsider” mentality and are mistrustful of outsiders and the medical community [1,13,14]. This mentality extends into many rural communities and the mistrust of the medical community is common among older adults [5,13]. Patients enrolled into studies from rural areas, such as many parts of Appalachia that are extremely rural and geographically isolated, are often less educated, from lower socioeconomic backgrounds, and lack the medical literacy necessary to clearly understand research [1,6,7,13]. There are numerous infrastructure barriers, including transportation availability, distance, and roads, as well as web-based access, that can also hinder recruitment [1,13]. Older adults, aged 60 years and older, are a common HTR subgroup and share many of the same recruitment barriers as rural populations. Older adults can be more difficult to recruit through digital or web-based methods due to reduced web-based access, attitudes toward technology, and age-related medical issues such as hearing and vision loss [2,15]. As a result, recruitment and effective recruitment strategies of Appalachian populations, as well as rural and older adult populations, can be difficult and are limited in the literature [13].

There are numerous studies about the health disparities and recruitment barriers in the Appalachian population [1,6,16-18]; however, there are limited literature and data specifically about recruitment strategies that demonstrate efficient and successful recruitment into studies that target Appalachian, rural-dwelling, and older adult populations. Studies that target HRT populations often state the need for cultural competency and trustworthiness, which can be accomplished by human-mediated outreach (physical or digital) [2,6,15,16]. Existing studies that did address web-based recruitment typically used social media and email but lacked human-mediated follow-up [9-11,19]. Information about the demographics of enrollees was not always detailed and depended heavily on what the researchers were investigating. Studies that did report information and enrollment rate statistics were widely inconsistent and varied in target population, sample size, and media type [2,11,12]. A recent review by Frampton et al [2] summarized existing papers that used digital tools for the recruitment and retention of participants in randomized controlled trials. Of the 104 articles in the review, only 16 focused on minority or underserved populations [20-35] with 14 deemed HTR populations [22-25,27-29,33,35-40] by the authors of this paper. Only 3 articles included populations in the Appalachian region (Scranton, Pennsylvania [PA], and Pittsburgh, PA) [41-43], but Appalachian populations were not the primary target in any of these studies. The review had 9 studies that focused on rural areas and counties [41,44-51]; however, only 2 of the studies conducted in rural areas were located the United States [41,50], with the other studies occurring in rural and remote parts of the United Kingdom and Australia. Ten of the studies reviewed used human-mediated calls for recruitment, with recruitment rates varying from 0% to 88% in these studies [40,52-59]. It must be noted though, of the 10 studies that used human-mediated calling, human-mediated calls were not the only methods of recruitment. Social media and web advertisements were used in 7 of the 10 studies [40,53-55,58-60]. Studies that used social media had higher overall enrollment rates but enrollment rates using telephone alone ranged between 0% and 17% [47-51,53]. Overall, authors of this review concluded that there is limited literature about efficacy of recruitment strategies, and further research is needed to determine which digital or nondigital tools and combinations would be best suited for specific populations [2]. While HTR populations are being included in research more readily, the primary focus is more directed at underrepresented racial, ethnic, sex, and LGBTQIA+ (lesbian, gay, bisexual, transgender, queer, intersex, and asexual) groups, with less focus on other marginalized groups such Appalachian residents, rural-dwelling residents, and older adults [19-29,40,61-63].

This study's main goal is to add knowledge to the field regarding a recruitment strategy that enrolled participants from Appalachian, rural-dwelling, and older adult populations in a rural PA health care system. The secondary goal is to determine whether adding participant follow-up yielded higher enrollment numbers than without participant follow-up. The novelty of this study is the outreach of this recruitment strategy in Appalachian, rural-dwelling, and older adult populations and the study focus on substance abuse and genetics [64]. In a recent review article [2], only 2 cited studies focused on recruitment of Appalachian populations [41,42] and none on genetics and substance abuse. The Frampton’s review had 13 studies that focused on substance abuse (primarily alcohol and tobacco) [44,45,65-73], and only one of the studies was recruiting from biobank samples [56]. The target enrollment number for the overall study is 20,000 people (enrollment ongoing), and this email recruitment is one of several recruitment arms used in the study [20]. Email recruitment often reaches larger amounts of people than mailings and face-to-face recruitment [2,9,10,13], and follow-up has been shown to increase enrollment in a study that did postal mailings [74]. Thus, the email and follow-up enrollment strategies described here may be helpful to adapt in studies of similar populations of older adults and rural Appalachian residents, as well as other HTR populations.

 Methods Participants

This study was conducted at Geisinger, an integrated health system based in central, south-central, and northeastern PA, that serves about 3.8 million patients, including older adults in rural Appalachia. PA has 67 counties, with 48 designated as rural, 19 designated as urban, and 52 designated as Appalachian. To identify the available recruitment population for this study, any patient with a Geisinger health record was screened for eligibility. Patients were still eligible if they had moved outside of the service area, which included Appalachian and rural counties outside the specified service area. Recruitment data are based on the county of current residence. In 2020, approximately 3.4 million people or 26% of the state’s population lived in rural counties and 64% of rural municipalities in PA had fewer than 2500 residents [75,76]. In addition, PA has 2.7 million (21.3%) people older than 60 years and 1.4 million (11.1%) people older than 70 years as of 2012 [76]. Data used in this manuscript were collected from July 2018 to July 2021 and recruitment is ongoing for an internal review board–approved study, the design of which has been published previously [64]. A US $10 electronic gift card was offered to participants who completed the survey as compensation for their time.

For the analyses described here, we focus on 1 specific recruitment arm of the study, in which recruitment letters were sent to potential participants who met eligibility criteria via a secure email, patient portal message, and paper copy. A waiver of consent was obtained for screening and recruitment, and names, addresses, phone numbers, email addresses, and medical record numbers of patients who met the inclusion criteria were obtained through a Geisinger data broker. The inclusion criteria consist of patients in the Geisinger Health System between the ages of 18 and 75 years and have received at least 2 opioid prescriptions over their lifetime. Participants must be of European ancestry, be English speaking, be enrolled in Geisinger’s biobank [77], and with no history of metastatic cancer.

 Recruitment

Email letter recruitment batches of 1-499 emails were sent from a secure REDCap (Research Electronic Data Capture) platform. REDCap is a secure, web-based software platform designed to support data capture for research studies and is approved to house-protected health information, including patient contact information, which can be used to send automated email invitations to participate in research [78,79]. Following the initial email, a reminder email was sent after 1 week. Any patient who did not complete or did not decline the survey after the automated follow-up email received a follow-up phone call. If a patient answered the telephone and was interested in completing the survey, the recruitment email with the survey link or a secure patient portal message containing the survey link was resent. Patients who expressed interest but did not have access to a computer or the internet were offered the option to do the survey over the phone or receive a paper packet. Paper packets included hard copies of the consent, survey, and a prepaid envelope for participants to return the survey. Patients who were not interested were thanked for their time, recorded as a “decline,” and not recontacted. If a patient did not answer, a brief voicemail message containing study details and contact information was left. If voicemail was not an option, 1 additional reminder email was sent as the third contact attempt. See Textbox 1 for an overview of the recruitment process.

Patient contacts across modalities (email, phone, patient portal, and paper packets) were limited to 3 contacts. Patients who received more than 3 contacts were patients who partially completed the survey, requested a link resend, or requested a paper packet and never returned it. Partial completion patients received an additional reminder email with an access code to the survey and an additional follow-up phone call. Patients who requested a paper packet received a follow-up phone call if they did not return the packet within several months or if they returned an incomplete survey and consent form. If a link resend was requested, these patients received 1 additional follow-up email reminder. Information about the email and follow-up call responses were recorded in REDCap, including patient status and call status (message, enrolled, declined, link resend, etc), date of consent, and any relevant notes about the patient or call.

Eligibility requirements and step-by-step recruitment process for recruitment into the study via email and follow-up.

Eligible patients are sent a recruitment letter via REDCap (Research Electronic Data Capture).

Eligibility requirements: 18-75 years, European descent, >1 opioid prescription, current enrollment in Geisinger biobank, read and write fluent English, no metastatic cancer.

Recruitment letter contains a link to complete a 20- to 30-minute survey.

Reminder email is sent 1-week later.

Research assistant (RA) calls patients who have not completed the survey 1 day after reminder email. Patients receive only 1 follow-up call.

If no answer, RA leaves a ~30-second voicemail that contains the study name and contact information.

If a patient answers, the RA explains the study and asks whether the patient is interested.

Interested patients are sent another email.

Patients who start the survey and stop part way are also sent an additional reminder with an access code so that they may finish.

Patients who are not interested are marked as a “decline” in our database.

Patients who complete the survey are compensated with a US $10 gift card

 Survey

Participants received an email link and were asked to complete surveys via the email link on a personal computer or other digital device, by phone interview, or by requesting a paper packet to be mailed. Questionnaires include the Brief Risk Questionnaire, Fagerstrom Test for Nicotine Dependence, Alcohol Use Disorder Identification Test, Generalized Anxiety Disorder 7-Item, Patient Health Questionnaire-9, and questions derived from the criteria for opioid use disorder diagnosis from the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition]), among others [64]. After the questionnaires are completed, there is no further study-related follow-up or patient contact.

 Enrollment Demographics

The overall enrollment rate and population characteristics were evaluated to determine recruitment success of a combined email and follow-up recruitment strategy and its outreach in Appalachian and older adult populations.

The characteristics evaluated included (1) enrollment rate before versus after research assistant (RA) follow-up; (2) zip code and county of enrollee to determine rural status; (3) age to verify recruitment in all eligible age brackets; and (4) sex distribution in rural versus urban counties and age brackets.

Overall enrollment rate: to determine the overall enrollment rate, the number of completed surveys was divided by the total number of patients contacted for enrollment.

Recruitment strategy: Success of the combined email and follow-up strategy was determined by separating enrollees into 2 groups: those completing the survey before follow-up and those completing the survey after follow-up. Once in these categories, a percentage of enrollment in each category was determined.

Rural versus urban: outreach into rural PA was calculated by cross-referencing zip codes of enrolled patients against a list of all PA zip codes and sorting patients into counties. Enrollees with zip codes in a rural county were classified as “rural” and enrollees with a zip code in an urban county were classified as “urban.” Rural and urban designations were based on population density being either below 291 persons per square mile (rural) or above 291 persons per square mile (urban) [75]. With this information, the number and percentage of patients enrolled in rural and urban PA counties were determined. PA counties were designated as Appalachian and non-Appalachian counties, based on the Appalachian Regional Commission designations [14].

Age: patients were separated into 6 different age brackets, based on age at date of consent. The distribution among the different age brackets allowed us to determine whether our recruitment strategy was reaching older adults.

Sex distribution: enrolled patients were divided by self-reported sex and then further divided by sex and age bracket, sex, age bracket, and rural versus urban distribution. These distributions allow for further examination of underserved populations in each age, sex, and geographic designation.

Enrollment rate before and after follow-up for each recruitment batch in a series of 74 batches was compared to determine whether participant follow-up yielded greater enrollment using a paired t test. Descriptive statistics, separated by age brackets, sex, and geographic region, were used to summarize data across the cohort.

 Ethical Considerations

Ethical approval for the overall study was obtained from the Geisinger internal review board (study number 2017-0190). All participants received a thorough explanation of the study consent form and were given the opportunity to ask questions prior to and during enrollment. All participants were asked to complete a digital consent form during enrollment. Participants are free to withdraw from the study at any time. Participants were reimbursed US $10 for their time and effort, which was paid for using funds from the National Institutes of Health (NIH) grant (R01DA044015) and from a grant from the Pennsylvania Department of Health. A certificate of confidentiality has been obtained from the NIH for this study. Identifiable data from the study will not be shared with participants’ doctors. All patient data are stored on a HIPAA (Health Insurance Portability and Accountability Act)-compliant portal. All study staff interacting with potential or enrolled participants received training on human subjects research [64].

 Results

From July 2018 to July 2021, a total of 6106 participants completed the survey via the email and follow-up recruitment arm of the overall study. Approximately 35,037 patients were contacted, resulting in a 17.4% (6106/35037) enrollment rate. Approximately 37.6% (2297/6106) of the participants enrolled before the follow-up and 62.4% (3809/6106) enrolled after follow-up by an RA, either by phone call or by additional email follow-up or correspondence. In addition, of the 3809 enrollees who had RA follow-up, 87% (3313/3809) had email and telephonic follow-up, 13% (497/3809) enrolled after 1 voicemail and no additional RA contact or follow-up, 3.7% (141/3809) enrolled via paper, and 2.6% (10/3809) participants enrolled over the phone.

The email and follow-up recruitment strategy enrolled patients in 64% (43/67) of PA counties and 66% (32/48) of rural PA counties. The study also enrolled 72 patients in 25 states outside of PA. Non-PA participants were excluded from this assessment, which resulted in the inclusion of 6030 patients for age, sex, rural versus urban, and Appalachian data. Of participants who enrolled from rural PA counties, 76.30% (4603/6030) enrolled from rural PA Appalachian counties. Only 1 patient living in a rural PA county did not live in a rural PA Appalachian county. Appalachia includes some urban areas and 22.47% (1355/5956) of the Appalachian population were from urban PA Appalachian counties. In total, 98.77% (5956/6030) of the enrolled population were from PA Appalachian counties and 1.26% (76/6030) were not. See Table 1 for the breakdown of the enrolled population as Appalachian or non-Appalachian, Table 2 for the demographic breakdown of the enrolled population, and Table S1 in Multimedia Appendix 1 for more fine-grained demographic breakdown of rural versus urban enrollees.

Female enrollment represented 72.49% (4371/6030) of the total inclusion population and male enrollment represented 27.5% (1659/6030) of the total inclusion population. The average age at enrollment was 54.9 (SD 13.3) years, with ages ranging from 20.2 to 75.9 years. The 60-70 years age bracket enrolled the most patients, including male and female participants. Of note, no patients between the ages of 18 and 19 years enrolled via the email and follow-up arm of the overall study. As a result, the 18-19 years age bracket was excluded. Sex was based on patient identification of binary sex, male or female, via the health record. See Table S1 in Multimedia Appendix 1 for a breakdown of age brackets by urban and rural designation.

Characteristics of rural and urban Appalachian and non-Appalachian counties for all participants recruited via email and follow-up method.

Appalachia, n (%) Non-Appalachia, n (%) Total, n (%)
Rural 4601 (76.3) 1 (0.02) 4602 (76.3)a
Urban 1355 (22.5) 73 (1.2) 1428 (23.7)a
Total 5956 (98.7)a 74 (1.3)a 6030 (100)a

aValues in italics indicate that significance testing was not performed on the data.

Demographic characteristics of participants enrolled via email and follow-up recruitment by age and sex.

Age range (years) Female, n (%) % of total population Male, n (%) % of total population Total, n (%)
20-30 191 (4.37) 3.17 23 (1.4) 0.38 214 (3.55)
30-40 718 (16.43) 11.91 132 (8) 2.19 850 (14.10)
40-50 837 (19.15) 13.88 219 (13.1) 3.63 1056 (17.51)
50-60 1039 (23.77) 17.23 379 (22.9) 6.29 1418 (23.52)
60-70 1077 (24.64) 17.86 552 (33.2) 9.15 1629 (27.01)
70-75 509 (11.64) 8.44 354 (21.4) 5.87 863 (14.31)
Total 4371 (100) 72.49 1659 (100) 27.51 6030 (100)
Discussion Principal Results

The primary aim of this analysis was to determine whether an email and follow-up recruitment strategy is reaching HTR populations: namely, rural, Appalachian, and older adult populations. Our secondary aim is to determine whether the follow-up component of our recruitment strategy (live person calling and email correspondence) increased enrollment numbers in these populations. In addition to these aims, a novel component of this study is the in-depth reporting of the strategy used; while many review articles and publications mention the need for this type of study, few exist for this enrollment strategy and the enrolled populations. We report a recruitment strategy that reached the historically marginalized Appalachian population, as well as other rural-dwelling residents and older adults.

Most patients reached using the methods described here resided in PA Appalachian counties and other rural residences and were classified as older adults. Thus, this study indicates that previously reported mistrust that is common in Appalachian and rural populations [1,13,16] and the negative attitudes toward technology found in older adults [4,15] may be overcome using similar recruitment strategies. We also found that more participants enrolled after human-mediated follow-up than with just email alone. We expected the email and follow-up recruitment arm of the study to yield enrollment rates between 6% and 10%, based on conservative estimates from NIH and Institute of Medicine [80]. However, the email and follow-up recruitment strategy yielded a percentage of enrollment nearly double (17.4%) this expectation. There was a significant difference between mean enrollment before RA follow-up with a mean of 30.6 (SD 11.17) participants per batch and mean enrollment after RA follow-up of 50.77 (SD 19.31) participants per batch; conditions (t74=1.99; P<.001). With this information, we rejected the null hypothesis that follow-up does not significantly affect web-based recruitment enrollment. This led us to determine that follow-up to email recruitment, primarily in the form of live person telephonic calls and additional email correspondence, significantly increases participant enrollment in web-based recruiting studies compared with email recruitment alone. Our live person telephonic call and email follow-up strategy may be useful in reaching other rural and marginalized populations via web-based method. It may also be useful for situations where digital recruitment is a necessity, such as during the COVID-19 pandemic. Recruitment using this method continued throughout the entire COVID-19 pandemic and proved to be highly adaptable for recruitment from home [77].

Our study was not designed to understand the specific reasons for the success of these recruitment strategies in HTR populations. However, we speculate that the trust and connection offered by a live person phone call may contribute to the success of this recruitment strategy [3,16,61,81]. Of the participants who enrolled after follow-up, 87% received a phone call and additional email correspondence. We speculate that having a live person call via telephone may validate the legitimacy of the study to potential participants and establishes a point of contact and component of trust. Participants can be reassured that the study is legitimate, not a scam, and have confidence that any questions will be answered by a knowledgeable study team member in a timely manner. This humanizing component and the element of trust may be necessary for recruiting HTR populations via digital or web-based recruiting methods, especially in rural, Appalachian, and older adult populations [1,5,6,8,52,81]. Future qualitative studies should explore the specific reasons for the success of this strategy.

For older adults, rural-dwelling and Appalachian populations, general distrust of health care providers and researchers, fear of being taken advantage of, medical/health/research illiteracy, poverty, transportation, chronic health conditions and technology constraints (cell phones or internet) are common barriers to study enrollment [5,16,35,82]. By following up with a patient via phone and email correspondence instead of a singular email, participants can establish study legitimacy, ask questions to a real person, and have the study explained verbally in understandable language and request accommodations, such as an over the phone or paper survey. These simple but impactful actions may have allowed RAs to bridge the many barriers that often affect HTR populations. It should be noted that the degree of trust generated by the humanizing component of follow-up may be influenced by the cultural competency of researchers [6,13,17]. The RAs contacting patients in this study were both female, spoke clear, fluent English, were both from Appalachian county’s within Geisinger’s primary coverage area, spoke with the region’s dialect, understood local colloquialisms, and spoke to all patients with an air of respect and an altruistic tone. Patients in HTR populations respond better to people with adequate competence of their culture or community [13,17,63]. These qualities have been described previously within models of cultural competence [6].

The success of this recruitment strategy and its associated supporting data and statistics are important to share with the scientific community given their potential use and broader application. We validated that including follow-up to email recruitment, especially in the form of human-mediated telephonic and email correspondence, significantly increased enrollment in web-based recruitment arms and can reach certain Appalachian, rural-dwelling, and older adult populations. This information is extremely valuable for researchers looking to find cost-effective ways for recruitment, while also targeting potential HTR populations and mitigating recruitment concerns related to COVID-19 and other health threats. Our study adds data to the literature that will provide more realistic comparisons for researchers looking to use email recruitment for large-scale recruitment studies, reach underserved or marginalized populations, and determine strategies to optimize web-based enrollment.

 Limitations

The overall population for this study were patients of European descent. The recruitment of an entirely European American sample was due to the sampling requirements for the overarching study [64], which will use population-based genomic analytic methods that have substantially improved power within homogeneous samples at this relatively small sample size. As a result, this study lacks racial and ethnic diversity and would need to be replicated in other types of HTR populations with more diversity to establish efficacy in other groups. In addition, the before and after follow-up data could be further refined to determine whether additional factors led to higher after call enrollment, such as the number of additional email links resent after the call. Patients could receive upward of 3 additional reminders after being contacted if they stated interest, requested a new email link, and partially completed the survey. These factors were not all built into REDCap and the emailing and calling process evolved as the needs of the project evolved. It should also be noted that we did not manipulate when the call occurred, so we cannot gauge whether patients would have still enrolled without a phone call if they had been given more time. As the proposed reasons for this study’s success are speculative, mixed methods studies including qualitative data will be needed to replicate and further support conclusions described here. This recruitment strategy would not work well in populations that had little or no access to telephone service or the internet. While our rural-dwelling, Appalachian, and older adult populations have reduced web-based access relative to more urban populations, a majority still have web-based access. In addition, currently we do not have access to whether each individual phone number is a mobile or landline, or information regarding service plans, or information on web-based access or plans, but future work should explore whether type of phone line impacts recruitment.

The COVID-19 pandemic also struck during the recruitment for this project. This recruitment arm of the project was able to continue recruitment, but recruitment percentages were reduced slightly (17% per batch before COVID-19 vs 15% per batch after COVID-19; see Table S2 in Multimedia Appendix 1). The effect of the pandemic on overall enrollment is difficult to determine. Web-based recruitment and increased time of this population at home may have given this project an advantage but the health and socioeconomic impact of COVID-19 on potential enrollees may have equally disadvantaged the recruitment process. These external factors, as well as a host of other external factors (season, weather, and holidays), make comparing individual batch success challenging.

 Conclusions

Recruitment in HTR populations has been notoriously difficult for researchers and these struggles are complicated by the need for cost-effective recruitment methods. Web-based recruitment of patients through a combination of email and follow-up with live person telephone calls and email correspondence is a safe, cost-effective method to attempt outreach to Appalachian, rural-dwelling, and older adult populations. The success of this web-based recruitment strategy may be due to the trust and validity established through a humanizing follow-up in the form of live person phone calls and email correspondence. Adding follow-up to email recruitment strategies positively impacts enrollment and can increase enrollment numbers in rural, Appalachian, and older adult populations using web-based recruitment methods.

The enrollment demographics and recruitment methods mentioned in this paper may help establish standards for recruitment success in various populations and study designs that attempt to reach Appalachian, rural-dwelling, and older adult populations. Reaching target populations and enrolling and retaining adequate sample sizes are imperative to the success of all studies. We further suggest that reporting participant demographics of enrollees and recruitment strategy statistics as a standard in future papers will fill knowledge gaps and assist future studies with recruitment goals.

Supplement with additional figure and tables depicting the geographical distribution of survey enrollment in Pennsylvania and detailed demographic breakdown of rural versus urban and Appalachian versus non-Appalachian for gender and age brackets.

 Abbreviations DSM-5

Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition)

 HIPAA

Health Insurance Portability and Accountability Act

 HTR

hard-to-reach

 LGBTQIA+

lesbian, gay, bisexual, transgender, queer, intersex, and asexual

 NIH

National Institutes of Health

 PA

Pennsylvania

 RA

research assistant

 REDCap

Research Electronic Data Capture

Data for this study and salary support of all authors were supported by NIDA [R01DA044015; PI: Troiani] and a grant from the Pennsylvania Department of Health.

 Data Availability

A deidentified summary file with patient enrollment counts can be made available upon reasonable request to the senior author.

 Disclaimer

The work is solely the responsibility of the authors and does not necessarily reflect the official views of the National Institutes of Health. The funding source did not play a role in determining study design, data collection, analysis, or interpretation, authoring the report, or the decision to submit the report for publication. Generative artificial intelligence was not used in any aspect of this research or manuscript writing or editing.

TM contributed to data collection, analysis, interpretation, and wrote the manuscript. DB contributed to data collection, analysis, and edited the manuscript. SH contributed to the data collection and edited the manuscript. AO contributed to literature review and edited the manuscript. VT led the overall study, contributed to data analysis and interpretation, and wrote the manuscript. All authors contributed to the research design and approved the final manuscript.

None declared.

 Cudney S Craig C Michols E Weinert C Barriers to recruiting an adequate sample in rural nursing research Online J Rural Nurs Health Care 2004 4 2 78 88 10.14574/ojrnhc.v4i2.201 Frampton GK Shepherd J Pickett K Griffiths G Wyatt JC Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map Trials 2020 21 1 478 10.1186/s13063-020-04358-3 32498690 10.1186/s13063-020-04358-3 PMC7273688 Chhatre S Jefferson A Cook R Meeker CR Kim JH Hartz KM Wong YN Caruso A Newman DK Morales KH Jayadevappa R Patient-centered recruitment and retention for a randomized controlled study Trials 2018 19 1 205 10.1186/s13063-018-2578-7 29587805 10.1186/s13063-018-2578-7 PMC5870194 Bonevski B Randell M Paul C Chapman K Twyman L Bryant J Brozek I Hughes C Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups BMC Med Res Methodol 2014 14 42 10.1186/1471-2288-14-42 24669751 1471-2288-14-42 PMC3974746 Dibartolo MC McCrone S Recruitment of rural community-dwelling older adults: barriers, challenges, and strategies Aging Ment Health 2003 7 2 75 82 10.1080/1360786031000072295 12745386 124EGGE1DM9LUGRP Carpenter R Theeke LA Strategies for recruiting a sample of adults with type 2 diabetes from primary care clinics in rural appalachia: incorporating cultural competence Int J Nurs Sci 2018 5 3 230 237 10.1016/j.ijnss.2018.06.003 31406830 S2352-0132(17)30363-0 PMC6626207 Ford JG Howerton MW Lai GY Gary TL Bolen S Gibbons MC Tilburt J Baffi C Tanpitukpongse TP Wilson RF Powe NR Bass EB Barriers to recruiting underrepresented populations to cancer clinical trials: a systematic review Cancer 2008 112 2 228 242 10.1002/cncr.23157 18008363 Shaghaghi A Bhopal RS Sheikh A Approaches to recruiting 'Hard-To-Reach' populations into re-search: a review of the literature Health Promot Perspect 2011 1 2 86 94 10.5681/hpp.2011.009 24688904 PMC3963617 Heerman WJ Jackson N Roumie CL Harris PA Rosenbloom ST Pulley J Wilkins CH Williams NA Crenshaw D Leak C Scherdin J Muñoz D Bachmann J Rothman RL Kripalani S Recruitment methods for survey research: findings from the mid-south clinical data research network Contemp Clin Trials 2017 62 50 55 10.1016/j.cct.2017.08.006 28823925 S1551-7144(17)30202-1 Blatch-Jones A Nuttall J Bull A Worswick L Mullee M Peveler R Falk S Tape N Hinks J Lane AJ Wyatt JC Griffiths G Using digital tools in the recruitment and retention in randomised controlled trials: survey of UK clinical trial units and a qualitative study Trials 2020 21 1 304 10.1186/s13063-020-04234-0 32245506 10.1186/s13063-020-04234-0 PMC7118862 Blumenberg C Menezes AMB Gonçalves H Assunção MCF Wehrmeister FC Barros AJD How different online recruitment methods impact on recruitment rates for the web-based coortesnaweb project: a randomised trial BMC Med Res Methodol 2019 19 1 127 10.1186/s12874-019-0767-z 31217008 10.1186/s12874-019-0767-z PMC6585038 Byaruhanga J Tzelepis F Paul C Wiggers J Byrnes E Lecathelinais C Cost per participant recruited from rural and remote areas into a smoking cessation trial via online or traditional strategies: observational study J Med Internet Res 2019 21 11 e14911 10.2196/14911 31714253 v21i11e14911 PMC6880230 Kim NH Wilson N Mashburn T Reist L Westrick SC Look K Kennelty K Carpenter D Lessons learned recruiting a diverse sample of rural study participants during the COVID-19 pandemic Int J Drug Policy 2021 97 103344 10.1016/j.drugpo.2021.103344 34186474 S0955-3959(21)00249-8 PMC8556070 Appalachian Regional Commission Pennsylvania 2017 2021-01-01 https://www.arc.gov/pennsylvania/ Gitlow L Technology use by older adults and barriers to using technology Phys Occup Ther Geriatr 2014 32 3 271 280 10.3109/02703181.2014.946640 Bergeron CD Foster C Friedman D Tanner A Kim SH Clinical trial recruitment in rural South Carolina: a comparison of investigators' perceptions and potential participant eligibility Rural Remote Health 2013 13 4 2567 24325179 2567 MacAvoy S Lippman DT Teaching culturally competent care: nursing students experience rural Appalachia J Transcult Nurs 2001 12 3 221 227 10.1177/104365960101200306 11989037 Denham SA Does a culture of Appalachia truly exist? J Transcult Nurs 2016 27 2 94 102 10.1177/1043659615579712 25824239 1043659615579712 Lane TS Armin J Gordon JS Online recruitment methods for web-based and mobile health studies: a review of the literature J Med Internet Res 2015 17 7 e183 10.2196/jmir.4359 26202991 v17i7e183 PMC4527014 Coronado GD Ondelacy S Schwarz Y Duggan C Lampe JW Neuhouser ML Recruiting underrepresented groups into the carbohydrates and related biomarkers (CARB) cancer prevention feeding study Contemp Clin Trials 2012 33 4 641 646 10.1016/j.cct.2012.03.017 22504222 S1551-7144(12)00093-6 PMC3516589 Bickmore TW Utami D Matsuyama R Paasche-Orlow MK Improving access to online health information with conversational agents: a randomized controlled experiment J Med Internet Res 2016 18 1 e1 10.2196/jmir.5239 26728964 v18i1e1 PMC4717285 Brodar KE Hall MG Butler EN Parada H Stein-Seroussi A Hanley S Brewer NT Recruiting diverse smokers: enrollment yields and cost Int J Environ Res Public Health 2016 13 12 1251 10.3390/ijerph13121251 27999280 ijerph13121251 PMC5201392 Erickson LC Ritchie JB Javors JM Golomb BA Recruiting a special sample with sparse resources: lessons from a study of Gulf War veterans Clin Trials 2013 10 3 473 482 10.1177/1740774512470040 23325028 1740774512470040 Iribarren SJ Ghazzawi A Sheinfil AZ Frasca T Brown W Lopez-Rios J Rael CT Balán IC Crespo R Dolezal C Giguere R Carballo-Diéguez A Mixed-method evaluation of social media-based tools and traditional strategies to recruit high-risk and hard-to-reach populations into an HIV prevention intervention study AIDS Behav 2018 22 1 347 357 10.1007/s10461-017-1956-6 29124420 10.1007/s10461-017-1956-6 PMC5760436 Jones R Lacroix LJ Porcher E Facebook advertising to recruit young, urban women into an HIV prevention clinical trial AIDS Behav 2017 21 11 3141 3153 10.1007/s10461-017-1797-3 28528463 10.1007/s10461-017-1797-3 PMC5669997 Kennedy BM Kumanyika S Ard JD Reams P Johnson CA Karanja N Charleston JB Appel LJ Maurice V Harsha DW Overall and minority-focused recruitment strategies in the PREMIER multicenter trial of lifestyle interventions for blood pressure control Contemp Clin Trials 2010 31 1 49 54 10.1016/j.cct.2009.10.002 19879377 S1551-7144(09)00173-6 PMC2818364 Kim R Hickman N Gali K Orozco N Prochaska JJ Maximizing retention with high risk participants in a clinical trial Am J Health Promot 2014 28 4 268 274 10.4278/ajhp.120720-QUAN-355 23875989 PMC5110435 Ramsey TM Snyder JK Lovato LC Roumie CL Glasser SP Cosgrove NM Olney CM Tang RH Johnson KC Still CH Gren LH Childs JC Crago OL Summerson JH Walsh SM Perdue LH Bankowski DM Goff DC SPRINT Study Research Group Recruitment strategies and challenges in a large intervention trial: systolic blood pressure intervention trial Clin Trials 2016 13 3 319 330 10.1177/1740774516631735 26911833 1740774516631735 PMC4965303 Rellis L Haidari G Ridgers H Jones CB Miller A Shattock R Mccormack S Brodnicki E Banbury S How the use of social media and online platforms can enhance recruitment to HIV clinical trials HIV Med 2015 16 64 65 Sharp LK Fitzgibbon ML Schiffer L Recruitment of obese black women into a physical activity and nutrition intervention trial J Phys Act Health 2008 5 6 870 881 10.1123/jpah.5.6.870 19164821 Staffileno BA Zschunke J Weber M Gross LE Fogg L Tangney CC The feasibility of using Facebook, craigslist, and other online strategies to recruit young African American women for a web-based healthy lifestyle behavior change intervention J Cardiovasc Nurs 2017 32 4 365 371 10.1097/JCN.0000000000000360 27428356 Tate DF LaRose JG Griffin LP Erickson KE Robichaud EF Perdue L Espeland MA Wing RR Recruitment of young adults into a randomized controlled trial of weight gain prevention: message development, methods, and cost Trials 2014 15 326 10.1186/1745-6215-15-326 25128185 1745-6215-15-326 PMC4150977 Ünlü Ince B Cuijpers P van 't Hof E Riper H Reaching and recruiting Turkish migrants for a clinical trial through Facebook: a process evaluation Internet Interv 2014 1 2 74 83 10.1016/j.invent.2014.05.003 Usadi RS Diamond MP Legro RS Schlaff WD Hansen KR Casson P Christman G Wright Bates G Baker V Seungdamrong A Rosen MP Lucidi S Thomas T Huang H Santoro N Eisenberg E Zhang H Alvero R Reproductive Medicine Network Recruitment strategies in two reproductive medicine network infertility trials Contemp Clin Trials 2015 45 Pt B 196 200 10.1016/j.cct.2015.09.010 26386293 S1551-7144(15)30086-0 PMC4766588 Volkova E Michie J Corrigan C Sundborn G Eyles H Jiang Y Mhurchu CN Effectiveness of recruitment to a smartphone-delivered nutrition intervention in New Zealand: analysis of a randomised controlled trial BMJ Open 2017 7 6 e016198 10.1136/bmjopen-2017-016198 28674144 bmjopen-2017-016198 PMC5734449 Buckingham L Becher J Voytek CD Fiore D Dunbar D Davis-Vogel A Metzger DS Frank I Going social: success in online recruitment of men who have sex with men for prevention HIV vaccine research Vaccine 2017 35 27 3498 3505 10.1016/j.vaccine.2017.05.002 28526330 S0264-410X(17)30587-X PMC5922263 Burrell ER Pines HA Robbie E Coleman L Murphy RD Hess KL Anton P Gorbach PM Use of the location-based social networking application GRINDR as a recruitment tool in rectal microbicide development research AIDS Behav 2012 16 7 1816 1820 10.1007/s10461-012-0277-z 22851153 PMC3692292 Horvath KJ Nygaard K Danilenko GP Goknur S Oakes JM Rosser BRS Strategies to retain participants in a long-term HIV prevention randomized controlled trial: lessons from the MINTS-II study AIDS Behav 2012 16 2 469 479 10.1007/s10461-011-9957-3 21538084 PMC3423320 Taylor-Piliae RE Boros D Coull BM Strategies to improve recruitment and retention of older stroke survivors to a randomized clinical exercise trial J Stroke Cerebrovasc Dis 2014 23 3 462 468 10.1016/j.jstrokecerebrovasdis.2013.03.031 23643477 S1052-3057(13)00113-4 Miller EG Nowson CA Dunstan DW Kerr DA Solah V Menzies D Daly RM Recruitment of older adults with type 2 diabetes into a community-based exercise and nutrition randomised controlled trial Trials 2016 17 1 467 10.1186/s13063-016-1589-5 27669823 10.1186/s13063-016-1589-5 PMC5037626 Rollman BL Fischer GS Zhu F Belnap BH Comparison of electronic physician prompts versus waitroom case-finding on clinical trial enrollment J Gen Intern Med 2008 23 4 447 450 10.1007/s11606-007-0449-0 18373143 PMC2359503 Raviotta JM Nowalk MP Lin CJ Huang H Zimmerman RK Using Facebook™ to recruit college-age men for a human papillomavirus vaccine trial Am J Mens Health 2016 10 2 110 119 10.1177/1557988314557563 25389213 1557988314557563 PMC4427547 Lesher LL Matyas RA Sjaarda LA Newman SL Silver RM Galai N Hovey KM Wactawski-Wende J Emerick L Lynch AM Mead B Townsend JM Perkins NJ Mumford SL Stanford J Schisterman EF Recruitment for longitudinal, randomised pregnancy trials initiated preconception: lessons from the effects of aspirin in gestation and reproduction trial Paediatr Perinat Epidemiol 2015 29 2 162 167 10.1111/ppe.12177 25682951 PMC6194508 Frandsen M Walters J Ferguson SG Exploring the viability of using online social media advertising as a recruitment method for smoking cessation clinical trials Nicotine Tob Res 2014 16 2 247 251 10.1093/ntr/ntt157 24127266 ntt157 Frandsen M Thow M Ferguson SG The effectiveness of social media (Facebook) compared with more traditional advertising methods for recruiting eligible participants to health research studies: a randomized, controlled clinical trial JMIR Res Protoc 2016 5 3 e161 10.2196/resprot.5747 27511829 v5i3e161 PMC4997003 Caperchione CM Duncan MJ Rosenkranz RR Vandelanotte C Van Itallie AK Savage TN Hooker C Maeder AJ Mummery WK Kolt GS Recruitment, screening, and baseline participant characteristics in the WALK 2.0 study: a randomized controlled trial using web 2.0 applications to promote physical activity Contemp Clin Trials Commun 2015 2 25 33 10.1016/j.conctc.2015.12.004 29736444 S2451-8654(15)30040-5 PMC5935835 McGregor J Brooks C Chalasani P Chukwuma J Hutchings H Lyons RA Lloyd K The Health Informatics Trial Enhancement Project (HITE): using routinely collected primary care data to identify potential participants for a depression trial Trials 2010 11 39 10.1186/1745-6215-11-39 20398303 1745-6215-11-39 PMC2864261 Nash EL Gilroy D Srikusalanukul W Abhayaratna WP Stanton T Mitchell G Stowasser M Sharman JE Facebook advertising for participant recruitment into a blood pressure clinical trial J Hypertens 2017 35 12 2527 2531 10.1097/HJH.0000000000001477 28704263 Sanders KM Stuart AL Merriman EN Read ML Kotowicz MA Young D Taylor R Blair-Holt I Mander AG Nicholson GC Trials and tribulations of recruiting 2,000 older women onto a clinical trial investigating falls and fractures: vital D study BMC Med Res Methodol 2009 9 78 10.1186/1471-2288-9-78 19930724 1471-2288-9-78 PMC2790455 Switzer JA Hall CE Close B Nichols FT Gross H Bruno A Hess DC A telestroke network enhances recruitment into acute stroke clinical trials Stroke 2010 41 3 566 569 10.1161/STROKEAHA.109.566844 20056929 STROKEAHA.109.566844 Treweek S Pearson E Smith N Neville R Sargeant P Boswell B Sullivan F Desktop software to identify patients eligible for recruitment into a clinical trial: using SARMA to recruit to the ROAD feasibility trial Inform Prim Care 2010 18 1 51 58 10.14236/jhi.v18i1.753 20429978 753 Schroy PC Glick JT Robinson P Lydotes MA Heeren TC Prout M Davidson P Wong JB A cost-effectiveness analysis of subject recruitment strategies in the HIPAA era: results from a colorectal cancer screening adherence trial Clin Trials 2009 6 6 597 609 10.1177/1740774509346703 19933718 1740774509346703 Bailey JV Pavlou M Copas A McCarthy O Carswell K Rait G Hart G Nazareth I Free C French R Murray E The sexunzipped trial: optimizing the design of online randomized controlled trials J Med Internet Res 2013 15 12 e278 10.2196/jmir.2668 24334216 v15i12e278 PMC3868980 Bull S Pratte K Whitesell N Rietmeijer C McFarlane M Effects of an internet-based intervention for HIV prevention: the youthnet trials AIDS Behav 2009 13 3 474 487 10.1007/s10461-008-9487-9 19037719 PMC2844766 Mohan Y Cornejo M Sidell M Smith J Young DR Re-recruiting young adult women into a second follow-up study Contemp Clin Trials Commun 2017 5 160 167 10.1016/j.conctc.2017.02.006 29740632 S2451-8654(16)30086-2 PMC5936695 Rorie DA Flynn RWV Mackenzie IS MacDonald TM Rogers A The Treatment in Morning Versus Evening (TIME) study: analysis of recruitment, follow-up and retention rates post-recruitment Trials 2017 18 1 557 10.1186/s13063-017-2318-4 29169373 10.1186/s13063-017-2318-4 PMC5701451 Ryan C Dadabhoy H Baranowski T Participant outcomes from methods of recruitment for videogame research Games Health J 2018 7 1 16 23 10.1089/g4h.2017.0049 29394108 PMC5797327 Shere M Zhao XY Koren G The role of social media in recruiting for clinical trials in pregnancy PLoS One 2014 9 3 e92744 10.1371/journal.pone.0092744 24671210 PONE-D-12-38063 PMC3966825 Stanton AL Morra ME Diefenbach MA Miller SM Slevin Perocchia R Raich PC Fleisher L Wen K Tran ZV Mohamed NE George R Bright MA Marcus AC Responding to a significant recruitment challenge within three nationwide psychoeducational trials for cancer patients J Cancer Surviv 2013 7 3 392 403 10.1007/s11764-013-0282-x 23595235 PMC3737366 Rabin C Horowitz S Marcus B Recruiting young adult cancer survivors for behavioral research J Clin Psychol Med Settings 2013 20 1 33 36 10.1007/s10880-012-9317-0 22810954 PMC3541434 Remillard ML Mazor KM Cutrona SL Gurwitz JH Tjia J Systematic review of the use of online questionnaires of older adults J Am Geriatr Soc 2014 62 4 696 705 10.1111/jgs.12747 24635138 PMC4098903 Savard I Kilpatrick K Tailoring research recruitment strategies to survey harder-to-reach populations: a discussion paper J Adv Nurs 2022 78 4 968 978 10.1111/jan.15156 35084799 Winter SS Page-Reeves JM Page KA Haozous E Solares A Nicole Cordova C Larson RS Inclusion of special populations in clinical research: important considerations and guidelines J Clin Transl Res 2018 4 1 56 69 30873495 PMC6410628 Troiani V Crist RC Doyle GA Ferraro TN Beiler D Ranck S McBryan K Jarvis MA Barbour JS Han JJ Ness RJ Berrettini WH Robishaw JD Genetics and prescription opioid use (GaPO): study design for consenting a cohort from an existing biobank to identify clinical and genetic factors influencing prescription opioid use and abuse BMC Med Genomics 2021 14 1 253 10.1186/s12920-021-01100-z 34702274 10.1186/s12920-021-01100-z PMC8547564 Buller DB Meenan R Severson H Halperin A Edwards E Magnusson B Comparison of 4 recruiting strategies in a smoking cessation trial Am J Health Behav 2012 36 5 577 588 10.5993/AJHB.36.5.1 22584086 PMC3357184 Coday M Richey P Thomas F Tran QT Terrell SB Tylavsky F Miro D Caufield M Johnson KC The recruitment experience of a randomized clinical trial to aid young adult smokers to stop smoking without weight gain with interactive technology Contemp Clin Trials Commun 2016 2 61 68 10.1016/j.conctc.2015.12.010 26949747 PMC4772746 Free C Hoile E Robertson S Knight R Three controlled trials of interventions to increase recruitment to a randomized controlled trial of mobile phone based smoking cessation support Clin Trials 2010 7 3 265 273 10.1177/1740774510367687 20484492 7/3/265 Gioia CJ Sobell LC Sobell MB Agrawal S Craigslist versus print newspaper advertising for recruiting research participants for alcohol studies: cost and participant characteristics Addict Behav 2016 54 24 32 10.1016/j.addbeh.2015.11.008 26675247 S0306-4603(15)30059-9 PMC4713235 Hamilton FL Hornby J Sheringham J Linke S Ashton C Moore K Stevenson F Murray E DIAMOND (DIgital Alcohol Management ON Demand): a feasibility RCT and embedded process evaluation of a digital health intervention to reduce hazardous and harmful alcohol use recruiting in hospital emergency departments and online Pilot Feasibility Stud 2018 4 1 114 10.1186/s40814-018-0303-7 29946479 303 PMC6003139 Heffner JL Wyszynski CM Comstock B Mercer LD Bricker J Overcoming recruitment challenges of web-based interventions for tobacco use: the case of web-based acceptance and commitment therapy for smoking cessation Addict Behav 2013 38 10 2473 2476 10.1016/j.addbeh.2013.05.004 23770645 S0306-4603(13)00134-2 PMC3725211 Severi E Free C Knight R Robertson S Edwards P Hoile E Two controlled trials to increase participant retention in a randomized controlled trial of mobile phone-based smoking cessation support in the United Kingdom Clin Trials 2011 8 5 654 660 10.1177/1740774511416524 21933834 1740774511416524 PMC3573670 Stanczyk NE Bolman C Smit ES Candel MJJM Muris JWM de Vries H How to encourage smokers to participate in web-based computer-tailored smoking cessation programs: a comparison of different recruitment strategies Health Educ Res 2014 29 1 23 40 10.1093/her/cyt104 24287402 cyt104 Clark L Ronaldson S Dyson L Hewitt C Torgerson D Adamson J Electronic prompts significantly increase response rates to postal questionnaires: a randomized trial within a randomized trial and meta-analysis J Clin Epidemiol 2015 68 12 1446 1450 10.1016/j.jclinepi.2015.01.016 25709076 S0895-4356(15)00024-4 Nakash RA Hutton JL Jørstad-Stein EC Gates S Lamb SE Maximising response to postal questionnaires—a systematic review of randomised trials in health research BMC Med Res Methodol 2006 6 1 5 10.1186/1471-2288-6-5 16504090 1471-2288-6-5 PMC1421421 Center for Rural Pennsylvania Rural urban definitions 2014 2021-01-01 https://www.rural.palegislature.us/demographics_rural_urban.html United States Census Bureau Census.gov 2019 2021-01-01 https://www.census.gov/ Carey DJ Fetterolf SN Davis FD Faucett WA Kirchner HL Mirshahi U Murray MF Smelser DT Gerhard GS Ledbetter DH The geisinger MyCode community health initiative: an electronic health record-linked biobank for precision medicine research Genet Med 2016 18 9 906 913 10.1038/gim.2015.187 26866580 S1098-3600(21)04435-X PMC4981567 Harris PA Taylor R Thielke R Payne J Gonzalez N Conde JG Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support J Biomed Inform 2009 42 2 377 381 10.1016/j.jbi.2008.08.010 18929686 S1532-0464(08)00122-6 PMC2700030 Harris PA Taylor R Minor BL Elliott V Fernandez M O'Neal L McLeod L Delacqua G Delacqua F Kirby J Duda SN REDCap Consortium The REDCap consortium: building an international community of software platform partners J Biomed Inform 2019 95 103208 10.1016/j.jbi.2019.103208 31078660 S1532-0464(19)30126-1 PMC7254481 Harper BD Projecting realistic enrollment rates principles and methodology Monitor 2004 5 15 18 Weng C Bigger JT Busacca L Wilcox A Getaneh A Comparing the effectiveness of a clinical registry and a clinical data warehouse for supporting clinical trial recruitment: a case study AMIA Annu Symp Proc 2010 2010 867 871 21347102 PMC3041383 Jonnalagadda SR Adupa AK Garg RP Corona-Cox J Shah SJ Text mining of the electronic health record: an information extraction approach for automated identification and subphenotyping of HFpEF patients for clinical trials J Cardiovasc Transl Res 2017 10 3 313 321 10.1007/s12265-017-9752-2 28585184 10.1007/s12265-017-9752-2