Obesity is a pressing health concern nationwide, particularly in small rural communities. Rural residents are more obese, on average, than their urban counterparts [1] and often have no or limited access to obesity prevention and treatment programming [1,2]. Furthermore, individuals who use primary care are proportionally more obese than the general public [3], highlighting rural primary care patients as a high-need population regarding weight management. Primary care providers are often the only resource to support healthful eating, physical activity, and weight management in rural communities [4]. Primary care systems may offer a practical and sustainable venue for implementing evidence-based weight management interventions; however, little is known about how rural primary care physicians can pragmatically refer and enroll a large and representative group of individuals into an evidence-based weight management program.

A challenge for weight management interventions is ensuring that not only is an intervention effective but it also has the potential to reach populations at risk that could most benefit [5]. Factors beyond program effectiveness and total sample size, such as proportional yield from those recruited and sample representativeness, are important indicators of an intervention’s impact [6]. Few studies provide a comprehensive report on the methods used to recruit participants, and even fewer report on the representativeness of the sample when compared with the target population [7,8]. In a systematic review of rural weight loss interventions, only 2 of 53 studies compared the demographic characteristics of the intervention sample with those of the target population [8]. In a 2016 systematic review of recruitment strategies for young adult weight gain prevention interventions, 23 of 25 studies were reported to have insufficiently described the recruitment process [7]. Documenting recruitment methods and the representativeness outcomes of those efforts can lead to a better overall understanding of how best to engage patients and maximize the reach of evidence-based behavioral interventions. This understanding is important for scale-up efforts, for preventing underrepresentation of populations experiencing disparities in research and in health care, and for physicians to improve their standard of care [9].

However, when drawing more broadly from the behavioral intervention literature, a number of examples of both active and passive recruitment strategies can be found that have been used to engage various target populations [7,10-13]. On the basis of the available literature, active recruitment strategies—those with direct interaction with potential participants, such as outreach telephone calls—appear to yield a lower absolute number of participants but a higher proportion of those exposed to recruitment strategies when compared with passive recruitment strategies (ie, those without direct interaction with potential participants, such as flyers or targeted mailings), which yield a higher number but a lower proportion of participants [5]. Active recruitment strategies may also yield a more representative sample than passive recruitment strategies [12,14,15]. The limited knowledge in this area warrants further study.

The purpose of this study is to examine the utility and cost of different physician referral and engagement processes for improving reach (ie, number, proportion, and representativeness of participants) for a rural, evidence-based, and technology-delivered weight management program with counseling support. Our design is based on feedback gathered from prior qualitative work [16,17]; that is, an expressed clinical interest in testing the relative reach of engaging patients at the point of care (POC) during a well or chronic care visit or proactively reaching out to patients using a population health management approach facilitated by an electronic health record.

We use the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to guide our assessment of reach [18] and hypothesize that active recruitment strategies would yield a more representative sample than passive recruitment strategies. Reach refers to the absolute number, proportion, and representativeness of the participants compared with those who were exposed to recruitment efforts [18]. We also hypothesize that POC referrals, compared with the population health registry–derived letter referrals, would yield a higher proportion of enrolled participants from those who received a referral.

Before this pilot trial, we conducted focus groups with primary care staff employed at rural primary care clinics regarding the feasibility of implementing a weight loss program through primary care. Overwhelmingly, primary care staff agreed that a program to which physicians and nurses could refer eligible patients and track their progress throughout was more favorable than a program that would require physician- or nurse-led delivery [16,17]. We then assembled primary care physicians, staff, and obesity treatment experts from the Great Plains Practice-Based Research Network to engage in a participatory selection process [19] to identify and, if necessary, adapt an evidence-based intervention to pilot-test through a rural primary care clinic. The selection process is described elsewhere [17]. Ultimately, a digitally delivered intervention was selected by the group for local testing, and it was agreed that testing potential recruitment strategies was a priority over testing the relative effectiveness of the different weight loss programs.

All referred and interested patients were required to undergo a brief telephone screening with a member of the research team and, if eligible, were given instructions over the phone and via email on how to complete web-based program enrollment. In addition, all interested and eligible patients were scheduled for an in-person enrollment visit with a member of the research team where they were given intervention program materials (ie, home scale) and a tutorial of the intervention’s mobile app, provided with tips on how to stay engaged with the program, and completed the web-based program enrollment (if not already complete). Patient consent to participate in the weight loss program was obtained during the web-based enrollment process. Patients were given the opportunity to raise questions before consent and during the in-person enrollment visit.

All referred patients were offered a 12-month, digitally delivered, evidence-based weight management program free of charge. The program featured a social cognitive theory–based curriculum with counseling support delivered through daily emails and text messages. Program features also included daily meal plans and physical activity recommendations [23]. Estabrooks et al [23] provided a comprehensive overview of the program. In addition, modest financial incentives were offered with the intent to increase program reach and retention [24]. Incentives were offered to participants who lost a minimum of 5% of their initial body weight (US $15/quarter reward) graded up to a maximum of 30% body weight reduction (US $150/quarter reward). Program participants were provided a Bluetooth-enabled home scale (Smart Scale, incentaHEALTH, LLC), which connected to the program smartphone app that was installed on their smartphones during the enrollment visit. Participants were instructed to record their weight using this scale no less than once per quarter. The program also featured a website where participants could receive feedback on their weigh-ins, take health quizzes, and self-assess their progress with regard to healthy eating, physical activity, and weight loss. Upon completion of the program, participants could keep the home scale. This paper focuses on reach, which is the primary outcome of this study; data on weight loss are not presented here.

Program reach was measured by the number and proportion of individuals who were (1) referred, (2) screened, and (3) enrolled and was compared across referral strategies relative to the number of eligible patients who visited the clinic during the recruitment period. Representativeness of the enrolled sample was assessed relative to the demographic characteristics of the region, as measured by the US Census [25]. We used chi-square tests to examine group differences in terms of screening and participation rates among referral strategies and to determine whether screening and participation rates differed according to (1) ATF versus no ATF and (2) POC versus letter. We applied a one-sample test of proportion (in the case of comparing proportions) to examine representativeness in terms of demographic characteristics of the enrolled participants compared with census data. We further conducted group comparisons using one-way analysis of variance tests for continuous variables and chi-square tests for categorical variables. Two-tailed P values <.05 were considered statistically significant for this study.

Costs of recruitment were prospectively and retrospectively estimated based on the costs of recruitment materials (eg, handouts or flyers), supplies, and recruitment activities, including telephone follow-up, telephone screening, and in-person sessions. Labor costs were calculated using the research assistant (RA) annual salary (US $25,000) and publicly available average salary estimates for primary care physicians (US $187,013) and clinic managers (US $65,356). RAs tracked the time spent on various recruitment activities, number of phone calls made, enrollment sessions completed, and number of flyers printed in a custom computer database. Cost results are presented for each referral strategy.

Over a period of 16 weeks, 2534 eligible patients visited the clinic. The maximum number of referrals that could have been made during well or chronic care visits over this time were approximately 30% of patient visits (n=991; hereafter referred to as potential referrals). The actual number of referrals made by the 5 physicians was 573 (274, 47.8% women; average age 55.7, SD 16.8 years) and, out of the 573 referrals, 98 (17.1%) patients were enrolled, representing an overall enrollment rate of 10% of the potentially eligible patient population. Of the 485 potential letter referrals, 485 (100%; 46% women; average age 56.3 years) were completed, 229 (47.2%; 48% women; average age 56.1 years) patients were screened by telephone, and 57 (11.8%; 58% women; average age 55.8 years) were enrolled. Of the 506 potential POC referrals, 88 (17.4%; 56% women; average age 52.0 years) patients were referred, 60 (11.9%; 57% women; average age 51.6 years) were screened, and 41 (8.1%; 61% women; average age 47.6 years) were enrolled. Patients receiving ATF were more likely to be screened (49% vs 7%; P<.001) and enrolled (15% vs 7%; P<.001) than those without ATF. Chi-square test results revealed significant differences in terms of patient screening (χ²₃[573]=238.6; P<.001) and enrollment status (χ²₃[573]=69.2; P<.001) among referral strategies (Table 2). Specifically, based on the number of referrals completed, there were variations in the proportion and absolute number of enrollees among the four referrals strategies (POC with ATF: 27/190, 50%; POC no ATF: 14/316, 41%; letter ATF: 30/199; 15.1%; letter no ATF: 26/286, 9.1%). Table 2 outlines the number and proportion of potential referrals, referrals made, patients screened, and patients enrolled.

Tables 3 and 4 present the demographic characteristics of participants and the target population and include comparisons of those characteristics among the referral strategies. Among those who enrolled, 59% were women, had an average age of 52.3 (SD 14.3) years, and an average BMI of 35.5 (SD 7.5) kg/m². When compared with the target population (Butler County, Nebraska), participants did not significantly differ among demographic characteristics. Comparing by referral strategies, participants differed significantly in age.

Table 5 provides an overview and categorization of costs across referral strategies. Costs were determined based on nonlabor costs (US $738) and labor costs (US $5380), which were summed to provide the overall costs of recruitment (US $6118). Table 6 provides costs by referral strategy. All reported costs are rounded to the nearest dollar. We estimated an average of 2 minutes per POC referral based on anecdotal reports from the physicians; therefore, costs related to labor for POC referrals were estimated at US $162 for POC with ATF and US $102 for POC without ATF. Labor costs related to RA time spent making recruitment calls were the highest single line item costs and varied greatly between ATF referrals (US $1788) and referrals without ATF (US $135). The varying labor and nonlabor costs yielded different costs per enrolled participant via each referral strategy: letters with ATF was the costliest, at US $86 per enrolled participant, whereas POC with ATF (US $50), POC without ATF (US $61), and letter without ATF (US $51) costs were comparable.

Our results provided some support for the hypothesis that a physician letter referral with ATF is most effective for enrolling a large number of participants into a digitally delivered weight management program compared with POC referrals with and without telephone follow-up and physician letter referrals without follow-up. However, when considering the combined outcomes of penetration into the target population, overall and proportional yield, and cost of recruitment, letter referral without ATF may be the most attractive option for rural primary care clinics that wish to recruit patients into a weight management program. Importantly, all (100%) of the 485 potential letter referrals were made, as compared with the 88 (17%) of the 506 potential POC referrals. Letter with ATF (n=30), letter without ATF (n=27), and POC with ATF (n=27) yielded a similar number of participants. Finally, the costs associated with the letter without ATF referral strategy were relatively low (US $51 per enrolled participant).

Owing to the financial constraints of this pilot study, we limited the number of potential referrals to 991, knowing that this would be far less than the total number of eligible patients seen by the 5 physicians during the recruitment period. Awareness of the finite number of referrals may have influenced the physicians’ decision to refer a patient to the program, perhaps causing them to pass on referring an eligible patient to save that referral for a patient in greater need. However, we do not believe this influence was significant because not all of those potential referrals were used; <60% (573) of the potential referrals were distributed to eligible patients. Although not assessed in this study, primary care physicians often cite a lack of time during patient visits as a barrier to health counseling [26,27], which may also be the reason why the referral distribution rate was low among POC referrals. In addition, it is possible that a physician attempted to refer a patient during a clinic visit but the patient immediately declined the referral to the program. We were unable to quantitatively track this, which may have caused our POC distribution rate to reflect an underrepresentation of reality.

We calculated the proportional yield per referral strategy using potential referrals and referrals made as the denominator. These two analyses revealed different stories. When considering enrollees compared with potential referrals, the letter with active follow-up (15%) and POC with active follow-up (14%) strategies appear to be the best for enrolling a large number of participants into the weight loss program. However, when considering enrollees compared with the number of referrals made, POC with active follow-up (50%) emerges as the clear leader for proportional yield. When considering costs, POC with ATF again appeals as the least expensive recruitment strategy (US $50 per participant), although letter referrals without ATF (US $51) had similar costs. The time-intensive nature of recruitment strategies with ATF not only drives up costs but also places a higher burden on recruitment personnel compared with strategies without ATF. In kind, the cumulative time associated with POC referrals may be similar to that of clinic administrator time spent preparing the letter referrals. Primary care clinics that are considering referring patients to a weight management program should reflect on these approaches with regard to clinical flow and costs. Resource costs in the large volume of letters and telephone calls for letter referral with ATF may make POC referral strategies with active follow-up appealing to small rural clinics, or if system processes are in place to make patient identification and mailing easy, the letter referral without ATF may be the most attractive.

Similar to previous studies [5,11], our findings supported our hypothesis that referrals with ATF would yield a higher proportion of enrollees from those who received a referral. Conversely, our results did not match prior literature [5] and did not support our hypothesis that ATF would yield a lower absolute number of participants. The two conditions with ATF yielded the highest absolute number of participants (letter with ATF, 30 participants; POC with ATF, 27 participants), whereas the letter-only condition yield was comparable (26 participants). Our results supported our hypothesis that POC referrals would have a higher proportional yield of enrolled participants from those who received a referral compared with those who received a letter referral. The behavioral intervention literature has not definitively ruled on a dominant recruitment strategy [10,15]. Our findings provide valuable insight into the reach, representativeness, and cost of POC referrals and electronic health record registry–derived referral letters with and without ATF.

Our sample was older and had a higher proportion of females than the target population. This is easily explained by the following: (1) our intervention was limited to adults aged ≥19 years, whereas the male age of Butler County represents children and adults; (2) women traditionally have higher health care use than men [28]; and (3) excluding infants aged <1 year, older adults aged ≥65 have the highest physician rate by age group [29]. Our sample did not differ on any other demographic variable that was measured, which aligns with representativeness comparisons of active and passive recruitment strategies made by Lee et al [14] but contrasts with results reported by Linnan et al [12], who found that passive strategies yielded lower enrollment but a more diverse and higher-risk sample. Furthermore, among the four referral strategies, those with ATF had the highest (58.7 years, letter with ATF) and lowest (46.0 years, POC with ATF) mean age among participants, therefore providing no support for our hypothesis that ATF strategies would yield a more representative sample than referral strategies without ATF. Although not a significant difference, baseline BMI among enrollees from both POC referral strategies was higher than both letter referral conditions. This may point to an unintentional bias by physicians to refer patients with much higher BMIs than is required for eligibility. Interestingly, POC strategies were more successful at enrolling younger patients (<65 years) than the letter strategies. Although the literature is scant on this specific outcome and drawing any generalizable conclusions from this finding is unjustified, it nonetheless provides an intriguing area for further investigation.

Identifying the best method for maximizing the reach of a weight management program is important for future scale-up efforts. However, maximizing participant retention is a critical next step in the scale-up of any behavioral intervention. An interesting future point of investigation will be the intervention attrition gap of our intervention—the proportion of patients who initially enroll in the program and subsequently do not engage with any program features. This is an especially curious investigation given the conflicting literature regarding the effectiveness of active versus passive recruitment strategies for retaining participants in behavioral interventions [12,13].

There are limitations to our study that should be considered when interpreting our results. As the selected intervention and referral methods were tailored to fit the needs of our target population, the findings of our feasibility study may not be fully generalizable to other primary care clinics. However, the process by which we selected an evidence-based intervention is likely translatable to other researchers and practitioners attempting to select and implement an evidence-based intervention. As previously mentioned, we were not able to track the number of times a physician attempted to make a referral during a clinic visit but was immediately rejected by the patient being seen. Anecdotally, we can attest that this was a rare occurrence, as the physicians reported that the majority of patients were willing to receive or inquire for more information about the program. We did not collect demographic or qualitative data from the 5 physicians involved in referrals and therefore cannot comment on their national representativeness to physicians caring for rural patients or their perceived lack of time for distributing referrals. In addition, census data used to compare our sample with the target population were drawn from Butler County, Nebraska. However, the clinic from which we recruited participants may draw patients from outside Butler County, and comparisons may not directly align. However, when our sample is compared with the demographic characteristics of all adjacent counties (9 total), it remains a representative sample. Indeed, our study has several strengths. Our sample was 59% female, which is a higher proportion of male participants than typical community weight loss programs [30]. Our sample was representative of the racial and ethnic characteristics of the region. Although this feasibility study is small in scale, it addresses key areas of translational research as defined by RE-AIM [31-33] as well as areas of focus cited as integral to feasibility studies, such as adaptation, acceptability, practicality, and integration [34].

We conclude that primary care physicians serving a rural community are capable of referring patients to a digitally delivered behavioral weight management intervention through in-person and population health record–generated letter referrals. When compared with POC methods, letter referrals were more effective at achieving high penetration in the target population and had a slightly higher yield of enrolled patients in this study. However, POC referrals yielded a considerably higher proportion of enrollees from those who received a referral. Participants who received ATF were more likely to be screened and enrolled in the program. These results suggest that either a letter or POC referral strategy can be effective with ATF; however, when resource costs are considered, POC with ATF may be the best method to engage rural primary care patients in a weight management program.