Published on in Vol 5, No 10 (2021): October

Preprints (earlier versions) of this paper are available at, first published .
Reach Outcomes and Costs of Different Physician Referral Strategies for a Weight Management Program Among Rural Primary Care Patients: Type 3 Hybrid Effectiveness-Implementation Trial

Reach Outcomes and Costs of Different Physician Referral Strategies for a Weight Management Program Among Rural Primary Care Patients: Type 3 Hybrid Effectiveness-Implementation Trial

Reach Outcomes and Costs of Different Physician Referral Strategies for a Weight Management Program Among Rural Primary Care Patients: Type 3 Hybrid Effectiveness-Implementation Trial

Original Paper

1Department of Health Promotion, University of Nebraska Medical Center, Omaha, NE, United States

2Center for Reducing Health Disparities, University of Nebraska Medical Center, Omaha, NE, United States

3University of Nebraska Medical Center, Omaha, NE, United States

4Department of Population Health Sciences, University of Utah, Salt Lake City, UT, United States

Corresponding Author:

Gwenndolyn Porter, PhD

Department of Health Promotion

University of Nebraska Medical Center

984365 Nebraska Medical Center

Omaha, NE, 68198

United States

Phone: 1 4025591082


Background: Rural residents are at high risk for obesity; however, little resources exist to address this disproportional burden of disease. Primary care may provide an opportunity to connect primary care patients with overweight and obesity to effective weight management programming.

Objective: The purpose of this study is to examine the utility of different physician referral and engagement processes for improving the reach of an evidence-based and technology-delivered weight management program with counseling support for rural primary care patients.

Methods: A total of 5 rural primary care physicians were randomly assigned a sequence of four referral strategies: point-of-care (POC) referral with active telephone follow-up (ATF); POC referral, no ATF; a population health registry–derived letter referral with ATF; and letter referral, no ATF. For registry-derived referrals, physicians screened a list of patients with BMI ≥25 and approved patients for participation to receive a personalized referral letter via mail.

Results: Out of a potential 991 referrals, 573 (57.8%) referrals were made over 16 weeks, and 98 (9.9%) patients were enrolled in the program (58/98, 59.2% female). Differences based on letter (485/991, 48.9%) versus POC (506/991, 51.1%) referrals were identified for completion (100% vs 7%; P<.001) and for proportion screened (36% vs 12%; P<.001) but not for proportion enrolled (12% vs 8%; P=.10). Patients receiving ATF were more likely to be screened (47% vs 7%; P<.001) and enrolled (15% vs 7%; P<.001) than those not receiving ATF. On the basis of the number of referrals made in each condition, we found variations in the proportion and number of enrollees (POC with ATF: 27/190, 50%; POC no ATF: 14/316, 41%; letter ATF: 30/199; 15.1%; letter no ATF: 27/286, 9.4%). Across all conditions, participants were representative of the racial and ethnic characteristics of the region (60% female, P=.15; 94% White individuals, P=.60; 94% non-Hispanic, P=.19). Recruitment costs totaled US $6192, and the overall recruitment cost per enrolled participant was US $63. Cost per enrolled participant ranged from POC with ATF (US $47), registry-derived letter without ATF (US $52), and POC without ATF (US $56) to registry-derived letter with ATF (US $91).

Conclusions: Letter referral with ATF appears to be the best option for enrolling a large number of patients in a digitally delivered weight management program; however, POC with ATF and letters without ATF yielded similar numbers at a lower cost. The best referral option is likely dependent on the best fit with clinical resources.

Trial Registration: NCT03690557;

JMIR Form Res 2021;5(10):e28622




Obesity is a pressing health concern nationwide, particularly in small rural communities. Rural residents are more obese, on average, than their urban counterparts [1] and often have no or limited access to obesity prevention and treatment programming [1,2]. Furthermore, individuals who use primary care are proportionally more obese than the general public [3], highlighting rural primary care patients as a high-need population regarding weight management. Primary care providers are often the only resource to support healthful eating, physical activity, and weight management in rural communities [4]. Primary care systems may offer a practical and sustainable venue for implementing evidence-based weight management interventions; however, little is known about how rural primary care physicians can pragmatically refer and enroll a large and representative group of individuals into an evidence-based weight management program.

A challenge for weight management interventions is ensuring that not only is an intervention effective but it also has the potential to reach populations at risk that could most benefit [5]. Factors beyond program effectiveness and total sample size, such as proportional yield from those recruited and sample representativeness, are important indicators of an intervention’s impact [6]. Few studies provide a comprehensive report on the methods used to recruit participants, and even fewer report on the representativeness of the sample when compared with the target population [7,8]. In a systematic review of rural weight loss interventions, only 2 of 53 studies compared the demographic characteristics of the intervention sample with those of the target population [8]. In a 2016 systematic review of recruitment strategies for young adult weight gain prevention interventions, 23 of 25 studies were reported to have insufficiently described the recruitment process [7]. Documenting recruitment methods and the representativeness outcomes of those efforts can lead to a better overall understanding of how best to engage patients and maximize the reach of evidence-based behavioral interventions. This understanding is important for scale-up efforts, for preventing underrepresentation of populations experiencing disparities in research and in health care, and for physicians to improve their standard of care [9].

However, when drawing more broadly from the behavioral intervention literature, a number of examples of both active and passive recruitment strategies can be found that have been used to engage various target populations [7,10-13]. On the basis of the available literature, active recruitment strategies—those with direct interaction with potential participants, such as outreach telephone calls—appear to yield a lower absolute number of participants but a higher proportion of those exposed to recruitment strategies when compared with passive recruitment strategies (ie, those without direct interaction with potential participants, such as flyers or targeted mailings), which yield a higher number but a lower proportion of participants [5]. Active recruitment strategies may also yield a more representative sample than passive recruitment strategies [12,14,15]. The limited knowledge in this area warrants further study.


The purpose of this study is to examine the utility and cost of different physician referral and engagement processes for improving reach (ie, number, proportion, and representativeness of participants) for a rural, evidence-based, and technology-delivered weight management program with counseling support. Our design is based on feedback gathered from prior qualitative work [16,17]; that is, an expressed clinical interest in testing the relative reach of engaging patients at the point of care (POC) during a well or chronic care visit or proactively reaching out to patients using a population health management approach facilitated by an electronic health record.

We use the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to guide our assessment of reach [18] and hypothesize that active recruitment strategies would yield a more representative sample than passive recruitment strategies. Reach refers to the absolute number, proportion, and representativeness of the participants compared with those who were exposed to recruitment efforts [18]. We also hypothesize that POC referrals, compared with the population health registry–derived letter referrals, would yield a higher proportion of enrolled participants from those who received a referral.

Intervention Selection

Before this pilot trial, we conducted focus groups with primary care staff employed at rural primary care clinics regarding the feasibility of implementing a weight loss program through primary care. Overwhelmingly, primary care staff agreed that a program to which physicians and nurses could refer eligible patients and track their progress throughout was more favorable than a program that would require physician- or nurse-led delivery [16,17]. We then assembled primary care physicians, staff, and obesity treatment experts from the Great Plains Practice-Based Research Network to engage in a participatory selection process [19] to identify and, if necessary, adapt an evidence-based intervention to pilot-test through a rural primary care clinic. The selection process is described elsewhere [17]. Ultimately, a digitally delivered intervention was selected by the group for local testing, and it was agreed that testing potential recruitment strategies was a priority over testing the relative effectiveness of the different weight loss programs.

Study Design


The study design used the hybrid methodologies described by Curran et al [20], which allow for blended design components targeting effectiveness and implementation but focus on systems-based approaches to improve dissemination at the participant level rather than implementing the evidence-based intervention. As such, we classified this trial as a hybrid type 3 effectiveness-implementation trial focused on dissemination at the participant level (ie, reach). This allowed us to test the utility of different dissemination strategies to increase program reach as a primary outcome while concurrently gathering information on intervention effectiveness [20]. We partnered with a rural primary care clinic and the 5 physicians serving their patients to implement an evidence-based and digitally delivered weight loss program. The 5 physicians were randomly assigned to a sequence of four referral strategies over a span of 16 weeks: POC referral with active telephone follow-up (ATF); POC referral, no ATF; a population health registry–derived letter referral with ATF; and letter referral, no ATF.

Dissemination Strategies: Referral Methods

Referral strategies varied by POC versus a population health electronic health record–derived letter referral and ATF versus no telephone follow-up. For POC referral, physicians were instructed to refer any adult patient to an evidence-based weight loss program with diet and physical activity counseling based on (1) BMI≥25 kg/m2, (2) no contraindications to participation, and (3) visiting the clinic for a chronic care or well visit. In the population health approach, a clinic administrator pulled a list of patients from the electronic health record system with BMI≥25 who had visited the clinic in the previous 2 weeks. Each physician reviewed this list and removed any patients with contraindications to participation in the evidence-based weight loss program. The remaining patients were mailed a personalized invitation letter to participate in the weight loss intervention that was signed by their physician.

For ATF, the referred patient was informed that they would be contacted by a member of the research team to determine if they would like to participate in the program (an opt-out telephone number was also provided). In conditions without ATF, the referred patient was provided a telephone number to call if they were interested in discussing participation in the program. The study was reviewed and approved by the University of Nebraska Medical Center Institutional Review Board (#581-18-EP).

We designed this pilot study to recruit over a 16-week period, with the goal of enrolling approximately 100 participants and projected a 10% enrollment rate based on the overall denominator of potential referrals based on prior work [21,22]. Each physician (n=5) was randomly assigned a sequence of the four referral strategies, shifting strategies every 2 weeks (Table 1). Over the course of 16 weeks, physicians used each of the four referral strategies twice to eliminate any potential time effect on a single strategy. Each physician’s referral strategy sequence was randomized to prevent any order effect on the yield of patients per referral strategy. A member of the research team visited the clinic every 2 weeks to remind physicians of their new referral strategy; a nurse at the clinic monitored the physicians daily to ensure that the referral strategies were implemented with fidelity.

Table 1. Randomization sequence of referral strategy by physician.
WeeksPhysician APhysician BPhysician CPhysician DPhysician E

aL: letter referral without active telephone follow-up.

bPOC+: point-of-care referral with active telephone follow-up.

cPOC: point-of-care referral without active telephone follow-up.

dL+: letter referral with active telephone follow-up.


All referred and interested patients were required to undergo a brief telephone screening with a member of the research team and, if eligible, were given instructions over the phone and via email on how to complete web-based program enrollment. In addition, all interested and eligible patients were scheduled for an in-person enrollment visit with a member of the research team where they were given intervention program materials (ie, home scale) and a tutorial of the intervention’s mobile app, provided with tips on how to stay engaged with the program, and completed the web-based program enrollment (if not already complete). Patient consent to participate in the weight loss program was obtained during the web-based enrollment process. Patients were given the opportunity to raise questions before consent and during the in-person enrollment visit.

Evidence-Based Weight Loss Program

All referred patients were offered a 12-month, digitally delivered, evidence-based weight management program free of charge. The program featured a social cognitive theory–based curriculum with counseling support delivered through daily emails and text messages. Program features also included daily meal plans and physical activity recommendations [23]. Estabrooks et al [23] provided a comprehensive overview of the program. In addition, modest financial incentives were offered with the intent to increase program reach and retention [24]. Incentives were offered to participants who lost a minimum of 5% of their initial body weight (US $15/quarter reward) graded up to a maximum of 30% body weight reduction (US $150/quarter reward). Program participants were provided a Bluetooth-enabled home scale (Smart Scale, incentaHEALTH, LLC), which connected to the program smartphone app that was installed on their smartphones during the enrollment visit. Participants were instructed to record their weight using this scale no less than once per quarter. The program also featured a website where participants could receive feedback on their weigh-ins, take health quizzes, and self-assess their progress with regard to healthy eating, physical activity, and weight loss. Upon completion of the program, participants could keep the home scale. This paper focuses on reach, which is the primary outcome of this study; data on weight loss are not presented here.

Data Analysis

Program reach was measured by the number and proportion of individuals who were (1) referred, (2) screened, and (3) enrolled and was compared across referral strategies relative to the number of eligible patients who visited the clinic during the recruitment period. Representativeness of the enrolled sample was assessed relative to the demographic characteristics of the region, as measured by the US Census [25]. We used chi-square tests to examine group differences in terms of screening and participation rates among referral strategies and to determine whether screening and participation rates differed according to (1) ATF versus no ATF and (2) POC versus letter. We applied a one-sample test of proportion (in the case of comparing proportions) to examine representativeness in terms of demographic characteristics of the enrolled participants compared with census data. We further conducted group comparisons using one-way analysis of variance tests for continuous variables and chi-square tests for categorical variables. Two-tailed P values <.05 were considered statistically significant for this study.

Costs of recruitment were prospectively and retrospectively estimated based on the costs of recruitment materials (eg, handouts or flyers), supplies, and recruitment activities, including telephone follow-up, telephone screening, and in-person sessions. Labor costs were calculated using the research assistant (RA) annual salary (US $25,000) and publicly available average salary estimates for primary care physicians (US $187,013) and clinic managers (US $65,356). RAs tracked the time spent on various recruitment activities, number of phone calls made, enrollment sessions completed, and number of flyers printed in a custom computer database. Cost results are presented for each referral strategy.


Over a period of 16 weeks, 2534 eligible patients visited the clinic. The maximum number of referrals that could have been made during well or chronic care visits over this time were approximately 30% of patient visits (n=991; hereafter referred to as potential referrals). The actual number of referrals made by the 5 physicians was 573 (274, 47.8% women; average age 55.7, SD 16.8 years) and, out of the 573 referrals, 98 (17.1%) patients were enrolled, representing an overall enrollment rate of 10% of the potentially eligible patient population. Of the 485 potential letter referrals, 485 (100%; 46% women; average age 56.3 years) were completed, 229 (47.2%; 48% women; average age 56.1 years) patients were screened by telephone, and 57 (11.8%; 58% women; average age 55.8 years) were enrolled. Of the 506 potential POC referrals, 88 (17.4%; 56% women; average age 52.0 years) patients were referred, 60 (11.9%; 57% women; average age 51.6 years) were screened, and 41 (8.1%; 61% women; average age 47.6 years) were enrolled. Patients receiving ATF were more likely to be screened (49% vs 7%; P<.001) and enrolled (15% vs 7%; P<.001) than those without ATF. Chi-square test results revealed significant differences in terms of patient screening (χ23[573]=238.6; P<.001) and enrollment status (χ23[573]=69.2; P<.001) among referral strategies (Table 2). Specifically, based on the number of referrals completed, there were variations in the proportion and absolute number of enrollees among the four referrals strategies (POC with ATF: 27/190, 50%; POC no ATF: 14/316, 41%; letter ATF: 30/199; 15.1%; letter no ATF: 26/286, 9.1%). Table 2 outlines the number and proportion of potential referrals, referrals made, patients screened, and patients enrolled.

Table 2. Reach results by referral strategy (N=991).
StrategyPotential referralsReferrals made, n (%)Patients screened, n (%)Patients screened and eligible, n (%)Patients enrolled, n (%)
POCa+ATFb19054 (28.4)45 (23.7)c,d,e27 (14.2)27 (14.2)c
POC31634 (10.8)14 (4.4)d,e,f14 (4.4)14 (4.4)d,e,f
Letter+ATF199199 (100)147 (73.9)c,e,f30 (15.1)30 (15.1)c
Letter286286 (100)30 (10.5)c,d,f27 (9.4)26 (9.1)c
Total991573 (57.8)236 (23.8)98 (9.9)97 (9.8)

aPOC: point of care.

bATF: active telephone follow-up.

cSignificantly different from POC at α=.05.

dSignificantly different from letter+ at α=.05.

eSignificantly different from letter at α=.05.

fSignificantly different from POC+ at α=.05.

Tables 3 and 4 present the demographic characteristics of participants and the target population and include comparisons of those characteristics among the referral strategies. Among those who enrolled, 59% were women, had an average age of 52.3 (SD 14.3) years, and an average BMI of 35.5 (SD 7.5) kg/m2. When compared with the target population (Butler County, Nebraska), participants did not significantly differ among demographic characteristics. Comparing by referral strategies, participants differed significantly in age.

Table 3. Representativeness of study sample (N=97).
DemographicsTarget population: Butler County, NEaTotal enrolled participants
Age, median (years)43.355.0b
Sex (female), n (%)4026 (50.1)58 (59)
Obese at baseline, %(32)c75 (77)b
White, n (%)7846 (97.7)91 (94)
Hispanic or Latino, n (%)287 (3.6)3 (3)

aNE: Nebraska; data as per 2019 estimates.

bSignificant difference between target population and sample at α=.05.

cOnly percentage estimate available.

Table 4. Representativeness comparisons among participants by referral strategies (N=97).
DemographicsTotal samplePOCa+ATFb (n=27)POC no ATF (n=14)Letter+ATF (n=30)Letter no ATF (n=26)Chi-square (df)ANOVAc (df)P value
Age (years), mean (SD)52.3 (14.3)46.0 (14.7)d51.9 (9.3)58.7e (13.1)53.4 (15.1)N/AfF93=4.1 (3).009
Sex (female), n (%)58.8596457580.2 (3)N/A.97
Baseline BMI mean (SD), kg/m235.5 (7.5)37.1 (6.8)38.1 (9.1)38.1 (9.1)33.2 (6.6)N/AF72=1.5 (3).22
Age >65, n (%)23 (23.7)7.4d7.143.3e26.912.6 (3)N/A.07
White individuals, n (%)949393100922.3 (3)N/A.51
Hispanic or Latino, n (%)33.7007.73.1 (3)N/A.37

aPOC: point of care.

bATF: active telephone follow-up.

cANOVA: analysis of variance.

dSignificantly different from letter+ATF at α=.05.

eSignificantly different from POC+ATF at α=.05.

fN/A: not applicable.


Table 5 provides an overview and categorization of costs across referral strategies. Costs were determined based on nonlabor costs (US $738) and labor costs (US $5380), which were summed to provide the overall costs of recruitment (US $6118). Table 6 provides costs by referral strategy. All reported costs are rounded to the nearest dollar. We estimated an average of 2 minutes per POC referral based on anecdotal reports from the physicians; therefore, costs related to labor for POC referrals were estimated at US $162 for POC with ATF and US $102 for POC without ATF. Labor costs related to RA time spent making recruitment calls were the highest single line item costs and varied greatly between ATF referrals (US $1788) and referrals without ATF (US $135). The varying labor and nonlabor costs yielded different costs per enrolled participant via each referral strategy: letters with ATF was the costliest, at US $86 per enrolled participant, whereas POC with ATF (US $50), POC without ATF (US $61), and letter without ATF (US $51) costs were comparable.

Table 5. Breakdown of recruitment costs, rounded to the nearest dollar.
Cost elementCost (US $)

Printing letters and program descriptions, n=97043


Envelopes, n=48558

POC cards170

ITa (phone line)200

Physician time spent making referrals264

Clinic manager time spent pulling patient list, 2 hours63
Research assistant time

Training clinical staff, 2 hours48

Letter preparation, 24 hours577

POCb preparation, 8 hours192

Recruitment calls, 80 hours1923

Enrollment visit preparation, 4 hours96

Enrollment visit, 49 hours1178

Mileage reimbursement to enrollment visits1039
Total recruitment costs6118
Total recruitment costs per enrolled participant 63

aIT: information technology.

bPOC: point of contact.

Table 6. Recruitment costs in terms of yield by referral strategy, rounded to the nearest dollar (N=98).
Cost elementPOCa+ (US $; n=27)POC (US $; n=14)Letter+ (US $; n=30)Letter (US $; n=27)Total sample (US $)

Printing letters and program descriptions, n=970N/AbN/A182543


Envelopes, n=485N/AN/A243458

POC cards9971N/AN/A170

ITc,d (phone line)50505050200

Physician time spent making referrals16210200264

Clinic manager time spent pulling patient list, 2 hours0031.5031.5063
Research assistant time

Training clinical staff, 2 hoursd1212121248

Letter preparation, 24 hoursN/AN/A237340577

POC preparation, 8 hours73119N/AN/A192

Recruitment calls, 80 hours346581442771923

Enrollment visit prep, 4 hours2424242496

Enrollment visit, 49 hours3241653653241178

Mileage reimbursement to enrollment visitsd259.75259.75259.75259.751039
Total recruitment costs1350861257313346118
Total recruitment costs per enrolled participant5061865163

aPOC: point of contact.

bN/A: not applicable.

cIT: information technology.

dThese items or activities were not specific to any one referral strategy; therefore, their costs were split evenly among each of the referral strategies.

Principal Findings

Our results provided some support for the hypothesis that a physician letter referral with ATF is most effective for enrolling a large number of participants into a digitally delivered weight management program compared with POC referrals with and without telephone follow-up and physician letter referrals without follow-up. However, when considering the combined outcomes of penetration into the target population, overall and proportional yield, and cost of recruitment, letter referral without ATF may be the most attractive option for rural primary care clinics that wish to recruit patients into a weight management program. Importantly, all (100%) of the 485 potential letter referrals were made, as compared with the 88 (17%) of the 506 potential POC referrals. Letter with ATF (n=30), letter without ATF (n=27), and POC with ATF (n=27) yielded a similar number of participants. Finally, the costs associated with the letter without ATF referral strategy were relatively low (US $51 per enrolled participant).

Owing to the financial constraints of this pilot study, we limited the number of potential referrals to 991, knowing that this would be far less than the total number of eligible patients seen by the 5 physicians during the recruitment period. Awareness of the finite number of referrals may have influenced the physicians’ decision to refer a patient to the program, perhaps causing them to pass on referring an eligible patient to save that referral for a patient in greater need. However, we do not believe this influence was significant because not all of those potential referrals were used; <60% (573) of the potential referrals were distributed to eligible patients. Although not assessed in this study, primary care physicians often cite a lack of time during patient visits as a barrier to health counseling [26,27], which may also be the reason why the referral distribution rate was low among POC referrals. In addition, it is possible that a physician attempted to refer a patient during a clinic visit but the patient immediately declined the referral to the program. We were unable to quantitatively track this, which may have caused our POC distribution rate to reflect an underrepresentation of reality.

We calculated the proportional yield per referral strategy using potential referrals and referrals made as the denominator. These two analyses revealed different stories. When considering enrollees compared with potential referrals, the letter with active follow-up (15%) and POC with active follow-up (14%) strategies appear to be the best for enrolling a large number of participants into the weight loss program. However, when considering enrollees compared with the number of referrals made, POC with active follow-up (50%) emerges as the clear leader for proportional yield. When considering costs, POC with ATF again appeals as the least expensive recruitment strategy (US $50 per participant), although letter referrals without ATF (US $51) had similar costs. The time-intensive nature of recruitment strategies with ATF not only drives up costs but also places a higher burden on recruitment personnel compared with strategies without ATF. In kind, the cumulative time associated with POC referrals may be similar to that of clinic administrator time spent preparing the letter referrals. Primary care clinics that are considering referring patients to a weight management program should reflect on these approaches with regard to clinical flow and costs. Resource costs in the large volume of letters and telephone calls for letter referral with ATF may make POC referral strategies with active follow-up appealing to small rural clinics, or if system processes are in place to make patient identification and mailing easy, the letter referral without ATF may be the most attractive.

Similar to previous studies [5,11], our findings supported our hypothesis that referrals with ATF would yield a higher proportion of enrollees from those who received a referral. Conversely, our results did not match prior literature [5] and did not support our hypothesis that ATF would yield a lower absolute number of participants. The two conditions with ATF yielded the highest absolute number of participants (letter with ATF, 30 participants; POC with ATF, 27 participants), whereas the letter-only condition yield was comparable (26 participants). Our results supported our hypothesis that POC referrals would have a higher proportional yield of enrolled participants from those who received a referral compared with those who received a letter referral. The behavioral intervention literature has not definitively ruled on a dominant recruitment strategy [10,15]. Our findings provide valuable insight into the reach, representativeness, and cost of POC referrals and electronic health record registry–derived referral letters with and without ATF.

Our sample was older and had a higher proportion of females than the target population. This is easily explained by the following: (1) our intervention was limited to adults aged ≥19 years, whereas the male age of Butler County represents children and adults; (2) women traditionally have higher health care use than men [28]; and (3) excluding infants aged <1 year, older adults aged ≥65 have the highest physician rate by age group [29]. Our sample did not differ on any other demographic variable that was measured, which aligns with representativeness comparisons of active and passive recruitment strategies made by Lee et al [14] but contrasts with results reported by Linnan et al [12], who found that passive strategies yielded lower enrollment but a more diverse and higher-risk sample. Furthermore, among the four referral strategies, those with ATF had the highest (58.7 years, letter with ATF) and lowest (46.0 years, POC with ATF) mean age among participants, therefore providing no support for our hypothesis that ATF strategies would yield a more representative sample than referral strategies without ATF. Although not a significant difference, baseline BMI among enrollees from both POC referral strategies was higher than both letter referral conditions. This may point to an unintentional bias by physicians to refer patients with much higher BMIs than is required for eligibility. Interestingly, POC strategies were more successful at enrolling younger patients (<65 years) than the letter strategies. Although the literature is scant on this specific outcome and drawing any generalizable conclusions from this finding is unjustified, it nonetheless provides an intriguing area for further investigation.

Identifying the best method for maximizing the reach of a weight management program is important for future scale-up efforts. However, maximizing participant retention is a critical next step in the scale-up of any behavioral intervention. An interesting future point of investigation will be the intervention attrition gap of our intervention—the proportion of patients who initially enroll in the program and subsequently do not engage with any program features. This is an especially curious investigation given the conflicting literature regarding the effectiveness of active versus passive recruitment strategies for retaining participants in behavioral interventions [12,13].

There are limitations to our study that should be considered when interpreting our results. As the selected intervention and referral methods were tailored to fit the needs of our target population, the findings of our feasibility study may not be fully generalizable to other primary care clinics. However, the process by which we selected an evidence-based intervention is likely translatable to other researchers and practitioners attempting to select and implement an evidence-based intervention. As previously mentioned, we were not able to track the number of times a physician attempted to make a referral during a clinic visit but was immediately rejected by the patient being seen. Anecdotally, we can attest that this was a rare occurrence, as the physicians reported that the majority of patients were willing to receive or inquire for more information about the program. We did not collect demographic or qualitative data from the 5 physicians involved in referrals and therefore cannot comment on their national representativeness to physicians caring for rural patients or their perceived lack of time for distributing referrals. In addition, census data used to compare our sample with the target population were drawn from Butler County, Nebraska. However, the clinic from which we recruited participants may draw patients from outside Butler County, and comparisons may not directly align. However, when our sample is compared with the demographic characteristics of all adjacent counties (9 total), it remains a representative sample. Indeed, our study has several strengths. Our sample was 59% female, which is a higher proportion of male participants than typical community weight loss programs [30]. Our sample was representative of the racial and ethnic characteristics of the region. Although this feasibility study is small in scale, it addresses key areas of translational research as defined by RE-AIM [31-33] as well as areas of focus cited as integral to feasibility studies, such as adaptation, acceptability, practicality, and integration [34].


We conclude that primary care physicians serving a rural community are capable of referring patients to a digitally delivered behavioral weight management intervention through in-person and population health record–generated letter referrals. When compared with POC methods, letter referrals were more effective at achieving high penetration in the target population and had a slightly higher yield of enrolled patients in this study. However, POC referrals yielded a considerably higher proportion of enrollees from those who received a referral. Participants who received ATF were more likely to be screened and enrolled in the program. These results suggest that either a letter or POC referral strategy can be effective with ATF; however, when resource costs are considered, POC with ATF may be the best method to engage rural primary care patients in a weight management program.


The authors would like to express their gratitude to the physicians, nurses, and administrative staff involved in recruitment and data collection.

Conflicts of Interest

None declared.

  1. Befort C, Nazir N, Perri M. Prevalence of obesity among adults from rural and urban areas of the United States: findings from NHANES (2005-2008). J Rural Health 2012;28(4):392-397 [FREE Full text] [CrossRef] [Medline]
  2. Phillips CD, McLeroy KR. Health in rural America: remembering the importance of place. Am J Public Health 2004 Oct;94(10):1661-1663. [CrossRef] [Medline]
  3. Befort CA, VanWormer JJ, DeSouza C, Ellerbeck EF, Kimminau KS, Greiner A, et al. Protocol for the Rural Engagement in Primary Care for Optimizing Weight Reduction (RE-POWER) trial: comparing three obesity treatment models in rural primary care. Contemp Clin Trials 2016 Mar;47:304-314. [CrossRef] [Medline]
  4. Phillips SM, Glasgow RE, Bello G, Ory MG, Glenn BA, Sheinfeld-Gorin SN, MOHR Study Group. Frequency and prioritization of patient health risks from a structured health risk assessment. Ann Fam Med 2014;12(6):505-513 [FREE Full text] [CrossRef] [Medline]
  5. Estabrooks P, You W, Hedrick V, Reinholt M, Dohm E, Zoellner J. A pragmatic examination of active and passive recruitment methods to improve the reach of community lifestyle programs: The Talking Health Trial. Int J Behav Nutr Phys Act 2017 Jan 19;14(1):7 [FREE Full text] [CrossRef] [Medline]
  6. Glasgow RE, Nelson CC, Strycker LA, King DK. Using RE-AIM metrics to evaluate diabetes self-management support interventions. Am J Prev Med 2006 Jan;30(1):67-73. [CrossRef] [Medline]
  7. Lam E, Partridge SR, Allman-Farinelli M. Strategies for successful recruitment of young adults to healthy lifestyle programmes for the prevention of weight gain: a systematic review. Obes Rev 2016 Feb;17(2):178-200. [CrossRef] [Medline]
  8. Porter GC, Laumb K, Michaud T, Brito F, Petreca D, Schwieger G, et al. Understanding the impact of rural weight loss interventions: a systematic review and meta-analysis. Obes Rev 2019 May;20(5):713-724 [FREE Full text] [CrossRef] [Medline]
  9. Estabrooks P, Harden S, Almeida F, Hill JL, Johnson SB, Porter GC, et al. Using integrated research-practice partnerships to move evidence-based principles into practice. Exerc Sport Sci Rev 2019 Jul;47(3):176-187 [FREE Full text] [CrossRef] [Medline]
  10. Foster CE, Brennan G, Matthews A, McAdam C, Fitzsimons C, Mutrie N. Recruiting participants to walking intervention studies: a systematic review. Int J Behav Nutr Phys Act 2011 Dec 15;8(1):137 [FREE Full text] [CrossRef] [Medline]
  11. Ives DG, Kuller LH, Schulz R, Traven ND, Lave JR. Comparison of recruitment strategies and associated disease prevalence for health promotion in rural elderly. Prev Med 1992 Sep;21(5):582-591. [CrossRef] [Medline]
  12. Linnan LA, Emmons KM, Klar N, Fava JL, LaForge RG, Abrams DB. Challenges to improving the impact of worksite cancer prevention programs: comparing reach, enrollment, and attrition using active versus passive recruitment strategies. Ann Behav Med 2002;24(2):157-166. [CrossRef] [Medline]
  13. Raynor HA, Osterholt KM, Hart CN, Jelalian E, Vivier P, Wing RR. Evaluation of active and passive recruitment methods used in randomized controlled trials targeting pediatric obesity. Int J Pediatr Obes 2009;4(4):224-232 [FREE Full text] [CrossRef] [Medline]
  14. Lee RE, McGinnis KA, Sallis JF, Castro CM, Chen AH, Hickmann SA. Active vs. passive methods of recruiting ethnic minority women to a health promotion program. Ann Behav Med 1997;19(4):378-384. [CrossRef] [Medline]
  15. UyBico SJ, Pavel S, Gross CP. Recruiting vulnerable populations into research: a systematic review of recruitment interventions. J Gen Intern Med 2007 Jun 21;22(6):852-863 [FREE Full text] [CrossRef] [Medline]
  16. Porter G. Identifying, adapting, and implementing a weight management program into rural primary care. Theses & Dissertations, University of Nebraska Medical Center. 2020.   URL: [accessed 2021-03-08]
  17. Porter GC, Schwab R, Hill JL, Bartee T, Heelan KA, Michaud TL, et al. Examining the feasibility and characteristics of realistic weight management support for patients: focus groups with rural, micropolitan, and metropolitan primary care providers. Prev Med Rep 2021 Apr 30;23:101390 [FREE Full text] [CrossRef] [Medline]
  18. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [CrossRef] [Medline]
  19. Estabrooks P, Smith-Ray R, Almeida F, Hill J, Gonzales M, Schreiner P, et al. Move More: translating an efficacious group dynamics physical activity intervention into effective clinical practice. Int J Sport Exerc Psychol 2011 Mar;9(1):4-18 [FREE Full text] [CrossRef]
  20. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012 Mar;50(3):217-226 [FREE Full text] [CrossRef] [Medline]
  21. Almeida FA, Michaud TL, Wilson KE, Schwab RJ, Goessl C, Porter GC, et al. Preventing diabetes with digital health and coaching for translation and scalability (PREDICTS): a type 1 hybrid effectiveness-implementation trial protocol. Contemp Clin Trials 2020 Jan;88:105877. [CrossRef] [Medline]
  22. Almeida FA, Pardo KA, Seidel RW, Davy BM, You W, Wall SS, et al. Design and methods of "diaBEAT-it!": a hybrid preference/randomized control trial design using the RE-AIM framework. Contemp Clin Trials 2014 Jul;38(2):383-396 [FREE Full text] [CrossRef] [Medline]
  23. Estabrooks PA, Wilson KE, McGuire TJ, Harden SM, Ramalingam N, Schoepke L, et al. A quasi-experiment to assess the impact of a scalable, community-based weight loss program: combining reach, effectiveness, and cost. J Gen Intern Med 2017 Apr;32(Suppl 1):24-31 [FREE Full text] [CrossRef] [Medline]
  24. Almeida FA, You W, Harden SM, Blackman KC, Davy BM, Glasgow RE, et al. Effectiveness of a worksite-based weight loss randomized controlled trial: the worksite study. Obesity (Silver Spring) 2015 Apr;23(4):737-745 [FREE Full text] [CrossRef] [Medline]
  25. 2007-2011 American Community Survey 5-year estimates. United States Census Bureau. 2011.   URL: [accessed 2021-03-08]
  26. Kolasa KM, Rickett K. Barriers to providing nutrition counseling cited by physicians: a survey of primary care practitioners. Nutr Clin Pract 2010 Oct;25(5):502-509. [CrossRef] [Medline]
  27. Shuval K, Leonard T, Drope J, Katz DL, Patel AV, Maitin-Shepard M, et al. Physical activity counseling in primary care: insights from public health and behavioral economics. CA Cancer J Clin 2017 May 06;67(3):233-244 [FREE Full text] [CrossRef] [Medline]
  28. Bertakis KD, Azari R, Helms LJ, Callahan EJ, Robbins JA. Gender differences in the utilization of health care services. J Fam Pract 2000 Feb;49(2):147-152. [Medline]
  29. Ashman J, Rui P, Okeyode T. Characteristics of Office-based Physician Visits, 2016. National Center for Health Statistics - Data Brief No. 331. 2019.   URL: [accessed 2021-03-07]
  30. Harden SM, Gaglio B, Shoup JA, Kinney KA, Johnson SB, Brito F, et al. Fidelity to and comparative results across behavioral interventions evaluated through the RE-AIM framework: a systematic review. Syst Rev 2015 Nov 08;4(1):155 [FREE Full text] [CrossRef] [Medline]
  31. Glasgow RE, Goldstein MG, Ockene JK, Pronk NP. Translating what we have learned into practice. Principles and hypotheses for interventions addressing multiple behaviors in primary care. Am J Prev Med 2004 Aug;27(2 Suppl):88-101. [CrossRef] [Medline]
  32. Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health 2003 Aug;93(8):1261-1267. [CrossRef] [Medline]
  33. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to "employ" the RE-AIM model? Eval Health Prof 2013 Mar 21;36(1):44-66. [CrossRef] [Medline]
  34. Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med 2009 May;36(5):452-457 [FREE Full text] [CrossRef] [Medline]

ATF: active telephone follow-up
POC: point of care
RA: research assistant
RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance

Edited by G Eysenbach; submitted 08.03.21; peer-reviewed by H Belli, D Downs; comments to author 10.05.21; revised version received 15.07.21; accepted 26.07.21; published 20.10.21


©Gwenndolyn Porter, Tzeyu L Michaud, Robert J Schwab, Jennie L Hill, Paul A Estabrooks. Originally published in JMIR Formative Research (, 20.10.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.