Enrollment and Retention Outcomes from the Veterans Health Administration for a Remote Digital Health Study: Multisite Observational Study

Introduction

Digital health technologies such as remote patient monitoring (RPM) and mobile health apps enable health care to be provided outside the time and space of a clinical encounter, greatly expanding the potential reach of health care interventions. RPM can leverage wearables and connected medical equipment like weight scales and heart monitors to increase the volume, time period, and accessibility of health data available to clinicians for the management of chronic health conditions. Mobile apps may offer a range of tools, including education and counseling, diet and exercise tracking, and medication reminders. Since the nature of RPM and mobile apps is that individuals use them in their home environment, research to establish safety, efficacy, effectiveness, or validity can often be conducted remotely. Electronic data tools, including patient-facing technologies, can be used for consenting as well as intervention administration, remote interviews, outcome assessment, adherence monitoring, and data transmission [1,2]. Patients can thus participate from their own homes, with minimal or no travel to a study site. Studies of this type are variously described as remote, virtual, decentralized, direct-to-patient, and online. Daniore et al [3], define remote digital health studies as “longitudinal studies that use mobile technologies to conduct all key steps of a study completely online.”

Remote studies mimic the real-world use case for RPM and mobile apps. They also untether potential participants from the constraints of time and travel, which present known obstacles to sufficient enrollment [3]. A survey of cancer patients in the United States found that, if given the opportunity to enroll in a cancer clinical trial requiring travel further than their usual source of care, most respondents were more likely to participate if the trials used remote technology and decentralization tools to reduce the need for travel [4]. Yet remote digital health studies continue to face enrollment challenges, just as traditional digital and nondigital health studies do. These challenges can be difficult to overcome [5,6] and include lack of trust in clinical research, privacy concerns, participant time commitment, required effort, required experience with technology, and insufficient internal and external incentives [1,6-8]. Over 20% of clinical trials, including remote digital health studies, are terminated due to failure to meet enrollment targets [9]. One potential approach to meeting enrollment targets is to identify a large eligible sample at the outset, using large databases. This increases the likelihood that, even if the overall enrollment yield is low, enrollment targets will be met.

The Veterans Health Administration (VA), represents the nation’s largest integrated health system and offers a very large sample of patients from which to identify and recruit eligible participants. The VA includes over 9 million Veteran enrollees across the United States. VA provides health care at 170 VA Medical Centers and 1380 outpatient sites of care [10] that share a common underlying electronic health record with information rolled up into a national data warehouse. The data warehouse enables linkage of study data collected by RPM or mobile app data with a vast amount of VA clinical, health care usage, and cost data. VA-affiliated nonprofit centers offer flexibility for VA researchers to carry out studies with support from industry, foundations, academia, and other governmental agencies. Thus, the VA offers an ideal setting for large remote digital health studies.

In this paper, we report on study enrollment and completion among patients from the VA health system for a remote digital health study. The overall goal of the study was to develop an algorithm using data from a handheld RPM device to predict clinical decompensation (ie, worsening of existing symptoms or emergence of new symptoms). The study enrolled patients who were hospitalized with COVID-19 in the previous 3-6 months and the follow-up period was 6 months. Data collected from VA patients was provided to the industry sponsor for algorithm development. We describe the enrollment yield, characteristics of enrolled versus nonenrolled patients, and characteristics of patients who completed versus did not complete the study. Our results provide insights into the promise of successful remote digital health studies conducted in the VA health system and may help others plan similar studies in other large integrated health systems.

Methods

Setting

The VA health system serves over 7 million of its 9 million enrolled Veteran patients annually [10]. Veterans as a group are older than the civilian population, though women, who make up about 11% of VA users, tend to be younger. The largest age group of female VA patients is 35-44 years (22%), while the largest age group of male VA patients is 65-74 years (30%). Overall, 77% of VA patients are White, 18% of them are Black, and 8% of them are Hispanic. Veterans have a higher burden of physical and mental health comorbidities, including diabetes, obesity, smoking, depression, anxiety, and posttraumatic stress disorder [11-16]. Rates are also higher for substance use disorders, brain injuries, mental health disorders, and have a higher rate of committing suicide [17].

The health system includes 170 medical centers and nearly 1200 community-based outpatient clinics [10]. In many areas of the country, medical centers and clinics work together to offer inpatient, outpatient, and long-term care services in a geographic area. Across VA health care facilities, there is a single underlying platform for the electronic health record, which is called VistA. Clinicians conduct patient care using the VistA interface called the Computerized Patient Record System. Local instances of VistA are pushed into the national VA Corporate Data Warehouse (CDW) nightly. The CDW is a relational SQL server database that supports data analysis and reporting across multiple patients. The CDW can be queried to identify potentially eligible patients and also to gather longitudinal clinical and health care utilization data that can then be linked to data from digital health tools to examine outcomes of interest and their predictors.

Study Design and Participants

This was an observational study of an RPM device. Details of the protocol, including information on recruitment procedures, have been published elsewhere [18]. The study team was based across 2 VA medical centers, with patients recruited from across the country without respect to their home VA facility. Briefly, Veterans were recruited to use a cellular-enabled, handheld, multisensory, FDA-approved device, MouthLab [19], that captures multiple health parameters through breath- and saliva-based biosensors and transmits the data to a cloud-based dashboard for viewing by clinical providers (ClinicalTrials.gov, NCT05713266). Enrolled patients used the device for 60 seconds twice daily over a 6-month follow-up period and filled out monthly web-based symptom surveys. Data from the device and surveys were combined with VA clinical and utilization data to support the development of a predictive algorithm for clinical decompensation among patients at high risk.

Patients aged 18 and older were eligible for the study if they had been hospitalized with a diagnosis of COVID-19 in the previous 3-6 months and had a pre-existing diagnosis of chronic obstructive pulmonary disease, asthma, hypertension, congestive heart failure, chronic kidney disease, or any type of diabetes mellitus as ascertained by CDW data using algorithms from the VA COVID-19 Shared Data Resource [20]. Access to as well as basic literacy with electronic platforms (ie, video calling and email) was an inclusion criterion. Patients with dementia were excluded given the coordinated physical effort required to use the device, and patients with pacemakers and implantable cardioverter-defibrillators were excluded due to the potential of these devices to interfere with device readings. Payments of US $85 were provided monthly for regular use of the device and completion of monthly web-based surveys. At the end of the study, return of the device qualified participants for up to US $175 additional, for a potential total payment of US $685.

The device developer in partnership with the Biomedical Advanced Research and Development Authority (BARDA), part of the US Department of Health and Human Services (HHS), funded the study, supplied devices to enrolled participants, supported training and device use, collected device data on a cloud-based platform, and developed the algorithm. The role of the VA team was to enroll patients, train them on the device, support device use throughout the study, collect monthly web-based survey data, and provide baseline as well as study period data on health conditions, medications, and health care usage. Health data was obtained from the VA CDW and via chart review of records obtained from non-VA care sources.

Recruitment

The study aimed to enroll 200 participants. All procedures were conducted remotely, with the principal investigator, the sponsor, and 5 study staff members spread across 5 different locations, sharing files via the VA-approved version of Box [21]. VA Box allows for subfolders that can be accessed by non-VA study staff outside the firewall. We used the CDW to identify potentially eligible Veterans across the United States and its territories. Every 3 weeks, up to 1000 potentially eligible patients were mailed a recruitment letter that summarized the study and provided the research team’s contact information.

Screening, Enrollment, and Device Training

Patients who called in response to the recruitment letter were provided basic information about the study and screened by telephone. If they were determined to be eligible, we scheduled a 30- to 45-minute video–based enrollment visit to thoroughly explain all study procedures and obtain informed consent. At the time of the enrollment visit, after study procedures were explained thoroughly, participants were emailed the informed consent and HIPAA (Health Insurance Portability and Accountability Act) authorization forms via the VA-approved version of DocuSign (Docusign, Inc). Patients had time to review those documents before choosing to electronically sign, which action automatically forwarded the forms to the study staff. The enrollment visit typically occurred on a day after the phone screening was completed. However, the visits could occur on the same day if the Veteran and study staff had time.

Following the study staff’s receipt of the signed informed consent, patients were considered enrolled. Enrolled patients were then called to schedule device training. Training was conducted over a 45- to 60-minute video visit after enrolled patients received shipment of their device directly from the sponsor. For all video visits, participants received an invitation to their email with the link to participate in the video visit using the VA-approved version of WebEx (Cisco), a videoconferencing platform.

Demographic information was available via the CDW for all potentially eligible patients. At the time of screening calls, patients volunteered their reasons for declining participation in their own words. Not all provided a reason for declining. Similarly, most but not all patients who withdrew provided a reason for doing so.

Analysis of Recruitment and Enrollment Data

Patient characteristics

Demographic characteristics used in the analysis were all from the CDW and included age, sex, marriage status, rurality, and race. VA priority group status was also included as a demographic characteristic. Priority groups are used to determine benefit eligibility. The highest priority group is Priority 1, 50%-100% disabled by a service-connected condition. Groups 2-4 capture lesser degrees of disability, as well as Purple Heart and Medal of Honor awardees, and former Prisoners of War. Priority group 5 is eligible on the basis of low income, and groups 6-8 have copays. Health conditions are identified as per the process for eligibility screening.

Analysis

Patient flow was described from mailing of the study invitation through study completion, or withdrawn, or dropped out status. Study completion was defined by patients having completed the final web-based survey. The geographic distribution of enrolled patients was mapped by county. Demographic characteristics and health conditions of nonenrolled versus enrolled patients, and of completers versus noncompleters, were compared using t tests and chi-square tests as appropriate. Multivariate logistic regression was used to evaluate variables associated with enrolling versus nonenrolling, and completing versus noncompleting. For both comparisons, all variables were entered simultaneously in a multivariable binary logistic regression predicting enrollment or completion status. Reasons for noncompletion were summed. Since reasons for declining or withdrawing were not always offered, and sometimes more than one reason was given, these data were presented descriptively without statistical analysis. We used SPSS (version 29.0.2; IBM Corp).

Ethical Considerations

The protocol was approved by the Central institutional review board of the Veterans Healthcare Administration on December 21, 2022 (1720625-1). Since this was a multisite study, individual institutional review boards at the VA Bedford Healthcare System and the VA Providence Healthcare System also reviewed and approved the study. Informed consent was obtained from all eligible participants, after which HIPAA authorization was obtained. No study-related tasks were performed before obtaining both consent and HIPAA authorization. The informed consent form was sent to patients via DocuSign and reviewed with them on a WebEx video call. Participants receive the executed informed consent form via email. Participants received US$ 85 paid by check at the end of months 1-5, and upon completion of study tasks (survey and device return) in month 6, participants received US $175. The total potential compensation was US $600.

Results

Overview

Of the 7714 who were mailed a study invitation, 560 were screened (Figure 1). Of the screened patients, 203 were enrolled (2.6% enrollment yield) and 166 completed the study (82% retention rate). Enrolled patients were broadly distributed across the United States (Figure 2).

‎

‎

Comparison of Patients Who Were Enrolled Versus Those Not Enrolled in the Study

Among those screened, characteristics of enrolled versus nonenrolled patients were very similar (Table 1). In bivariate analyses, enrolled patients were more likely to be married (55.7%) than unenrolled patients (40.3%; P<.001), and were more likely to have asthma (16.3%) than unenrolled patients (8.1%; P<.001). These differences were also significant in multivariable analyses controlling for all patient characteristics (Table 1). In addition, in multivariable analyses only, the odds of enrolling (vs not enrolling) were significantly higher among patients listed as Priority group 5 (eligible for VA benefits on the basis of low income; odds ratio 1.60, P=.02). No other differences were significant in bivariate or multivariable analyses.

Comparison of Patients Who Completed Versus Those Who Did Not Complete the Study

Among those enrolled, completers and noncompleters were also similar (Table 2). Those who completed the study were significantly more likely to have hypertension (92.2%) than those who did not complete the study (81.1%) in bivariate analyses (P=.04), though this difference was not significant in multivariable analyses controlling for all patient characteristics. No other differences were significant in bivariate or multivariable analyses (Table 2).

Reasons for Noncompletion

The three most common reasons for noncompletion were that participants were unable to be contacted for study tasks (37.8% of noncompleters), difficulty with device functionality (21.6%), and health challenges making it difficult to adhere to the study protocol (13.5%). The 34 participants who dropped out or withdrew did so at a mean of 59.5 (SD 53.6) days and a median of 31 (IQR 30-72) days (Table 3).

Discussion

Principal Findings

Remote digital health studies are increasingly common, but challenges exist to adequate enrollment. While remote studies may support more equitable enrollment in terms of including patients who live at a distance from study sites, digital health studies using RPM or mobile apps often require access to broadband; sufficient digital health literacy to participate in online information sharing, consenting, training, and outcomes reporting; and ability to use the digital technology correctly. One approach to successful enrollment in the context of these challenges is to draw from a very large pool of potentially eligible patients. In this paper, we demonstrate how we accomplished this in the VA health system. Identification of potential participants using the VA national data warehouse, followed by a large-scale mailing effort, was a successful approach to meeting enrollment targets for our remote digital health study. In addition, we found minimal evidence of selection bias or attrition bias using our approach.

The value of access to a large cohort for recruitment for such studies is underscored by our overall recruitment yield of 2.6%. Low enrollment yield does raise concern for selection bias in the study. However, using available data to explore this possibility, we found minimal differences in the characteristics of those who enrolled and those who did not. Those enrolled were more likely to be married than those who did not enroll. One reason could be that, this study required the use of internet-based video and consent, and some patients required caregiver support to remember when and how to use the device. It is possible that those without social support from a spouse, some potential participants, were not able to manage the requirements of enrollment. We also found that low-income Veterans were represented more among enrollees than nonenrollees, likely reflecting the greater relative value of the participant payments to those in this group. It is possible that there were unmeasured differences in patients who enrolled and did not enroll, for example, related to digital health literacy or previous experience with the VA or with research. While these are concerns highly relevant to the equitable conduct of digital health studies in particular, such bias would not negatively impact the results of this particular study, which focused on associations between clinical parameters. Studies with different outcome measures may need to consider the potential for selection bias along other dimensions than those measured in this project.

Enrolled participants were widely dispersed across the United States. Interestingly, we did not enroll any participants from several states in the Midwest and Northeast, or Alaska, or Hawaii. This may be because these states tend to be more sparsely populated, so there were fewer potential participants in those states to start with. Others recruiting from VA may choose to oversample to increase the likelihood of recruitment from across states, or along any dimension that is of particular interest for the study, such as demographic characteristics, specific health conditions, or certain social determinants of health that are available in the national data.

The literature on demographic predictors of enrollment in digital health studies is relatively nascent [22], with inconsistent findings across studies [23]. For example, predictors of interest in one digital health study included clinical characteristics relevant to the study topic, but not age or sex [24]. Another study found that the odds of consent were lower for older and women patients [25]. Both studies were related to heart health.

Of those who enrolled, 82% completed the study. This is quite good, given that remote digital health studies have a median study completion of only 48% (IQR 35%-76%) [3]. Daniore et al [3], describe 3 criteria that impact the retention of participants in remote digital health studies. These are participant task complexity (frequency, complexity, and duration), participant motivation, including incentives and nudges (intrinsic and extrinsic), and scientific requirements (study design and sample size). We achieved a high completion rate despite a complex set of participant tasks: over a 6-month period, participants needed to attend a web-based training session that could last up to an hour, use the device twice a day, confirm data was uploaded each time, call and work with study staff for any needed device troubleshooting, and complete monthly web-based surveys. A counter to high task complexity was likely motivation. As noted above, participants expressed appreciation for the payments. In terms of intrinsic motivation, several participants expressed the desire to contribute to science, especially through research related to Veteran health and improving COVID-19 outcomes. This study did not provide clinical data from the device back to patients; doing so may have further strengthened extrinsic motivation to remain in the study. Interestingly, Daniore et al [3] found that interventional studies tend to have higher completion rates (median 55%, IQR 38%-79%) than do observational studies (median 43%, IQR 22%-60%). Although this study required that patients use the RPM device, we did not intervene in terms of changing any aspect of their health care. Nonetheless, we achieved a completion rate well above the median for either type of study.

It is helpful to consider ahead of time whether particular demographic groups may be more likely to drop out or withdraw from a study, so that the research team can develop plans to monitor and address differential retention. Other studies have found that lower age [5,26] and male as compared with female sex [26,27] predict dropout from digital health studies, which we did not observe in our study. Thoughtful and judicious use of digital technology for study tasks [28] may help boost retention across demographic groups. For digital health studies, as for traditional studies, sufficient numbers of specialized, persistent, and highly trained trial staff [28] with the ability to innovate and adapt strategies are particularly important [28,29], especially as the retention pattern specific to a given study may differ depending on how demographic characteristics interact with the 3 criteria of Daniore et al [3] described above.

Limitations

Our study has limitations. In comparing characteristics of those who enrolled to those who did not, we were limited to those characteristics available in the CDW. Therefore, we were not able to examine the degree to which important factors like functional status, caregiver support, education, or digital literacy may have impacted enrollment. Those who had poor internet connectivity may have been more likely to not enroll or drop out, and those without internet connectivity could not participate in the study at all. We did not ask specifically why participants who did not complete the study dropped out or withdrew; we captured reasons that were offered spontaneously. It is thus possible that more detailed questioning would have revealed a richer understanding of reasons for noncompletion. For example, while several participants cited problems with the device as reasons for dropping out, problems such as slow data upload may actually have been related to cellular network capability in the participants’ location.

Conclusions

Recruiting patients through the VA for remote digital health studies, using the CDW to identify potentially eligible patients, can help ensure that clinical trials reach targets for enrollment and completion. Collaboration with VA-affiliated researchers is required, which can be initiated either through direct contact with researchers, through the National Association of Veterans’ Research and Education Foundations [30], or through the VA Partnered Research Program [31]. Once a research collaboration is identified, local VA-affiliated nonprofit corporations provide a straightforward mechanism for collaboration with foundations, industry partners in pharmaceuticals and devices, universities, government agencies, and donors to advance research in VA.