Smartphones offer an unprecedented anytime-anywhere medium for researchers to engage with and assess health-related behaviors in large populations in real-world settings [1,2]. As of 2020, the rate of smartphone ownership in the United States has reached over 80% [3]. The large-scale, high-frequency daily use of such devices coupled with increasingly multimodal onboard sensing capabilities offers an effective approach for conducting large-scale health research [4,5]. The adoption of digital health tools to develop and deploy digitally augmented trials has been rising steadily since the first fully remote decentralized trial in 2011 [6-8]. Recent studies have shown the benefits of remote monitoring using smartphones for assessing real-world behavior [9,10], for managing chronic pain [11], cancer care [12], diabetes [13], Parkinson symptom severity [14], and cardiovascular health [15] and for the delivery of remote interventions [16]. The COVID-19 pandemic has further accelerated this growth, enabling over 220 digitally augmented trials in 2021 alone [17,18].

Using smartphones for health research can also help achieve operational efficiency by relying less on traditional research facilities or intermediaries for data collection, which require in-person contact between the study participants and the research team [6,19,20]. Researchers can communicate asynchronously and synchronously with participants and assess their health by actively and passively collecting individualized real-world data [4,21,22]. Active data are defined as data generated through effortful participation (eg, completing a survey). In contrast, passive data are collected without direct input from participants (eg, the number of daily steps estimated through onboard sensors) [23]. Such scalable remote observational models [6,20] could help investigators to understand people’s day-to-day experiences of living with a health condition [4] and the relationship between individualized real-world behavior and health outcomes [22].

However, despite the promise of decentralized health research, several challenges related to the representation and inclusiveness of recruitment and the retention of target populations have surfaced [21,24,25], resulting in sparse, unbalanced, and nonrepresentative real-world data collection [21]. Typically, decentralized studies recruit from various web-based sources such as social media (Facebook [26] and Reddit [27]), crowdsourced platforms (Prolific [28]; Amazon Mechanical Turk, MTurk [29]; Centiment [30]; and CloudResearch [31]), and partnerships with patient registries or advocacy groups [32,33]. Although these recruitment channels have shown the potential to reach and recruit large populations remotely [34-36], the long-term and uniform retention of remote participants has been challenging. Recent findings show that retention rates vary from 1% to 50% [24], with monetary incentives being able to significantly improve long-term retention [10]. With large-scale open recruitment approaches, including the use of financial incentives, the risk of enrolling gamers or malicious actors increases [37].

With large studies using multiple web-based sources to reach and recruit participants remotely, there is a need to assess the impact that such strategies have on the characteristics of the enrolled cohorts and their retention in the studies. In addition, further research is needed to understand how variations in study participation incentives (eg, time and frequency of payments) and differences between Android and iOS operating systems [38] affect long-term data collection in decentralized studies.

To investigate some of these challenges in collecting health data through smartphones in real-world settings, we examined the recruitment, retention, and passive data-sharing patterns of more than 10,000 participants in a large, decentralized research study. Specifically, we evaluated the following three key questions: (1) Does combining different recruitment and incentive distribution approaches lead to a heterogeneous cohort with varying characteristics? (2) Can the participant retention and uniformity of data collection in remote studies be affected by cohort heterogeneity? (3) What are the factors that can affect passive data collection in real-world settings?

This study was approved and monitored by the Institutional Review Board of the University of Washington (STUDY00004997) and the Department of Defense Human Research Protection Office; the approval for the study was granted on February 11, 2020.

The participants in the Warfighter Analytics Using Smartphones for Health (WASH) study were volunteers who lived in the United States and agreed to engage in a 12-week smartphone-based study. The primary goal of the study was to understand the real-world factors that could help with the early prediction of cold, influenza, and the impact of traumatic brain injury on daily functioning. The eligible participants were individuals aged ≥19 years, English speakers, residents of the United States, owners, and primary users of iPhone or Android smartphones with internet access. The potential participants were required to complete an eligibility screener before consenting, and those who did not meet the inclusion criteria were not permitted to complete subsequent procedures.

Participant recruitment started on March 15, 2020, with rolling enrollment until January 4, 2022. The participants for the study were recruited in 2 phases, using different recruitment and incentive distribution approaches (Figure 1). Participants could receive up to US $90 for completing the baseline survey and 12 weeks of follow-up surveys. The final participation incentive was determined on the basis of the number of complete surveys. Participants were not informed about the financial breakdown during the consent process; however, additional details regarding when they would receive compensation and how much compensation they would receive were provided upon request.

Assessments were divided into 1 longer baseline survey and brief daily assessments. The baseline survey assessing participants’ health history, mood, physical activity, and phone use was administered 24 hours after consent was obtained in phase 1 of the study. However, in phase 2, the baseline health survey was administered immediately after consent was obtained. In both phases, the participants were administered the same scheduled health-related surveys twice daily for 12 weeks. The survey asked participants about their mood, physical activity, and phone use.

Sensor-based data were collected actively and passively from participants through the study app. Participants completed performance outcome measures [47] such as standing and walking tests and sharing voice recordings. The participants were also asked to allow the study app to collect passive data from their smartphones. Passive data included, but were not limited to, device movement and orientation; actual and relative location; the device’s status (eg, active use or connected to a data network); and local environmental information such as ambient light, temperature, and humidity. Participants had the option to not share the passive data and remain in the study. However, all participants enrolled in the study on or after August 28, 2020 (before the start of phase 2), were required to allow the study app to passively collect the accelerometer and gyroscope sensor data from their smartphones.

As of the data freeze date (February 9, 2022), the study recruited 10,768 participants. Most participants (6494/10,768, 60.3%) were recruited during phase 1, and the remaining (4274/10,768, 39.69%) were recruited during phase 2 (see the Methods section). A significant proportion of participants, most notably in phase 1, did not complete the baseline survey (phase 1: 3135/6494, 48.27%, vs phase 2: 918/4274, 21.47%). Figure 2 compares the recruitment rate of the study with the baseline survey submission rate over time. The number of baseline surveys completed generally was in line with the number of participants recruited during the study period. Recruitment peaked in mid-April and mid-May 2020 for phase 1 and in mid-January and early March 2021 for phase 2. However, during phase 1, between May and July 2020, the number of baseline surveys completed was significantly lower than the number of recruited participants, which explains the large proportion of missing baseline data in phase 1. We further assessed the effect of missing baseline surveys on participant engagement in the study (see the Retention Analysis section). Additional statistics on missingness and invalid data entries in the baseline surveys are summarized in Multimedia Appendix 2.

Most of the participants who completed the baseline sociodemographic survey were female (3817/6574, 58.06%). The median age was 30 (IQR 24-40) years, with a larger proportion of participants being aged 19 to 29 years (2949/6267, 47.05%). The non-Hispanic White population was the largest (3938/6677, 58.97%), followed by the Asian (931/6677, 13.94%) and Hispanic or Latino (783/6677, 11.72%) populations. Most participants were iOS users (5883/10,583, 55.58%). Table 1 summarizes the sociodemographic characteristics of the overall cohort.

The population recruited in phase 2 had a higher proportion of younger adults (aged 19 to 29 years; 1685/3194, 52.75%) and a lower proportion of older adults (aged ≥60 years; 94/3194, 2.94%) than that recruited in phase 1 (P<.001; Table 1). A higher proportion of Black or African American participants were recruited in phase 2 (phase 1: 267/3342, 7.98%; phase 2: 456/3339, 13.65%; P<.001). Notably, a larger proportion of participants (1942/3308, 58.71%) with lower levels of annual income (≤US $49,999) were recruited in phase 2 than in phase 1 (1062/2483, 42.77%; P<.001). The proportion of Android versus iOS users also varied across the recruitment phases. A significantly higher proportion of iOS users (P<.001) were recruited in phase 1 (3958/5883, 67.27%) than in phase 2 (1925/5883, 32.72%). Multimedia Appendix 3 further compares the sociodemographic characteristics of Android and iOS users across the 2 recruitment phases.

The number of data modalities that were passively collected by the study app varied across the Android (31 data modalities) and iOS (14 data modalities) operating systems. The variation in the number of passive data modalities available across Android and iOS devices is because of the available onboard sensors and data collection restrictions across the two operating systems [38]. Of the 31 Android passive data streams, 18 (58%) were shared by at least 50% of the Android users across the 2 study phases (Table 2). In contrast, 86% (12/14) of the distinct passive data streams were shared by at least 50% of the participants using iOS devices. Multimedia Appendix 4 summarizes data-sharing proportions per sensor stratified across Android and iOS devices. None of the participants with iOS devices shared passive data from the camera or barometer. Similarly, participants with Android devices did not share any data from some passive data streams, including temperature, camera, and humidity (Multimedia Appendix 4). This variation in passive data sharing could also be linked to the heterogeneity and nonavailability of specific sensors in some devices. It is worth noting that phase 2 of the study required participants to share accelerometer and gyroscope data passively. However, a small yet notable proportion of the cohort recruited in phase 2 did not share accelerometer (503/4089, 12.31%) and gyroscope (856/4089, 20.89%) data.

In addition, across the 8 passive data streams that were common between Android and iOS devices, the participants’ passive data sharing was linked to sociodemographic characteristics and device type. In total, 3 data-sharing patterns of participants sharing at least 2 (25%), 4 (50%), or 6 (75%) of the total 8 passive data streams were tested. Across all 3 data-sharing patterns, Black or African American participants were found to be statistically significantly less likely to share passive sensor data than non-Hispanic White participants (odds ratio [OR] 0.44-0.49, 95% CI 0.35-0.61; P<.001). Furthermore, participants sharing ≥75% (6/8) of the passive data streams were more likely to be iOS device users (OR 1.98, 95% CI 1.71-2.28; P<.001) and earning more than US $25,000 per year (OR 1.27-1.55, 95% CI 1.06-1.93; P<.001). Multimedia Appendix 5 provides further details on the association between participants’ sociodemographic characteristics and passive data sharing.

The median retention time of the overall cohort was 38 days, within the 84-day study observation period. No meaningful difference was observed in cohort retention across the active (median 37 days) and passive (median 38 days) data streams (Multimedia Appendix 6). The sensitivity analysis of participant retention also showed no significant difference in median survival across the active and passive data streams (Multimedia Appendix 7). Consequently, all subsequent retention analyses were conducted by combining the active and passive data streams.

Notable differences in retention were observed across the population recruited between phases 1 and 2. Participants recruited in phase 2 had a significantly higher median retention (+14 days) than those recruited in phase 1 (phase 1: median 37 days; phase 2: median 51 days; P<.001; Figures 3A and 3B). Older participants (≥60 years), recruited in both phases, remained engaged in the study for the longest duration (phase 1 and phase 2 median retention 38 days and 83 days, respectively) relative to the younger cohort (Figures 3E and 3F).

It is worth noting that certain characteristics, including socioeconomic factors, distinctly impacted participant retention across the cohorts recruited in phases 1 and 2 (Table 2). Participants who completed the baseline survey administered immediately after enrollment in phase 2 were retained for a significantly longer period (with median values of baseline survey: yes 59 days vs no 19 days in phase 2; Figure 3D). However, the same trend was not observed for participants recruited from social media platforms in phase 1. Similarly, in phase 1, the non-Hispanic White population was retained in the study for a significantly longer time (median 37 days) than the Hispanic or Latino population (median 20 days; Figure 3G). No meaningful differences were observed among non-Hispanic White and Hispanic, Latino, or Spanish populations in phase 2 (Figure 3H). Education level mainly impacted retention in phase 1. Participants reporting high school or lower education levels had the shortest retention (median 5 days) than other participants (median ≥36 days) in phase 1. Such a large difference in retention because of educational level was not seen in the population recruited in phase 2 from crowdsourcing platforms (Figures 3I-3K). Participants’ self-reported income was also found to be significantly associated with retention in phase 1 only. Participants with incomes of <US $49,999 were retained longer than participants earning >US $100,000 (phase 1: US $49,999 vs US $100,000 median retention 34 days vs 24 days, respectively; P<.001; Figures 3I and 3J). We also noticed a dramatic difference in median participant retention between Android and iOS users enrolled in phase 1 (iOS 37 days and Android 22 days; P<.001). Table 2 and Multimedia Appendix 8 provide additional results and details on the survival analysis.

Our results from the analysis of over 330,000 days of engagement data collected from over 10,000 participants in real-world settings showed that combining different recruitment and incentive distribution approaches can yield heterogeneous cohorts. To the best of our knowledge, this is one of the first studies to empirically assess real-world differences in participants’ sharing of multimodal passive data collected from iOS and Android devices using a bring your own device (BYOD) approach.

Overall, there were 5 key learnings. First, recruiting participants using different media, for example, digital advertisements on social media and web-based newspapers or crowdsourcing platforms, could result in heterogeneous subcohorts with varying characteristics. Second, participant engagement could vary significantly based on the recruitment source (eg, social media vs crowdsourced platforms) and incentive distribution approaches. Third, passive data collection could be substantially affected by technical variations in Android and iOS devices and the sociodemographic demographics of the cohort. Fourth, there is a greater likelihood of participants completing baseline health surveys if they are administered immediately after consent or enrollment. Fifth, monitoring patterns in real-world data collection at the study level could reveal technical glitches that could help guide contextual data filtering and cohort selection, leading to more reliable evidence generation. We now contextualize our principal findings to inform strategies to recruit, retain, and monitor trends in remote data collection to help collect real-world health data in a representative and equitable manner.

The analysis of participant recruitment and retention data from the WASH study should be interpreted within the context of certain limitations. First, large-scale, fully remote data collection started close to the declaration of the COVID-19 pandemic in the United States, which is known to have changed our behavior and interaction with technology and devices [74]. Indeed, Inverso et al [75] showed higher engagement rates during the COVID-19 pandemic because of an increased reliance on technology during the lockdown. The WASH study began recruitment on March 15, 2020, shortly after the World Health Organization declared COVID-19 a pandemic on March 11, 2020. Therefore, we did not have pre-post pandemic data to account for the potential impact of the pandemic on participant engagement with technology and devices. Second, the original purpose of the WASH study was to use the study app to detect cold and influenza symptoms. Thus, participants were not randomized among different recruitment platforms, incentive distribution frequency, and timing of baseline surveys that varied between phases 1 and 2 of the study. Consequently, our findings are not causal or linked to the impact of 1 factor on participant recruitment and retention between phases. For example, this analysis compares the population characteristics of those recruited from web-based crowdsourcing platforms (phase 2) compared with participants enrolling based on social media and local advertisements (open enrollment phase 1) as a whole. We were not able to explore within-phase recruitment differences; that is, between those recruited from social media versus those recruited from local news media advertisements. This is mainly because of the limited information available in the study data, which does not allow for such differences to be investigated. Further research studies using a randomized design are needed to investigate the impact of individual changes in recruitment and retention strategies and their effectiveness for use in decentralized research. Third, we could not control for the participants’ previous experience in crowdsourcing platforms and research tasks, which can be a confounder [41] depending on the nature of the assessment. Future research studies should assess participants’ prior participation in similar or other research studies to assess any differential impact on primary outcomes. Fourth, in phase 1, participant recruitment via press releases was centered in the Greater Seattle area, which may not be representative of the population of the United States. In addition, because of the high proportion of missingness in the baseline geolocation data, we could not determine the geospatial representativeness of the cohort. Future studies should prioritize collecting high-level geolocation data, such as the state, city, or zip code, to help assess the geospatial representativeness of the study cohort. Fifth, we could not account for all the underlying within-study differences in the outcomes; for instance, the probable technical glitches concerning the steep drop in participant engagement at the participant level on day 36 in phase 1 and fluctuations in sensor data collection or management in phase 2 (Multimedia Appendix 9). These technical issues could have impacted the participants’ willingness to remain engaged and increased the perceived burden of participants who were active in the study at the time of technical glitches. Sixth, despite our filtering out bad actors, some could still have been successfully enrolled by creating multiple accounts or using multiple devices. We suggest that future digital health research studies specifically report and compare the impact of different temporal recruitment and incentive strategies on enrolled cohorts’ characteristics and engagement metrics as well as fraudulent enrollments to allow for future replication and the establishment of a set of guidelines for successful methods of participant recruitment and retention.