Ecological momentary assessment (EMA) refers to a range of methods used for measuring daily life feelings, events, experiences, and behaviors in real-time, real-world settings [1]. These methods include pen-and-paper surveys and diary methods [2,3] and surveys delivered via palmtop computers [4,5], mobile phone SMS text messaging [6,7], and, most recently, smartphone apps [8-10]. In the past decade, smartphone app–based EMA has surged in popularity, coinciding with an exponential growth in smartphone ownership; in the United Kingdom, 87% of the population owned a smartphone in 2020 compared with only 27% in 2010 [11,12]. This presents an opportunity for participants to partake in app-based noncontact EMA, where they can download an app and run EMA schedules from their own smartphone, without the need for in-person interaction with researchers at study invitation, recruitment, initiation, or data collection.

Among its advantages, app-based noncontact EMA allows for the recruitment of large samples, or those drawn from specific or hard-to-reach populations, more easily than in-person EMA. It is convenient and can reduce or remove time and geographic barriers to participation. The participant burden associated with borrowing, becoming familiar with, and carrying around a supplied research smartphone is eliminated. For researchers, equipment costs are reduced, and allowing participants to use their own personal smartphone means that less training is required, thus saving time. When training is provided, it is more feasible to use a blanket approach (eg, distribution of generic written instructions only) with noncontact EMA than in-person EMA. Finally, noncontact EMA presents an alternative when situational factors, such as the COVID-19 restrictions, prevent in-person research.

However, there are also limitations associated with this method, not least that participants must own a smartphone. Furthermore, EMA schedules can be complex and demanding, and therefore require a certain degree of participant investment and a solid understanding among participants of what they are being asked to do, along with technological proficiency with the smartphone and app. This may be difficult to achieve without an in-person initiation. Finally, in allowing participants to run EMA on their own smartphones, researchers have less control over the study, and the wide variety of smartphone devices on the market makes it difficult to provide technical support to participants when required. These factors may have negative implications for the quality and quantity of the EMA data collected.

One may reasonably expect that previous experience of successful participation in EMA studies (in whatever form) would increase the likelihood of being able or willing to complete a subsequent app-based noncontact EMA study, although to the authors’ knowledge, this has never been explored. In contrast, participants who are new to the EMA method may be disadvantaged by the lack of in-person initiation and subsequently struggle with the noncontact EMA protocol. Regarding the potential effects of age, the high technological demand of EMA may prove challenging for older adults who tend to have lower rates of smartphone ownership, use [13], and competency [14]. Notably, Duncan et al [15] found that, among a sample of sexual minority men, older participants were less willing to download the EMA app to their smartphone than younger individuals. Hence, the limitations of app-based noncontact EMA may lower its feasibility for older participants and introduce sampling bias. As app-based noncontact EMA has become necessary under the COVID-19 restrictions, it is essential to understand the feasibility of this method.

To explore the prevalence of published app-based noncontact EMA research and uncover any evidence relating to the abovementioned effects of age and EMA experience, a literature search of the Scopus database was conducted. The search was conducted in April 2020, using the terms ecological AND momentary AND assessment AND smartphone. A total of 404 articles were found, all published from 2011 onward. After removing duplicates and inaccessible or irrelevant articles, 73.3% (296/404) of studies remained. Of the 296 studies, 220 (74.3%) used in-person recruitment or initiation techniques, and the studies described by 47 articles were indeterminable as in-person or noncontact EMA. In total, 29 articles reported noncontact procedures, and all were published since 2015. Of these 29 articles, 3 (10%) were protocol papers and 2 (7%) used SMS text message–based sampling measurements. Therefore, 83% (24/29) of publications described observational app-based noncontact EMA studies, demonstrating that this method has been enjoying minimal (in contrast to in-person EMA) but increasing application in recent years.

A meta-analytic approach was used to determine the characteristics of the 24 studies. The median sample size across the studies was 135 participants (range 17-6675). Among the studies which reported mean age and dichotomous male–female gender distribution of the sample, participants tended to be younger (mean 30.9, SD 8.0 years, based on 21 studies) and mostly women (mean 68.7% of the sample, based on 19 studies). The topics of study were smoking [16,17], alcohol consumption [18-22], mental health [23-29], dietary behavior [30-35], drug use [36,37], physical activity [38], and tinnitus management [39].

All studies used noncontact methods of recruitment, initiation, and EMA data collection, although the specific procedures varied across studies. Advertisement was conducted on the web via circulation of study details on social media [28,39] and within Reddit and other web-based communities [27,29], and offline via mass media [26,28], flyers [30], and networking or word of mouth [18,22,30]. When an existing database of university students was available, email invitations were used to advertise studies and recruit samples [21,35,38]. Recruitment and initiation of participants commonly involved directing participants to a webpage where they could read the study description, provide informed consent, complete baseline measures, and access instructions for downloading and using the EMA app [18,33]. In some studies, written instructions for the EMA phase were supplemented by a telephone call [31] or a prerecorded video demonstration [35], indicating that noncontact initiation is more intensive in some studies than others.

Although the specifics of EMA data collection varied depending on the purpose of the study, all 24 studies required participants to run the app on their own smartphone; some specified that this must be running iOS [19] or Android [24], whereas others allowed both [22]. In most studies, the feasibility of this aspect of the design was unaddressed, although McQuoid et al [37] reported that iOS and Android smartphones collected sensor data at different rates in their EMA study; Schlee et al [39] noted low controllability owing to participants using their own smartphones as a limitation of their study, and Wouters et al [35] concluded that the main cause of participant dropout in their study was related to a technical issue caused by an Android update during data collection. Notably, participant dropout rates were not systematically reported; in 71% (17/24) of the studies, it is implied, but not explicitly stated, that no dropout occurred, whereas some reported dropout rates ranging from 14% to 37% [22,23,32-35,38]. Finally, across the 6 studies that did not exclude participants based on low response rate and reported response rate as a percentage of delivered signal-contingent EMA surveys [18-21,32,39], the mean response rate was 65%.

None of the aforementioned 24 studies directly assessed the feasibility of app-based noncontact EMA in comparison with in-person EMA. To the authors’ knowledge, the only study to date to have done so is that of Carr et al [40], which was not published when we conducted the literature search. In their sample of men who have sex with men (N=100; mean age 27 years), participants were enrolled in the study and were initiated either in-person or via a live videoconferencing session. The measured outcomes included response rate, behavioral reactivity, and consistency and reliability of the EMA responses. The in-person and noncontact groups returned similar response rates, similar levels of behavioral reactivity, and equally consistent and reliable EMA responses. This suggests that app-based noncontact EMA is feasible and equivalent to EMA initiated in-person, at least with younger, male participants who are initiated via videoconferencing. The use of videoconferencing by Carr et al [40] to initiate noncontact participants means that their findings may not be generalizable to studies that use written instructions [15-17]. Indeed, videoconferencing might arguably be classified as in-person for the purposes of instruction and initiation.

From the foregoing, it is apparent that much remains unknown regarding the feasibility of app-based noncontact EMA. Specifically, evidence is lacking regarding the method’s feasibility with older adults and with experienced versus naïve EMA participants. Therefore, the research questions are as follows:

Given the paucity of previous evidence relating to these questions, this study adopted an exploratory approach rather than testing specific hypotheses.

In total, 151 members of our pre-existing participant pool were invited. Recruitment occurred in 2 stages. First, 32.5% (49/151) of people who had participated in an earlier in-person EMA study of listening and daily life fatigue conducted by Burke and Naylor [41] (referred to hereafter as the EMA Fatigue study) were invited. In the study, which ran 12 to 18 months before this study, participants were selected randomly from our participant pool, invited by postal letter, and attended three in-person sessions: an initiation session at baseline, a check-in session midway through the study, and a debriefing and interview session at the end. They used smartphones supplied by the researchers with the EMA app preinstalled (the same app as used in this study) to respond to 6 EMA surveys per day for 2 weeks. In all, 50.3% (76/151) of participants initiated the study, and 44.4% (67/151) completed the study. The participants who initiated but subsequently dropped out of that study were not invited to this study to reduce variation with respect to the level of EMA experience acquired. Of the 67 participants who completed the EMA Fatigue study, 18 (27%) were no longer contactable. Thus, 49 individuals with previous EMA experience of 2 weeks, including in-person initiation, were invited to participate in this study. Of these, 63% (31/49) accepted the invitation and formed the experienced EMA group.

Thereafter, an additional sample of 102 participants was invited. They had not participated in the EMA Fatigue study, although 19.6% (20/102) were selected randomly and invited to participate in that study—an overlap that resulted from sampling from the same participant pool for both studies. Recruitment of this naïve EMA group was conducted on a rolling basis to recruit a group that was similar in size, age, and gender distribution to the experienced EMA group. Initially, 26 invitations were sent, followed by 30 more, followed by 46 more, at which point the naïve group matched the experienced group to an acceptable extent. Of the 39 who accepted the invitation, all but one reported that they had not previously participated in smartphone-based research; one was unsure.

As the aim of recruitment was to recruit as many experienced participants as possible from a finite group of past participants, and subsequently to recruit a similarly sized naïve group, this drove our recruitment approach rather than power analyses, which would have been redundant. Notably, all members of our participant pool were recruited from the National Health Service Audiology in Glasgow, although not all experience hearing loss; issues of tinnitus, hyperacusis, and balance, for example, are also present within our participant pool. The only inclusion criterion was that participants had to own either an Android or iOS smartphone. Invitees’ smartphone ownership status was unknown before recruitment.

Ethical approval was received from the West of Scotland Research Ethics Committee (18/WS/0007) and the National Health Service Research and Development (GN18EN094). This study was not preregistered.

Data were collected from June to August 2020. Email invitations included a participant information sheet and a link to the web-based consent form and baseline questionnaire set. The participants who provided consent and completed the questionnaires were sent a follow-up email containing beginning the study and finishing the study instruction guides. These participants were asked to download the EMA app to their smartphone and run our EMA survey sequence, which was launched when participants completed a start-up survey. At this point, they received an email confirming their enrollment in the study, start and end dates of the EMA phase, and the date and time to expect a telephone exit interview. Participants who provided consent and completed the baseline questionnaire set, but did not run the EMA sequence, were emailed a link to a follow-up web-based survey that asked them why they had failed to complete the study.

The EMA sequence consisted of 7 full days of smartphone surveys. Each day, 5 daytime surveys were received randomly between 9 AM and 8 PM, with at least an hour between the surveys. Participants had up to 30 minutes to respond to the daytime surveys before they expired. One evening survey was received at 9 PM each night, and participants had up to 3 hours to respond. On the final day of the EMA data collection, participants received a final survey at 8:30 PM, instead of an evening survey. This informed participants that they had finished the EMA sequence and advised them to consult the finishing the study instruction guide, which instructed them to check their data upload status within the app, upload data if necessary, and uninstall the app. The final survey did not expire, so the response window was unlimited.

Telephone exit interviews were conducted with all participants who ran the EMA sequence. Participants were made aware of their invitation email that they would be compensated a fixed amount, a £20 (US $27) Amazon e-voucher, to complete the study. They were not paid on a per-survey basis; therefore, there was no monetary incentive to complete the EMA surveys. Participants who completed the baseline survey, 1-week EMA sequence, and telephone interviews received full compensation. It was decided after the data collection, to compensate participants who only completed the baseline survey with a £10 (US $13.5) Amazon e-voucher.

The web-based consent forms and baseline questionnaires were created and administered using the JISC Online Surveys [42]. The LifeData EMA platform [43] and its corresponding smartphone app, RealLife Exp [44], were used to create and administer smartphone EMA surveys. All the participants used their own smartphones. A participant information sheet explaining the study in detail, a beginning the study instruction guide, explaining how to install and use the app and respond to EMA surveys, and a finishing the study instruction guide, containing information about data upload and app uninstallation, were supplied as PDF documents.

Figure 1 illustrates the flow of participants and the number of participants who reached various stages of the study.

The phone operating system was related to response rate (marker 4); in comparison with Android users, participants using iOS returned a higher response rate (mean 74.8%, SD 20.3% vs mean 48.5%, SD 31.4%; t₄₅=−3.28; P=.002). Among iOS users, 25% (5/20) reported not being alerted to surveys (marker 5) compared with 41% (11/27) of Android users; however, this difference was not statistically significant (χ²₁=1.3; P=.26). All 3 participants who sought assistance from the research team (marker 6a) were Android users. Of the 6 who reported asking a family member or friend for help (marker 6b), 4 (67%) were Android users and 2 (33%) were iOS users. The effect of the phone operating system on markers 6a and 6b was not assessed because of the lack of variability of responses on these outcomes.

Although both older age and using an Android operating system were related to reporting issues with survey alerts, the mean age of Android users in this study (mean 65.9, SD 7.9 years) was not significantly different from that of iOS users (mean 65.6, SD 9.4 years; t₄₅=0.12; P=.91).

The 26 experienced EMA participants who completed the study were asked if they preferred using their own smartphone, as in this study, or using a supplied smartphone, as in the EMA Fatigue study. Of these 26 participants, 22 (85%) preferred using their own phone, whereas 4 (15%) preferred using the supplied research phone and/or preferred the face-to-face method in general.

This study aimed to assess the feasibility of noncontact EMA. To the authors’ knowledge, this is only the second study to do so, after Carr et al [40], and is the first to examine the effects of previous EMA experience, age, and phone operating system on the feasibility of this method. Although the findings suggest that previous in-person EMA experience, younger age, and iOS use may be advantageous, the observed effects were specific to certain markers of feasibility and not systematic across all possible indicators. It is therefore suggested that app-based noncontact EMA is feasible to run with naïve and older EMA participants.

Perhaps the most compelling finding from this study is that naïve participants were less likely to enroll in and complete the study than their experienced counterparts. However, confounding effects could not be ruled out. Specifically, experienced EMA participants had previously attended more in-person laboratory appointments than naïve participants, which may indicate a stronger degree of existing relationship with the researcher or greater willingness to participate in research, among these participants. Future research would benefit from the measurement and tight control of these factors. The observed higher rate of attrition among naïve participants between the baseline survey and EMA phase is concerning. This may indicate technological difficulties in downloading the app or a lack of understanding of the study demands. A low response to the follow-up survey among dropouts means that little information has been gathered regarding the reasons for attrition; therefore, the feasibility of app-based noncontact EMA in this respect remains unclear. Notably, age was unrelated to both enrollment and completion rates, suggesting that the feasibility of this method is not sensitive to age (within the range studied here; 45-78 years).

Turning to the responses to individual survey alerts, it is notable that the response rate was unrelated to both previous EMA experience and age. At 61.4%, the response rate in this study corresponds closely to the 65% mean of response rates across other noncontact EMA studies [18-21,32,39]. We may have expected a higher response rate, given that this study was conducted at a time when many people were confined to their homes and unable to work, travel, or attend social gatherings owing to the COVID-19 pandemic. Hence, some of the main reasons for nonresponse to EMA surveys, including being in a noisy environment and not hearing the alert, or being in a social situation where it would be inappropriate to respond [41,49], were less likely to occur. However, our participants were not paid per EMA survey completion, as in some other EMA studies [27,29], and it is also possible that participants’ motivation to respond about their social interactions was reduced when social activity was restricted. Overall, this study yielded acceptable response rates.

Finally, a sizable number of participants reported that they were not alerted to some or all EMA survey alerts, and their response rates were unsurprisingly lower than those returned by unaffected participants. Whether participants truly experienced a technical issue, or were just unaware of alerts, cannot be determined by the data. However, older participants were disproportionately affected by this issue. These findings undermine the feasibility of app-based noncontact EMA and suggest that supplying participants with a preprogramed smartphone (which would typically involve an in-person initiation session) yields better results in terms of data quantity. However, participants in this study who had experience using both their own smartphone and a supplied research smartphone for EMA overwhelmingly preferred using their own. Future app-based EMA research (both in-person and noncontact) should consider giving participants the option to use either their own smartphone or a supplied smartphone where possible [50], and the chosen app should be tested extensively with different phone makes, models, and operating systems.

In addition to caveats mentioned above with respect to specific results, there are several more wide-ranging factors which may limit the reliability and generalizability of this study’s results.

First, all participants had working email accounts, possibly indicating some level of technological competence. In support of this, the mean TRI 2.0 score obtained by this sample was slightly higher than that reported by the Parasuraman and Colby [48] representative sample of US-based adults, reported in the paper in which they introduced the TRI 2.0 (3.33 units vs 3.0 units on a 1-5 Likert scale). Moreover, all participants were recruited from our participant pool, meaning that they had attended at least one in-person appointment at our facility and therefore had some degree of existing relationship with the researchers. This is also a strength of the study, as it indicates some level of homogeneity of participant background, all having come from Glasgow National Health Service Audiology. However, these factors limit the generalizability of our findings. In addition, in the experienced group, the decision not to invite participants who had dropped out of the EMA Fatigue study to recruit a genuinely experienced EMA group in this study may have led to the recruitment of an unrepresentatively motivated or conscientious group.

Limited variation and lack of data pertaining to some of the markers of feasibility in this study suggest that they are not useful outcomes. Furthermore, despite attempts to explore the reasons for noncompletion of the study by sending a follow-up survey to the dropouts, only few participants responded. Even less is known about why individuals declined to participate in this study in the first place. Crucially, the smartphone ownership status of this group is not known, a key factor in terms of feasibility.

Finally, our method was noncontact insofar as participants did not attend in-person laboratory sessions; however, they did receive detailed instructions to guide them through the study and were contacted several times by email during recruitment, initiation, and enrollment, as detailed in the Procedure section. Consequently, the findings are most relevant to physical noncontact, but nonetheless interactive, EMA research.

This study assessed the feasibility of app-based noncontact EMA as a function of past EMA experience and age. Experienced EMA participants were more likely to enroll in and complete the study than naïve participants, whereas age was unrelated to both enrollment and completion rates. The response rate was acceptable and unrelated to both experience and age, but Android users returned markedly lower response rates than iOS users. Although a sizable number of (mostly older) participants reported in exit interviews that they were not always alerted to surveys, very few informed or sought assistance from the researchers during data collection. In summary, app-based noncontact EMA is feasible, although consideration should be given to how to increase enrollment and completion rates, especially when recruiting participants who are new to EMA.