This paper is in the following e-collection/theme issue:

Development and Evaluation of Research Methods, Instruments and Tools (439) Methods and New Tools in Mental Health Research (428) Electronic/Mobile Data Capture, Internet-based Survey & Research Methodology (792) Methods and Feasibility Studies (369) Ecological Momentary Assessment (EMA) (297) mHealth for Data Collection and Research (917) Formative Evaluation of Digital Health Interventions (2226)

Published on in Vol 7 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/42916, first published .
Feasibility of Using Research Electronic Data Capture (REDCap) to Collect Daily Experiences of Parent-Child Dyads: Ecological Momentary Assessment Study

Feasibility of Using Research Electronic Data Capture (REDCap) to Collect Daily Experiences of Parent-Child Dyads: Ecological Momentary Assessment Study

Feasibility of Using Research Electronic Data Capture (REDCap) to Collect Daily Experiences of Parent-Child Dyads: Ecological Momentary Assessment Study

Authors of this article:

Yola El Dahr1 Author Orcid Image ;   Florence Perquier1 Author Orcid Image ;   Madison Moloney1 Author Orcid Image ;   Guyyunge Woo1, 2 Author Orcid Image ;   Roksana Dobrin-De Grace1, 3 Author Orcid Image ;   Daniela Carvalho1, 4 Author Orcid Image ;   Nicole Addario1 Author Orcid Image ;   Emily E Cameron5 Author Orcid Image ;   Leslie E Roos5, 6 Author Orcid Image ;   Peter Szatmari1, 7 Author Orcid Image ;   Madison Aitken1, 7, 8 Author Orcid Image
Article Authors Cited by Tweetations (2) Metrics

    Original Paper

    1Cundill Centre for Child and Youth Depression, Centre for Addiction and Mental Health, Toronto, ON, Canada

    2Faculty of Arts & Science, University of Toronto, Toronto, ON, Canada

    3Department of Psychology, Toronto Metropolitan University, Toronto, ON, Canada

    4Schulich School of Medicine & Dentistry, Western University, London, ON, Canada

    5Department of Psychology, University of Manitoba, Winnipeg, MB, Canada

    6Children’s Hospital Institute of Manitoba, Winnipeg, MB, Canada

    7Department of Psychiatry, University of Toronto, Toronto, ON, Canada

    8Department of Psychology, York University, Toronto, ON, Canada

    Corresponding Author:

    Madison Aitken, PhD

    Cundill Centre for Child and Youth Depression

    Centre for Addiction and Mental Health

    80 Workman Way

    Toronto, ON, M6J 1H4

    Canada

    Phone: 1 416 535 8501 ext 34091

    Email: madison.aitken@camh.ca


    Background: Intensive longitudinal data collection, including ecological momentary assessment (EMA), has the potential to reduce recall biases, collect more ecologically valid data, and increase our understanding of dynamic associations between variables. EMA is typically administered using an application that is downloaded on participants’ devices, which presents cost and privacy concerns that may limit its use. Research Electronic Data Capture (REDCap), a web-based survey application freely available to nonprofit organizations, may allow researchers to overcome these barriers; however, at present, little guidance is available to researchers regarding the setup of EMA in REDCap, especially for those who are new to using REDCap or lack advanced programming expertise.

    Objective: We provide an example of a simplified EMA setup in REDCap. This study aims to demonstrate the feasibility of this approach. We provide information on survey completion and user behavior in a sample of parents and children recruited across Canada.

    Methods: We recruited 66 parents and their children (aged 9-13 years old) from an existing longitudinal cohort study to participate in a study on risk and protective factors for children’s mental health. Parents received survey prompts (morning and evening) by email or SMS text message for 14 days, twice daily. Each survey prompt contained 2 sections, one for parents and one for children to complete.

    Results: The completion rates were good (mean 82%, SD 8%) and significantly higher on weekdays than weekends and in dyads with girls than dyads with boys. Children were available to respond to their own survey questions most of the time (in 1134/1498, 75.7% of surveys submitted). The number of assessments submitted was significantly higher, and response times were significantly faster among participants who selected SMS text message survey notifications compared to email survey notifications. The average response time was 47.0 minutes after the initial survey notification, and the use of reminder messages increased survey completion.

    Conclusions: Our results support the feasibility of using REDCap for EMA studies with parents and children. REDCap also has features that can accommodate EMA studies by recruiting participants across multiple time zones and providing different survey delivery methods. Offering the option of SMS text message survey notifications and reminders may be an important way to increase completion rates and the timeliness of responses. REDCap is a potentially useful tool for researchers wishing to implement EMA in settings in which cost or privacy are current barriers. Researchers should weigh these benefits with the potential limitations of REDCap and this design, including staff time to set up, monitor, and clean the data outputs of the project.

    JMIR Form Res 2023;7:e42916

    doi:10.2196/42916

    Keywords

    ambulatory assessment; children; ecological momentary assessment; longitudinal; parents; survey 


    Overview

    Researchers are increasingly using intensive longitudinal data, in which responses are collected frequently across an interval of days or weeks, to understand health-related risk and protective factors [1]. One method commonly used to collect intensive longitudinal data is ecological momentary assessment (EMA) [1]. EMA is a collection of research methodologies that take multiple assessments from participants in their natural environment and in their current state [2,3].

    EMA has 3 main advantages over traditional survey designs. First, the use of frequent measurements in participants’ natural environments reduces recall, rater biases, and memory errors [2,4-6]. For example, EMA ratings correlate higher than retrospective recall ratings with accelerometer measures of physical activity [7-9]. Similarly, information collected through EMA ratings predicts children’s response to treatment for depression or anxiety above and beyond traditional self-report symptom measures [10]. Second, ratings collected through EMA are more ecologically valid than traditional surveys since they can include information on contextual factors that may influence ratings [11,12]. For example, a review of EMA use in the study of suicidal thoughts and behavior identified sleep quality and negative affect as short-term predictors of suicidal behavior [13]. In mental health research, EMA studies can therefore provide a more nuanced understanding of the presentation in question, which may not be accurately represented in questionnaires administered within a research environment. Third, data collected through EMA can be used to identify dynamic associations between variables over time [1-3], including potential causal associations [14].

    Despite its usefulness, current barriers make the use of EMA challenging. One potential barrier is the cost associated with the programs and applications that enable EMA data collection. EMA is often carried out by installing an application on participants’ smartphones [15]. Some of the current, most versatile software programs available in the market cost between US $1000 to $6000 [16], which may be prohibitive for many research teams. In addition, EMA applications may present privacy concerns [17-19]. For example, EMA applications may store participant data on external servers [15], including servers outside the country in which the researchers are working. Data are subject to the privacy laws of the country in which they are stored, which may vary in their level of protection and requirements for disclosure [20]. Storing data internally on a smartphone also poses a security risk as devices may be lost, misplaced, or stolen [19]. Another privacy concern is that EMA applications may collect extraneous data, such as phone use or location, when installed on participants’ smartphones [17]. Addressing cost and privacy barriers may increase the use of EMA, leading to results that support a more nuanced understanding of different health behaviors.

    Research Electronic Data Capture

    The use of existing and accessible software may help to overcome both cost and privacy barriers to using EMA. One such example is Research Electronic Data Capture (REDCap), developed by Vanderbilt University to improve the ease of capturing data for clinical research projects [21]. This web-client platform can be accessed through any device with a secure internet connection, and the easy-to-use tools require minimal programming abilities [22]. REDCap licenses are issued for free to nonprofit organizations that join the REDCap consortium. It is important to note that there are costs associated with running a REDCap server at an institution, which include costs for personnel, hardware, and network connections, among others.

    Features of REDCap also enhance privacy and data security. REDCap is compliant with the Health Insurance Portability and Accountability Act and has features that can safeguard personal health information [23]. These include the ability to limit access to identifying or sensitive information to specific team members and a 2-step authentication process. When building surveys within REDCap, researchers can mark specific fields as identifiable, allowing them to export only deidentified data. REDCap also stores data on servers designated by the institution for which it is set up, allowing researchers to ensure that data are protected by their country’s own privacy laws.

    An additional benefit of using REDCap is that it is a flexible and self-sufficient platform [24]. REDCap has a variety of features, such as surveys that include branching logic, the ability to design a matrix of questions for longer surveys, and the action of capturing electronic signatures for documenting informed consent, if required by institutional review boards [21]. Furthermore, REDCap has tools that allow users to easily export data to another statistical platform, such as Excel (Microsoft Corp), SPSS (IBM Corp), or R (The R Foundation). Integration with third-party applications also increases REDCap’s capabilities. Especially relevant for EMA, REDCap can interface with Twilio, a third-party application, to automatically send survey notifications to participants’ mobile phones through SMS text message [16] for a small fee per message (at the time of writing, approximately US $0.0079 per text sent). Participants are automatically sent an SMS text message containing a link to the survey at time intervals that are determined by the research team.

    While REDCap is a useful tool for data collection, researchers might experience challenges in trying to accommodate a longitudinal EMA design using its built-in features when working with older versions of the program [16]. As REDCap was initially developed for clinical trials, there are no easy-to-implement methods for sending out survey prompts frequently over a short period of time in these older versions. In addition, EMA studies are often carried out in a remote fashion, given that they require capturing data from participants in real time. Since this reduces the need for routine interactions with participants, it presents an opportunity for researchers to expand the reach of their recruitment, potentially spanning multiple time zones. Lastly, some participants may prefer to receive survey notifications through email, whereas others may prefer delivery by SMS text message. Setting up 2 separate delivery methods creates additional programming challenges that the REDCap administrator must navigate.

    Ultimately, REDCap provides an extensive tool kit of features that can be overwhelming to navigate for new or inexperienced users. In order to increase the capacity among health and social science researchers to use intensive longitudinal methods, it is important to share simplified designs that use recognizable features.

    This Study

    Despite the many potential benefits of ambulatory assessment methods such as EMA, the cost of software to facilitate EMA data collection and associated privacy concerns remain barriers to its widespread use. REDCap is a flexible tool that may allow researchers to overcome some of these key barriers to EMA implementation. Aside from 1 existing study that did not require survey delivery across time zones and used only a single survey per day [16], there is limited guidance available regarding how to use REDCap for EMA. The present report provides an overview of how REDCap can be used for EMA in a fully remote–capable, parent- and child-report EMA study. We present detailed instructions for survey setup, along with evidence of feasibility and information on user behavior based on a vanguard sample of 66 participants in an ongoing parent and child EMA study.


    Recruitment

    Participants were children aged between 9 and 13 years and their parents, who were part of a larger study of factors associated with variation in children’s emotional-behavioral functioning during the COVID-19 pandemic. Participants were recruited from the Canadian Healthy Infant Longitudinal Development (CHILD) cohort study [25], a multicenter, prospective, longitudinal birth cohort study that follows participants from pregnancy to the present day in 4 Canadian centers. CHILD was designed to understand the development of chronic disease by collecting data from various questionnaires, assessments, and biological samples [25,26]; however, it has since been used to look at health and development more broadly, including factors that influence children’s mental health.

    The CHILD participants (n=3542) from which the present sample was taken are predominantly White (2532/3542, 71.5%), with parents having at least a college or university-level education (5997/6794, 88.3%) [25]. Participants were recruited from CHILD if (1) the parent consented to be contacted about future studies; (2) the age 8 CHILD data collection visit was completed before January 2020; and (3) they were residing in Canada.

    At the time of data analysis, recruitment had only been conducted at 2 out of the 4 CHILD sites: Edmonton, Alberta, and Vancouver, British Columbia. Recruitment did not begin simultaneously at all sites due to differences across sites in the timing of securing agreements and ethical approvals. A total of 619 participants from the 2 initial sites were emailed a brief communication about this study, including a REDCap link to our e-consent form. Participants reported that they were in either the Mountain Standard Time or Pacific Standard Time zones. The current sample includes 66 parent-child dyads who had completed the study at the time of writing.

    For all participants, EMA data were collected over a 2-week period after the initial questionnaire was completed. The present analysis was carried out on July 4, 2022, and included all participants who completed the study between May 31 and June 16. More than 89% (85/95) of eligible participants completed the initial questionnaire and agreed to participate. Among those, 78% (66/85) of participants had completed the whole survey at the time of analysis (Figure 1).

    ‎
    Figure 1. Flowchart of inclusion and exclusion of participants in the current analysis. EMA: ecological momentary assessment.

    Measures

    Initial Survey

    Parents provided basic demographic information, along with their time zone, relationship to the child, and other study-specific information (eg, COVID-19 vaccination status). They also completed the 6-item Kessler Screening Questionnaire [27], a measure of psychological distress, with each item rated on a 5-point Likert scale. Parents indicated whether they would like to receive the links to the EMA surveys through email or SMS text message.

    EMA Surveys

    Parents were asked to estimate the amount of time their child spent in various activities (doing things with the parent, interacting face-to-face with family members who live outside the home, in meaningful face-to-face interactions with peers, in web-based or e-learning, watching television or digital media, and sleeping), and to rate their own stress levels, as well as their child’s positive and negative emotions and disruptive behavior, all over the previous 12 hours. Children were asked to rate how much time they spent playing or talking with friends face-to-face, their positive and negative emotions, and their interactions with their participating parent (warmth, harsh parenting, and inconsistent parenting) over the past 12 hours on a 5-point scale ranging from “not at all” to “a lot.” EMA items were adapted from the Positive and Negative Affect Schedule for Children [28], the Coronavirus Health Impact Survey [29], and from existing, single-item measures of relevant constructs [30-37]. Based on pilot testing within our research team, EMA surveys took approximately 5 minutes each to complete by both parent and child.

    Procedure

    Overview

    After completing the initial survey, parents received survey notifications beginning the following day through their preferred delivery method (email or SMS text message), twice daily (at 7:30 AM and 7:30 PM local time) for 14 days. The timings of the surveys were selected based on input from parent advisors to accommodate the availability of as many parent-child dyads as possible, since most children would be traveling to school in the mornings. The evening time was then selected to be spaced 12 hours from the morning survey.

    At the beginning of each survey, parents were asked if their child had been staying outside the home for the past 12 hours; if this was the case, the survey ended. After completing the parent items, parents were asked if their child was present and willing to answer survey questions; if so, children then completed EMA ratings using the same device. Parents who had not responded received 2 reminders at 30-minute intervals. The survey remained open for 3 hours following the initial notification to allow parents enough time to provide their responses and to accommodate potential differences in family schedules. We provide an overview of our REDCap setup in Multimedia Appendix 1. These instructions are based on REDCap version 11.1.21.

    Survey Setup in the REDCap Online Designer

    Since this study recruited participants from across Canada, the REDCap setup had to accommodate individuals from 6 time zones. Surveys for different time zones must be set up separately to facilitate the programming of survey notifications later on. We set up a morning survey and an evening survey for each of the time zones, leaving us with 6 morning surveys and 6 evening surveys in total. To differentiate between surveys for different time zones, we included the time zone abbreviation in each of our survey titles.

    Settings to Consider in Project Setup and the Online Designer

    Using the Project Setup page, we enabled both Surveys and Longitudinal Data Collection for this project. We then identified an email field (from the initial survey) in the “Enable optional modules and customizations” section to email participants the survey links should they choose this option. Then, in the Online Designer, we enabled all our data collection instruments as surveys. In each of the morning and evening surveys, we added a “Begin new section” field that separated the questions for parents from those of the children. Then, we accessed the Survey Settings of all the morning and evening surveys to edit some functions. First, we selected the “Enhanced radios and checkboxes” option under the Survey Design Options section. This enables larger buttons in the survey to appear, which enhances the user experience for participants completing their surveys on a mobile device. Second, under Survey Customizations, we selected “Multiple pages (display 1 section per page)” so that questions for child participants appear on their own page. Finally, under the Survey Access section, we imposed a 3-hour time limit for survey completion (from the time of survey delivery) using the “Time limit for survey completion” option.

    Twilio Setup

    Twilio was used to send surveys to participants who selected notifications through SMS text message. From the Project Setup page, we accessed the Twilio main menu to configure its settings. We enabled surveys to be sent as web pages only. We also identified which field asks for participants’ preferred method of survey delivery (email or SMS text message), as well as the field in which participants’ cell phone numbers could be found (from the initial survey).

    Events and Instruments Setup

    We split the project into 6 arms, each representing a time zone in Canada. We had 29 events in each of the arms. The first event we defined was the baseline event, and all subsequent events were EMA morning and evening surveys for days 1-14. In each arm, the Initial Questionnaire and the Parent Information and Contact Preferences form were assigned to the baseline event. We then designated the appropriate morning and evening EMA surveys for each of the arms.

    Setup of Automated Survey Notifications

    In the Online Designer, we used the Automated Invitations feature to set up the survey notifications for each of the events we defined in the previous step. We enabled up to 2 reminders for each of the surveys, programmed to be sent 30 minutes and 1 hour after the initial survey notification was delivered if the participant had not responded. Further details are provided in Multimedia Appendix 1.

    Staff Time

    Research staff members were involved in the setting up, piloting, and monitoring phases of the study. Setup in REDCap required a few days of work from building the instruments to programming the notifications, to then performing error checking on the project. Piloting lasted a total of 2 weeks, during which staff members were assigned a time zone to sign up for and pilot its surveys, delivered through email or SMS text message. In this phase, we ensured that survey delivery was successful at the desired times and that surveys were suitable for completion on both laptops and mobile phones. Once piloting was complete, we began the data collection phase. This required 1 staff member to monitor new sign-ups daily through REDCap and check for any errors that were flagged by the system. Daily monitoring did not take more than a few minutes each time. Finally, data clean up took a total of 2 days.

    Statistical Analysis

    The database was exported to Stata (StataCorp) using the automated export procedure. Adherence to the study was measured in terms of the number of EMA responses available across the 14 days (up to a maximum of 28), completion rates at each time point, and child availability to fill out their part of the questionnaire. Differences in completion rates according to time of day (morning or evening), weekends and weekdays, and notification type (SMS text message or email) were identified using univariate chi-square tests for categorical variables and the nonparametric Wilcoxon Mann-Whitney test for numeric variables. We also examined user behavior, including average response time (computed as the time lag between the first notification and the response time logged in REDCap). Finally, we compared included and excluded participants’ characteristics according to gender, income, and type of notification (email or SMS text message). For categorical variables, we used Fisher exact test of independence to test for differences, as some expected numbers were less than 5.

    Ethical Considerations

    This study was approved by the last author’s (MA) (Centre for Addiction and Mental Health) research ethics board (003-2022) and the ethics boards at each of the respective CHILD study sites (University of Alberta, Pro00117899_AME1; UBC Children's & Women’s Research Ethics Board, H07-03120). All participants participated voluntarily. Parents provided their consent, and children provided their assent using our hospital’s e-consent framework in REDCap. Identifiers, such as email and phone number, were collected initially and stored in REDCap to facilitate survey delivery. Fields that collected these data were marked as identifiers in the REDCap Online Designer to filter them out from any data exports and ensure that the main database is deidentified. Participating families were paid up to CAD $90 (US $67) in the form of e-gift cards at the end of this study, depending on the number of surveys completed (CAD $10 [US $7.40] for completing the baseline questionnaire, CAD $2.50 [US $1.85] for every survey completed, and an extra CAD $10 [US $7.40] for completing 21 or more out of the 28 daily surveys).


    Surveys Available

    From May 31 to July 4, 2022, a total of 1564 records were received from 66 parent-child dyads (66 baseline and 1498 EMA records). EMA records split almost equally between morning (n=739, 49.3%) and evening surveys (n=759, 50.7%). On 108 (7.2%) occasions, the child was staying outside the home for the previous 12 hours, and no EMA ratings were collected.

    Participant Characteristics

    Most parent participants were women (n=64, 97%) and all parents were biologically related to the child. Children ranged in age from 9 to 13 years old. The sample was of relatively high socioeconomic status, with 82% (54/66) having a total annual family income of CAD $90,000 (US $66,628) or more. The majority of participants spoke English at home, with 1 family speaking both English and French and 1 family speaking Russian. See Table 1 for detailed sample characteristics. More than half of the participants made the choice to receive the link to the EMA surveys by SMS text messages (34/66, 52%), but the proportion of participants who preferred to be notified by email was substantial (32/66, 48%).

    Compared to the 66 participants included, 19 participants who had not completed the study at the time of analysis had 18.8 (SD 8.0) records on average (vs 22.7, SD 7.7 for participants included). Participants who had not completed the study yet had a slightly higher income (4/12, 21% reported an annual income of CAD $90,000 [US $66,628]) than participants who completed it (12/66, 18%). No difference was found according to gender, or notification type. See Table 2 for detailed differences between included and excluded participants.

    There were no differences in participants’ characteristics between the 2 sites. See Table 3 for detailed comparisons between participant characteristics from the 2 sites.

    Table 1. Sociodemographic characteristics of participants (n=66). A currency exchange rate of CAD $1=US $0.74 is applicable.
    CharacteristicsValue, n (%)
    Parent’s gendera

    		Woman64 (97)

    		Man1 (2)

    		Genderqueer1 (2)

    		Gender fluid1 (2)

    		PNTAb1 (2)
    Child’s gendera

    		Girl21 (32)

    		Boy43 (65)

    		Nonbinary1 (2)

    		Gender expansive2 (3)
    Language

    		English only64 (97)

    		English and French1 (2)

    		Russian1 (2)
    Family income (CAD $)

    		0-29,9991 (2)

    		30,000-59,9992 (3)

    		60,000-89,9999 (14)

    		90,000-119,99918 (27)

    		120,000-149,9999 (14)

    		150,000 or more27 (41)
    Time zone

    		Pacific Standard Time11 (17)

    		Mountain Standard Time55 (83)

    aPercentages do not add up to 100 due to rounding and the option for participants to select multiple options.

    bPNTA: prefer not to answer.

    Table 2. Differences in included and excluded participants (n=85). A currency exchange rate of CAD $1=US $0.74 is applicable.
    Participant characteristicsExcluded (n=19)Included (n=66)P value
    Gender, n (%).14a

    		Girl10 (53)21 (32)

    		Boy8 (42)43 (65) 

    		Other1 (5)2 (3) 
    Income (CAD $), n (%).01a

    		<90,00012 (63)54 (82)

    		≥90,0004 (21)12 (18) 

    		No answer3 (16)0 (0) 
    Number of records, mean (SD)18.8 (8.0)22.7 (7.7).004b
    Notification type, n (%).80a

    		Email8 (42)32 (49)

    		SMS text message11 (58)34 (52)

    aFisher exact test.

    bNonparametric Wilcoxon Mann-Whitney test.

    Table 3. Differences in participant characteristics according to site (n=66). A currency exchange rate of CAD $1=US $0.74 is applicable.
    Participants’ characteristicsVancouver (n=11)Edmonton (n=55)P value
    Gender, n (%).16a

    		Girl1 (9)20 (36)

    		Boy10 (91)33 (60)

    		Other0 (0)2 (4)
    Income (CAD $), n (%)>.99a

    		<90,0009 (82)45 (82)

    		≥90,0002 (18)10 (18)
    Number of records, mean (SD)20.2 (11.0)23.2 (6.9).80b
    Notification type, n (%)>.99a

    		Email5 (46)27 (49)

    		SMS text message6 (55)28 (51) 

    aFisher exact test.

    bNonparametric Wilcoxon Mann-Whitney test.

    Completion Rates

    Participants completed on average 23 (SD 8) of the 28 EMA assessments, and more than half of the sample (37/66, 56%) completed at least 26 assessments. The number of assessments did not differ according to total family income (mean 22.5, SD 8.4 for participants whose total family income was more than CAD $90,000 [US $66,628] vs mean 23.8, SD 3.4 for participants with lower income; z score=1.12; P=.27) but was significantly higher when the child identified as a girl compared to when the child identified as a boy (mean 24.9, SD 6.2 vs mean 21.7, SD 8.4; n=64; z score=2.37; P=.02).

    Completion rates were similar across study time points, varying between 70% (46/66) and 88% (58/66) in the morning and 76% (50/66) and 88% (58/66) in the evening (Figure 2). The completion rate was higher on weekdays (Monday to Friday) than on weekend days (1087/1320, 82.4% vs 411/523, 77.8%; χ21=4.9; P=.02). Participants notified by SMS text message had higher rates of completion at most time points, with significant differences being found on several morning occasions compared to participants notified by email. Participants notified by SMS text message also had a significantly higher number of assessments available compared to participants notified by email (mean 24.5, SD 6.5 vs mean 20.8, SD 8.5; z score=–2.53; P=.01).

    Children were available to answer their own EMA survey questions in 75.7% (1134/1498) of assessments. Child availability was significantly higher in the evening than in the morning assessments (603/759, 79.5% vs 531/739, 71.9%; χ21=11.7; P=.001). No difference in child availability was found according to notification type (512/664, 77.1% in the email vs 622/834, 74.6% in the SMS text message group; χ21=1.3; P=.26) between weekdays and weekends (respectively, 817/1087, 75.2% and 317/411, 77.1%; χ21=0.6; P=.43), or according to child gender (390/522, 74.7% in girls vs 720/933, 77.2% in boys; n=64; χ21=1.1; P=.29).

    ‎
    Figure 2. Completion rates by time point and notification type. “*” denotes a significant difference in response rates between SMS text messages and emails (P<.05).

    Average Response Time

    The average response time was computed considering 1494 records with available time data (4 records had no logged time in REDCap, as the user was disconnected while filling out the questionnaire). The average response time was 47.0 (SD 52.3) minutes after the initial survey prompt. Response time was significantly shorter in the evening (mean 43.1, SD 40.3 minutes) than in the morning (mean 51.0, SD 62.1 minutes; z score=2.11; P=.03) and when participants were notified by SMS text message (mean 38.1, SD 41.30 minutes) than when notified by email (mean 58.2, SD 61.7 minutes; z score=9.33; P<.001).

    The distribution of assessments completed according to response time (in 5-minute intervals) is presented in Figure 3. The proportion of assessments completed in the first 5 minutes was significantly higher for the evening than for the morning assessments (106/757, 14% vs 77/737, 10.5%; χ21=4.4; P=.04), and for participants notified by SMS text message rather than email (143/831, 17.2% vs 40/663, 6%; χ21=42.9; P<.001). Assessments completed after 2 hours represented 7.8% (116/1494) of the records and were significantly less frequent in the evening than in the morning (47/757, 6.2% vs 69/737, 9.4%; χ21=5.2; P=.02) and when participants were notified by SMS text message rather than email (51/831, 6.1% vs 65/663, 9.8%; χ21=6.9; P=.009). A visual inspection of Figure 3 shows that the first reminder was followed by a substantial increase in the proportion of assessments completed, whereas the second reminder appears to have enhanced completion only in the evening and in the participants notified by SMS text message. An overview of key takeaways and future research directions is presented in Textbox 1.

    ‎
    Figure 3. Response time by time of day and notification type. Seven records with a response time higher than 3.5 hours (due to technical problems at Day 1 AM assessment) are not presented. AM=morning time; PM=evening time.
    Textbox 1. Key takeaways and directions for future research.

    Key takeaways

    • Research Electronic Data Capture (REDCap) can accommodate ecological momentary assessment (EMA) studies, including those that recruit across multiple time zones, require little to no communication with participants, and involve both parents and children.
    • The use of REDCap addresses privacy and cost concerns that are often associated with EMA studies that require the use of an application.
    • It is important to offer participants more than one mode of survey delivery. SMS text messages and email survey delivery were almost equally chosen in our sample. Completion rates were higher among participants who chose SMS text message delivery.
    • Implementing reminders in the project design enhances survey completion rates.
    • This design is most useful to users of REDCap who are new to or inexperienced with using REDCap but would like to implement an EMA study design.
    • Institutions that are part of the REDCap consortium are likely to be running different versions of REDCap, some more advanced than others. It is important to communicate designs that involve easily recognizable features of REDCap that can be used regardless of the REDCap version.

    Implications for future research using EMA

    • Future studies can build on this design to develop projects that require less time to build and produce cleaner data outputs.
    • Future EMA studies can investigate whether participants from rural areas have similar response rates and access to the required technology as participants from urban areas.
    • Future studies can focus on why response rates in EMA studies vary by gender and suggest methods to ensure all participants are equally engaged.

    Overview

    Ambulatory assessment methods, such as EMA, offer a number of potential benefits, such as decreased recall errors, the ability to collect more ecologically valid data in real-world settings, the collection of intensive longitudinal data that can be used to examine predictive associations over time, and improved prediction of mental health treatment outcomes [2,10,19]. We have developed a method for using a customizable data capture tool (REDCap) to collect EMA data that may allow researchers to overcome some of the cost and privacy barriers that make implementation of EMA methods challenging. By providing details of our survey setup in REDCap, we aim to increase the ability of other researchers to use similar methods in the future. This design would be most useful to researchers who are unfamiliar with or new to using REDCap, as it uses features that are easily accessible and recognizable by users. Our results also provide information on parent and child completion rates and respondent behavior.

    Our preliminary results support the feasibility of using EMA with parents and children. Completion rates for both parents and children were high. Moreover, we did not observe a decline in completion rates across the 2-week study period or a difference in completion rates by family income. It is important to note that our relatively high completion rates may be in part due to offering financial compensation based on the number of surveys completed [38] or to our recruitment from a cohort of families already engaged in another research study. However, our completion rates are comparable to those reported in previous child and parent EMA studies. Heron and colleagues [39] reported an average survey completion rate of 76% among youth participants involved in 54 different EMA studies. Similarly, an EMA study that prompted parents to answer questions about their child’s affect 3 times daily for 28 days found that parents completed an average of 83% of the 84 possible assessments [40]. Our sample also had a higher than average socioeconomic status compared to the Canadian population and was largely urban and suburban [41]. These participants likely had ready access to mobile devices with internet access and may have faced fewer challenges to study participation than participants from lower socioeconomic backgrounds. However, we did not find significant differences in completion rates across families with high or low income in our sample. Individuals who do not have access to devices they can use to respond to surveys or who do not live in an area with sufficient internet coverage might not be represented in similar studies. Further research is therefore needed to determine response rates in more representative samples.

    We also found that completion rates were higher on weekdays compared to weekends, though completion rates remained high on weekends. Our results are consistent with previous EMA studies in adults that have reported a similar pattern [42] but differ from a previous study of mothers and children, which found that child compliance with EMA prompts was higher on weekends than on weekdays [43]. Given that differences in affect and activity across weekends and weekdays have also been reported in EMA studies in children and adults [42,44], researchers may consider weekday versus weekend as a control variable in their analyses.

    In terms of survey delivery methods, approximately half of the sample preferred to receive survey links by email, whereas the other half preferred to receive notifications by SMS text message. Of note, completion rates were significantly higher for participants receiving SMS text message survey notifications compared to email notifications. Moreover, participants took less time to respond to survey notifications received by SMS text message than by email. Most existing EMA studies have not reported completion rates by notification method (email vs SMS text message), have not provided participants with different notification options [16,45,46], or have used mobile apps with built-in prompts for participants to take assessments at appropriate intervals [47,48]. However, our results are consistent with 1 previous study on drug use among adults in nonurban areas, in which EMA completion rates were higher when notifications were sent through SMS text message compared to email [49]. Therefore, while there is a small cost associated with using Twilio, the add-on program that can be used in conjunction with REDCap to send survey notifications through SMS text message, providing this option may be important for increasing participant responses.

    We used 2 survey reminders with the goal of increasing participant response rates, recognizing that participants may not have been available to answer questions immediately at the designated survey times (7:30 AM and 7:30 PM). The first reminder resulted in a substantial increase in responses, whereas the increase in responses following the second reminder (1 hour after the initial survey notification) was more modest. However, the survey also remained active for 3 hours after the initial notification, and we observed that a substantial proportion of participants responded between 2 and 3 hours after the initial survey notification. As a result, having a slightly extended response window may enhance completion rates, though potential effects on participant recall accuracy should be considered.

    Lastly, we found that the number of assessments completed was higher in dyads with girls than in dyads with boys. Our results are consistent with meta-analytic evidence that compliance rates are higher in samples containing more women than men [38]; however, our results differ from a previous study that found no difference in compliance with EMA prompts for children or mothers by child gender [43]. The reasons for the gender differences in response rates in our sample are unclear, and should they be replicated in other studies, further research will be needed to understand the reasons for any gender differences in response rates [38]. In terms of parent gender, most parents in our sample were mothers, consistent with the focus of the CHILD Study on perinatal enrollment and data collection. Fathers have been historically underrepresented in research on children’s mental health [50], and further research is needed to understand fathers’ response patterns to EMA surveys.

    Limitations

    A limitation of the design we present is that it does not recognize the time zone in which participants are located, which required us to add a separate arm for each time zone. This limitation creates downstream challenges. For example, data cleaning is required to remove redundant, empty fields from other time zones. Additionally, another problem can arise if a participant enters the study on a specific date (say, July 5), and it is already the next day in the time zone of the institution (say, July 6). REDCap will assume that the participants skipped day 1 and will send them the day 2 survey. As a result, daily monitoring of the REDCap project platform by a project administrator is required to manually adjust for this known issue. Finally, participants who travel from one time zone to another might not inform researchers of their move, which compromises the timing at which surveys are completed. If researchers become aware of a time zone change, they will have to manually change the timing of surveys as REDCap queues the surveys daily. Therefore, a limitation of this REDCap design is a lack of full automation, which requires some manual upkeep to ensure that the project runs smoothly and data are captured accurately. It is important to note that newer versions of REDCap support a feature called MyCap, which can be a valuable tool for EMA studies that can facilitate a more efficient and less labor-intensive design. Participants can download the MyCap app on their mobile devices and complete surveys through it. At the time of writing, our institution had not yet supported MyCap. Since versions of REDCap differ from one institution to another, researchers can connect with their REDCap administrators to learn about the capabilities of the REDCap version their institution has.

    Aside from design limitations, it is important to note that our sample of participants lacked socioeconomic diversity. While a previous EMA study involving a racially diverse sample found no differences in survey compliance based on participant demographics or other characteristics [51], there remains a dearth of research on the reliability of EMA data collection across diverse samples. Finally, we did not collect information on why participants missed some of the EMA surveys. This information could be important for improving survey completion rates. Future studies can use a feedback survey at the end of the study to inquire why participants were not able to complete all surveys or whether they have recommendations that can facilitate higher completion rates.

    Conclusions

    REDCap can be used by researchers for EMA and is flexible enough to accommodate multiple time zones and survey delivery methods (email and SMS text message). The use of SMS text message survey notifications appears to be an important way to increase participant responses. REDCap is freely available to nonprofit organizations that join the REDCap consortium, and there is only a small cost associated with the use of Twilio for SMS text message survey delivery. Moreover, its data storage and administrator controls reduce potential privacy concerns associated with EMA. Researchers wishing to implement an EMA in REDCap should consider the capabilities of this platform and of the REDCap version to which they have access, and the extent to which these meet the requirements of their project design.

    Acknowledgments

    Funding for this project was provided by the Canadian Institutes of Health Research. Team members also receive funding support from the Social Sciences and Humanities Research Council and the Cundill Centre for Child and Youth Depression. We thank the CHILD cohort study participant families for their dedication and commitment to advancing health research.

    Data Availability

    The data generated and analyzed during this study are not publicly available because this is an early data report on a subsample from an ongoing study. The full data set will be made available by the corresponding author once the study is completed.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Instructions for ecological momentary assessment set-up in REDCap.

    DOCX File , 2087 KB
    1. Hamaker EL, Wichers M. No time like the present: discovering the hidden dynamics in intensive longitudinal data. Curr Dir Psychol Sci. 2017;26(1):10-15. [FREE Full text] [CrossRef]
    2. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol. 2008;4:1-32. [CrossRef] [Medline]
    3. Shiffman S, Stone AA. Introduction to the special section: ecological momentary assessment in health psychology. Health Psychol. 1998;17(1):3. [CrossRef]
    4. Mofsen AM, Rodebaugh TL, Nicol GE, Depp CA, Miller JP, Lenze EJ. When all else fails, listen to the patient: a viewpoint on the use of ecological momentary assessment in clinical trials. JMIR Ment Health. 2019;6(5):e11845. [FREE Full text] [CrossRef] [Medline]
    5. Moskowitz DS, Young SN. Ecological momentary assessment: what it is and why it is a method of the future in clinical psychopharmacology. J Psychiatry Neurosci. 2006;31(1):13-20. [FREE Full text] [Medline]
    6. Stull DE, Leidy NK, Parasuraman B, Chassany O. Optimal recall periods for patient-reported outcomes: challenges and potential solutions. Curr Med Res Opin. 2009;25(4):929-942. [CrossRef] [Medline]
    7. de Brito JN, Loth KA, Tate A, Berge JM. Associations between parent self-reported and accelerometer-measured physical activity and sedentary time in children: ecological momentary assessment study. JMIR Mhealth Uhealth. 2020;8(5):e15458. [FREE Full text] [CrossRef] [Medline]
    8. Knell G, Gabriel KP, Businelle MS, Shuval K, Wetter DW, Kendzor DE. Ecological momentary assessment of physical activity: validation study. J Med Internet Res. 2017;19(7):e253. [FREE Full text] [CrossRef] [Medline]
    9. Zink J, Belcher BR, Dzubur E, Ke W, O'Connor S, Huh J, et al. Association between self-reported and objective activity levels by demographic factors: ecological momentary assessment study in children. JMIR Mhealth Uhealth. 2018;6(6):e150. [FREE Full text] [CrossRef] [Medline]
    10. Forbes EE, Stepp SD, Dahl RE, Ryan ND, Whalen D, Axelson DA, et al. Real-world affect and social context as predictors of treatment response in child and adolescent depression and anxiety: an ecological momentary assessment study. J Child Adolesc Psychopharmacol. 2012;22(1):37-47. [FREE Full text] [CrossRef] [Medline]
    11. Bolger N, Davis A, Rafaeli E. Diary methods: capturing life as it is lived. Annu Rev Psychol. 2003;54:579-616. [CrossRef] [Medline]
    12. Stevens A, Griffiths S. Body Positivity (#BoPo) in everyday life: an ecological momentary assessment study showing potential benefits to individuals' body image and emotional wellbeing. Body Image. 2020;35:181-191. [FREE Full text] [CrossRef] [Medline]
    13. Sedano-Capdevila A, Porras-Segovia A, Bello HJ, Baca-García E, Barrigon ML. Use of ecological momentary assessment to study suicidal thoughts and behavior: a systematic review. Curr Psychiatry Rep. 2021;23(7):41. [CrossRef] [Medline]
    14. Borsboom D, Deserno MK, Rhemtulla M, Epskamp S, Fried EI, McNally RJ, et al. Network analysis of multivariate data in psychological science. Nat Rev Methods Primers. 2021;1(1):58. [FREE Full text] [CrossRef]
    15. Domoff SE, Banga CA, Borgen AL, Foley RP, Robinson C, Avery K, et al. Use of passive sensing to quantify adolescent mobile device usage: feasibility, acceptability, and preliminary validation of the eMoodie application. Hum Behav Emerg Technol. 2021;3(1):63-74. [CrossRef]
    16. Tomko RL, Gray KM, Oppenheimer SR, Wahlquist AE, McClure EA. Using REDCap for ambulatory assessment: implementation in a clinical trial for smoking cessation to augment in-person data collection. Am J Drug Alcohol Abuse. 2019;45(1):26-41. [FREE Full text] [CrossRef] [Medline]
    17. de Vries LP, Baselmans BML, Bartels M. Smartphone-based ecological momentary assessment of well-being: a systematic review and recommendations for future studies. J Happiness Stud. 2021;22(5):2361-2408. [FREE Full text] [CrossRef] [Medline]
    18. Roth AM, Rossi J, Goldshear JL, Truong Q, Armenta RF, Lankenau SE, et al. Potential risks of ecological momentary assessment among persons who inject drugs. Subst Use Misuse. 2017;52(7):840-847. [FREE Full text] [CrossRef] [Medline]
    19. Carpenter RW, Wycoff AM, Trull TJ. Ambulatory assessment: new adventures in characterizing dynamic processes. Assessment. 2016;23(4):414-424. [FREE Full text] [CrossRef] [Medline]
    20. Abed Y, Chavan M. The challenges of institutional distance: data privacy issues in cloud computing. Sci Technol Soc. 2018;24(1):161-181. [FREE Full text] [CrossRef]
    21. Patridge EF, Bardyn TP. Research Electronic Data Capture (REDCap). JMLA. 2018;106(1):142. [FREE Full text] [CrossRef]
    22. Harris PA. Research Electronic Data Capture (REDCap)—planning, collecting and managing data for clinical and translational research. BMC Bioinformatics. 2012;13(S12):A15. [CrossRef]
    23. Bowman MA, Maxwell RA. A beginner's guide to avoiding Protected Health Information (PHI) issues in clinical research—with how-to's in REDCap data management software. J Biomed Inform. 2018;85:49-55. [FREE Full text] [CrossRef] [Medline]
    24. Hawley S, Yu J, Bogetic N, Potapova N, Wakefield C, Thompson M, et al. Digitization of measurement-based care pathways in mental health through REDCap and electronic health record integration: development and usability study. J Med Internet Res. 2021;23(5):e25656. [FREE Full text] [CrossRef] [Medline]
    25. Subbarao P, Anand SS, Becker AB, Befus AD, Brauer M, Brook JR, et al. The Canadian Healthy Infant Longitudinal Development (CHILD) study: examining developmental origins of allergy and asthma. Thorax. 2015;70(10):998-1000. [FREE Full text] [CrossRef] [Medline]
    26. Moraes TJ, Lefebvre DL, Chooniedass R, Becker AB, Brook JR, Denburg J, et al. The Canadian healthy infant longitudinal development birth cohort study: biological samples and biobanking. Paediatr Perinat Epidemiol. Jan 2015;29(1):84-92. [CrossRef] [Medline]
    27. Kessler RC, Barker PR, Colpe LJ, Epstein JF, Gfroerer JC, Hiripi E, et al. Screening for serious mental illness in the general population. Arch Gen Psychiatry. Feb 2003;60(2):184-189. [FREE Full text] [CrossRef] [Medline]
    28. Ebesutani C, Regan J, Smith A, Reise S, Higa-McMillan C, Chorpita BF. The 10-item positive and negative affect schedule for children, child and parent shortened versions: application of item response theory for more efficient assessment. J Psychopathol Behav Assess. 2012;34(2):191-203. [CrossRef]
    29. Nielson D. The coronavirus health impact survey (CRISIS). Open Science Framework. 2020. URL: https://osf.io/z63bm/ [accessed 2023-09-22]
    30. Colley RC, Wong SL, Garriguet D, Janssen I, Gorber SC, Tremblay MS. Physical activity, sedentary behaviour and sleep in Canadian children: parent-report versus direct measures and relative associations with health risk. Health Rep. 2012;23(2):45-52. [FREE Full text] [Medline]
    31. Ackard DM, Neumark-Sztainer D, Story M, Perry C. Parent-child connectedness and behavioral and emotional health among adolescents. Am J Prev Med. 2006;30(1):59-66. [CrossRef] [Medline]
    32. Papanikolaou F, Fried EI, Epskamp S. Mental health and social contact during the COVID-19 pandemic: an ecological momentary assessment study. Clin Psychol Sci. 2022;10(2):340-354. [FREE Full text] [CrossRef]
    33. Gassman-Pines A, Ananat EO, Fitz-Henley J. COVID-19 and parent-child psychological well-being. Pediatrics. 2020;146(4):e2020007294. [FREE Full text] [CrossRef] [Medline]
    34. Boyle MH, Duncan L, Georgiades K, Wang L, Comeau J, Ferro MA, et al. The 2014 Ontario Child Health Study Emotional Behavioural Scales (OCHS-EBS) part ii: psychometric adequacy for categorical measurement of selected disorders. Can J Psychiatry. 2019;64(6):434-442. [FREE Full text] [CrossRef] [Medline]
    35. Omowale SS, Casas A, Lai YH, Sanders SA, Hill AV, Wallace ML, et al. Trends in stress throughout pregnancy and postpartum period during the COVID-19 pandemic: longitudinal study using ecological momentary assessment and data from the postpartum mothers mobile study. JMIR Ment Health. 2021;8(9):e30422. [FREE Full text] [CrossRef] [Medline]
    36. Palou P, Muntaner-Mas A, Cantallops J, Borràs PA, Labayen I, Jiménez-Pavón D, et al. A single question of parent-reported physical activity levels estimates objectively measured physical fitness and body composition in preschool children: The PREFIT Project. Front Psychol. 2019;10:1585. [FREE Full text] [CrossRef] [Medline]
    37. Wang MT, Henry DA, Del Toro J, Scanlon CL, Schall JD. COVID-19 employment status, dyadic family relationships, and child psychological well-being. J Adolesc Health. 2021;69(5):705-712. [FREE Full text] [CrossRef] [Medline]
    38. Wrzus C, Neubauer AB. Ecological momentary assessment: a meta-analysis on designs, samples, and compliance across research fields. Assessment. 2023;30(3):825-846. [FREE Full text] [CrossRef] [Medline]
    39. Heron KE, Everhart RS, McHale SM, Smyth JM. Using mobile-technology-based ecological momentary assessment (EMA) methods with youth: a systematic review and recommendations. J Pediatr Psychol. 2017;42(10):1087-1107. [FREE Full text] [CrossRef] [Medline]
    40. Fogleman ND, Walerius DM, Rosen PJ, Leaberry KD. Peer victimization linked to negative affect in children with and without ADHD. J Appl Dev Psychol. 2016;46:1-10. [FREE Full text] [CrossRef]
    41. Statistics Canada. Market income, government transfers, total income, income tax and after-tax income by economic family type. Statistics Canada. 2023. URL: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1110019001 [accessed 2023-09-22]
    42. Nordgren R, Hedeker D, Dunton G, Yang CH. Extending the mixed-effects model to consider within-subject variance for ecological momentary assessment data. Stat Med. 2020;39(5):577-590. [CrossRef] [Medline]
    43. Dzubur E, Huh J, Maher JP, Intille SS, Dunton GF. Response patterns and intra-dyadic factors related to compliance with ecological momentary assessment among mothers and children. Transl Behav Med. 2018;8(2):233-242. [FREE Full text] [CrossRef] [Medline]
    44. Papini NM, Yang CH, Do B, Mason TB, Lopez NV. External contexts and movement behaviors in ecological momentary assessment studies: a systematic review and future directions. Int Rev Sport Exerc Psychol. 2020:1-31. [FREE Full text] [CrossRef]
    45. Mitchell RJ, Goggins R, Lystad RP. Synthesis of evidence on the use of ecological momentary assessments to monitor health outcomes after traumatic injury: rapid systematic review. BMC Med Res Methodol. 2022;22(1):119. [FREE Full text] [CrossRef] [Medline]
    46. Zarate D, Stavropoulos V, Ball M, de Sena Collier G, Jacobson NC. Exploring the digital footprint of depression: a PRISMA systematic literature review of the empirical evidence. BMC Psychiatry. 2022;22(1):421. [FREE Full text] [CrossRef] [Medline]
    47. Asselbergs J, Ruwaard J, Ejdys M, Schrader N, Sijbrandij M, Riper H. Mobile phone-based unobtrusive ecological momentary assessment of day-to-day mood: an explorative study. J Med Internet Res. 2016;18(3):e72. [FREE Full text] [CrossRef] [Medline]
    48. Bai R, Xiao L, Guo Y, Zhu X, Li N, Wang Y, et al. Tracking and monitoring mood stability of patients with major depressive disorder by machine learning models using passive digital data: prospective naturalistic multicenter study. JMIR Mhealth Uhealth. 2021;9(3):e24365. [FREE Full text] [CrossRef] [Medline]
    49. Biello K, Salhaney P, Valente PK, Childs E, Olson J, Earlywine JJ, et al. Ecological momentary assessment of daily drug use and harm reduction service utilization among people who inject drugs in non-urban areas: a concurrent mixed-method feasibility study. Drug Alcohol Depend. 2020;214:108167. [FREE Full text] [CrossRef] [Medline]
    50. Sheeber LB, Allen NB, Leve C, Davis B, Shortt JW, Katz LF. Dynamics of affective experience and behavior in depressed adolescents. J Child Psychol Psychiatry. 2009;50(11):1419-1427. [FREE Full text] [CrossRef] [Medline]
    51. Soyster PD, Bosley HG, Reeves JW, Altman AD, Fisher AJ. Evidence for the feasibility of person-specific ecological momentary assessment across diverse populations and study designs. J Pers Oriented Res. 2019;5(2):53-64. [FREE Full text] [CrossRef] [Medline]

    ‎
    CHILD: Canadian Health Infant Longitudinal Development
    EMA: ecological momentary assessment
    REDCap: Research Electronic Data Capture

    Edited by A Mavragani; submitted 23.09.22; peer-reviewed by J de Brito, G Rezniczek; comments to author 23.01.23; revised version received 31.05.23; accepted 20.09.23; published 09.11.23.

    Copyright

    ©Yola El Dahr, Florence Perquier, Madison Moloney, Guyyunge Woo, Roksana Dobrin-De Grace, Daniela Carvalho, Nicole Addario, Emily E Cameron, Leslie E Roos, Peter Szatmari, Madison Aitken. Originally published in JMIR Formative Research (https://formative.jmir.org), 09.11.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.