The Acceptability and Efficacy of Electronic Data Collection in a Hospital Neurodevelopmental Clinic: Pilot Questionnaire Study

Introduction

Electronic data collection (EDC) has been at the center of debate about the future of 21st century health care [1-3]. Such approaches have the potential to save billions in health care costs through improved data capture and clinical service responses that allow for more efficient patient-centered care [4-7]. There has, however, been a slow uptake of EDC approaches in clinical services globally, and limited evidence of successful technology integration in public hospital settings [5,8-10]. For instance, in most hospital-based clinics across Australia, data collection is largely paper based. Electronic medical records are being introduced in hospitals; however, this process has been slow and data entry into electronic medical records remains less than systematic [11]. As a result, a recent inquiry report by the Australian Government Productivity Commission suggested that Australia was falling behind in utilizing health care data for data linkage between health services for research purposes [12].

Despite these issues, EDC offers many benefits, warranting its evaluation in public health service settings. EDC provides the opportunity to engage families in more efficient services. For instance, patients can conveniently access forms and staff require less time to monitor, analyze, and interpret the gathered data, allowing for swift provision of feedback to patients and families [13,14]. In addition to this increased efficiency, EDC has been shown to result in fewer human errors in data processing and enables collection of data from a broader geography, increasing completeness of data collation and freeing clinical service resources for other needs, ultimately improving service outcomes [15,16]. The collection and integration of large amounts of data may then be better used to support clinical and research services that operate across rural and remote settings, where on-site attendance can be difficult [15-17].

One public health setting that could benefit considerably from EDC is child diagnostic and assessment services, specifically those clinics that assess children with neurodevelopmental concerns. These neurodevelopmental clinics aim to provide assessments at the earliest possible time in a child’s development to increase the opportunity for earlier assessment, diagnosis, and intervention [18-20], with growing evidence that early intervention is associated with better long-term outcomes for the child and family [21,22]. Currently, however, public services are inundated with assessment requests, long wait lists, and limited resources to complete these tasks. These clinics typically do not use EDC, relying instead on pencil and paper for the vast majority of assessments. These public neurodevelopmental clinics are also more likely to provide services to a higher proportion of children and caregivers from disadvantaged backgrounds, those of lower socioeconomic status, and a higher proportion of linguistically diverse and indigenous communities in comparison to private clinical practices. It is, therefore, important to evaluate the utility of EDC in services that attend to these diverse patient populations.

Prior research has shown that demographic factors, such as age, socioeconomic status, level of education, language, and ethnicity may influence the completion of online data collection [23]. For example, socioeconomic deprivation (measured by the Scottish Index of Multiple Deprivation) and age (>70 years) have been associated with poorer completion of EDC in an orthopedic clinic in Scotland. Socioeconomic deprivation and age were both independently associated with lack of internet access [24], which may have contributed to the study findings. Similarly, a study of orthopedic surgery patients in California found that patients who were older (>75 years), of Hispanic or Black ethnicity, and had Medicare or Medicaid insurance were less likely to complete EDC patient-reported outcome surveys [25]. The authors argued that internet use is less prevalent among older patients, who formed much of the Medicare group. Additionally, Medicaid insurance includes low-income and vulnerable families who may not have had internet access to complete EDC surveys [25].

This study aimed to evaluate an initial pilot for EDC in one of Australia’s busiest child diagnostic and assessment services, the Child Development Unit (CDU) at The Children’s Hospital Westmead, part of the publicly funded Sydney Children’s Hospital Network, Australia. The CDU assesses approximately 600 children per year, referred by pediatricians, who present with complex neurodevelopmental problems. The CDU provides multidisciplinary neurodevelopmental assessments to the state of New South Wales, with some families attending from regional and rural areas, and a high proportion of families from culturally and linguistically diverse backgrounds. Assessment in the CDU begins with gathering information on family and developmental history, via a questionnaire completed by caregivers before attending their first appointment. The CDU has traditionally mailed a paper version of this questionnaire to families prior to their appointment, with families asked to post the completed questionnaire back to the clinic ahead of their appointment so that clinicians can be prepared for the on-site assessments. Clinicians have noted, however, that response rates have been consistently low, with less than 50% of families returning the questionnaire. Such problems lead to delays in terms of clinicians needing to complete and interpret the questionnaire with the family during their appointment.

To the best of our knowledge, this is the first study to assess EDC in a child development clinic. In this pilot study, we digitized the CDU’s caregiver questionnaire into a format that families could access via email and complete electronically in a secure, convenient, and efficient manner. We aimed to examine whether this would improve response rates for the questionnaire when compared to the paper version. We also investigated whether families preferred the electronic modality over the paper version, and the demographic characteristics that were associated with these preferences.

Methods

Setting

A total of 246 families who had an appointment in 2018-2019 with the CDU for the initial developmental assessment of their child were invited to participate in this study. Participants were consecutively recruited into this research study using opt-out informed consent methods. This study was approved by the Sydney Children’s Hospital Network Ethics Committee (LNR/17/SCHN/293). The first 62 families entered into this study were sent the paper questionnaire by post. Subsequently, the service transitioned to EDC methods, and a further 184 families were sent the questionnaire via email. No family declined to participate in this study.

Children are referred to the CDU for assessment of complex neurodevelopmental difficulties, including possible autism spectrum disorder, intellectual disabilities, global developmental delay, or specific learning disorders. Prior to their appointment at the CDU, families are expected to complete a 6-page questionnaire, covering demographic information, family history, and child developmental history. The questionnaire has a Flesch Reading Ease score of 76.7, indicating fairly easy readability that should be understood by 12- to 15-year olds [26]. These data enable the CDU to assemble a suitable team and prepare the relevant assessment measures before the appointment. Originally completed by families on paper, we digitized the questionnaire, creating a digital form on the Research Enterprise Data CAPture (REDCap) platform. REDCap is an electronic data capture tool endorsed by the University of Sydney for the secure collection of all research data [27,28]. REDCap is designed to support data capture for research studies and allows questionnaires to be emailed to families ahead of their appointment. Families can click on the link provided in the email invitation to open the questionnaire. Data are automatically saved in REDCap as they are entered in the online form. Once the family exits the form, it is immediately available for clinicians to view.

Procedure

As per the existing CDU procedure, families were advised of their scheduled appointment via phone. During this phone call, they are informed that a questionnaire will be posted to them and are instructed to complete it and post it back to the clinic. A week before their appointment, families who have not returned the questionnaire via post receive a reminder phone call to do so. Those families who have not returned the questionnaire by the time of their appointment are required to complete it on the day of their appointment with a member of the clinical team (social worker or clinical nurse consultant). In this study, we implemented a pilot REDCap procedure for completion of the questionnaire. Families were advised by phone of their appointment confirmation and told to expect an email inviting them to complete the questionnaire on the REDCap platform. A week before their appointment, an email reminder was sent automatically via REDCap to families who had not completed the questionnaire. Families who had not completed the questionnaire by the time of their appointment were required to complete it on the day of their appointment with a member of the clinical team (social worker or clinical nurse consultant). The existing CDU procedure and pilot REDCap procedure are outlined in Figure 1.

‎View this figure

Statistical Analyses

Response rates were compared for questionnaires sent to families via post (paper completion rates) and those sent to families via email (EDC completion rates). Differences in questionnaire response rates for the postal (existing CDU procedure) and EDC (pilot REDCap procedure) groups were analyzed using chi-square tests.

Within the EDC group, we conducted additional chi-square tests to assess questionnaire modality preference (online or paper). To investigate the influence of demographic characteristics (age, primary language spoken, and highest level of education of caregiver completing questionnaire) on questionnaire modality preference in the EDC group, independent samples t tests and chi-square tests were used. Additionally, given that the CDU services families from diverse ethnic backgrounds, chi-square tests were used to assess questionnaire modality preference in respondents who requested an interpreter for the developmental assessment, and respondents who identified as being Aboriginal or Torres Strait Islander (ATSI).

Results

Study Population

Data were collected from 246 families across 2018 and 2019. Postal data were collected on 62 families seen in a 3-month period between June and August 2018, and online data were collected on 184 families seen between March and November 2019. Table 1 shows the distribution of responding across the postal and EDC groups.

Differences in Response Rates Between the Postal and EDC Groups

As shown in Table 1, there was a significantly higher response rate in the EDC group (164/184, 89.1%) compared to the postal group (24/62, 39%; χ²₃=78.8, P<.001). There was no variability in the number of partially completed responses between the EDC (10/184, 5.4%) and postal (5/62, 8%) groups (χ²₁=0.2, P=.17).

Modality Preference

As shown in Table 2, families in the EDC group reported a significantly greater preference for completing and submitting the questionnaire online (151/173, 87.3%) compared to via post (22/173, 12.7%; χ²₁=96.2, P<.001). This preference did not vary as a function of when the questionnaire was completed (ie, prior to or during appointment) or the amount of the questionnaire completed (ie, partial or full completion).

Influence of Demographic Characteristics on Questionnaire Modality Preference

Table 3 displays key demographic characteristics for individuals who completed the questionnaire in the EDC group. Caregiver ages ranged from 24 to 72 years (mean 37.41 [SD 7.05]) and most respondents (145/173, 83.8%) reported English as the main language spoken at home, either alone or in conjunction with a second language. Education level of caregivers who completed the questionnaires was stratified into nontertiary education (high school/vocational/trade) or tertiary education (undergraduate degree/postgraduate degree). In considering the influence of these characteristics on modality preference, the age of the person completing the questionnaire did not influence preference for online compared to paper completion, t₁₆₇=0.99, P=.32, nor did the primary language spoken by the person completing the questionnaire, χ²₂=2.9, P=.24. However, our results revealed an association between education level and questionnaire modality preference, χ²₂=4.2, P=.04. Overall, those individuals who had received tertiary education were less likely to report a preference for completing the questionnaire on paper, relative to those individuals who had received nontertiary education. Looking at the education levels of individuals who reported a preference for completing the online questionnaires, the opposite pattern was observed, such that a higher proportion of tertiary educated individuals reported a preference for online completion, compared to nontertiary educated individuals.

Table 4 shows questionnaire modality preferences (online or paper) for caregivers who requested an interpreter for the assessment (10/173, 5.8%), and caregivers of ATSI origin (10/173, 5.8%). Within both subgroups of caregivers, there was no statistically significant difference in the number of families preferring online or paper completion (P=.53 and .21, respectively).

Discussion

The results of this study show that EDC was associated with significantly increased questionnaire completion rates (P<.001) from caregivers prior to attending their first appointment. Response rates from EDC were more than double the rate from paper-based data collection methods. This overall superior completion rate was shown across families of different ethnic backgrounds and from caregivers with different education levels. Almost 90% (151/173, 87.3%) of caregivers who completed EDC reported a continued preference for using EDC over paper-based methods. This preference did not vary as a function of age, primary language spoken, or belonging to a minority subgroup. Consistent with previous findings, however, a higher level of education (tertiary compared to nontertiary) appeared to be associated with modality preference [23]. Of those caregivers who indicated a preference for the paper version, a greater proportion reported nontertiary education as their highest level. Overall, this pilot study supports the continued evaluation of EDC to improve efficiency, cost, and clinical and research services in public hospital–based child development clinics and supports its utility across diverse education levels and cultural groups [29].

Our finding of reduced questionnaire completion rates when paper-based data collection methods were used align with the clinical experiences of the CDU team, with staff reporting a long history of low response rates for the paper version of the questionnaire. This low response rate results in added clinical burden, as clinicians are required to complete the questionnaire with families at the time of their on-site appointment. This is far from ideal, given the logistics involved in preparing for each on-site assessment. For instance, the CDU carries out approximately 15 comprehensive assessments per week, spanning 1-3 full days. Assessments include tests of intelligence, developmental delay, language, neuropsychological assessments, comprehensive parent interviews, and medical examinations. Assessments are complex, requiring specific rooms, materials, and team members to be organized in advance. Without receiving the completed questionnaire prior to a family’s appointment, the team of multidisciplinary clinicians are unable to adequately prepare for the type of assessment required in advance of the appointment. Our findings indicate that response rates are markedly improved when EDC is used, thereby giving the clinical team time to adequately prepare for assessments and optimizing time with families during assessments.

The increased response rates for the online questionnaire may be related to the increased preference seen for the electronic mode of completion [30-32]. Of note, in this study, we found that respondent age did not influence questionnaire preference for EDC. This finding is in contrast to previous studies that have reported a link between respondent age, response rates, and modality preferences [24,25,33]. Past studies have reported, however, that older age (eg, >60 years) is associated with greater preference for paper-based methods [33]. Given our sample principally comprised parents of young children, with a mean age of 37 and only 1 respondent above the age of 60, future studies may need to evaluate the utility of EDC in this service where primary caregivers are older (eg, grandparents). Our study also showed that the primary language spoken by the respondent did not influence questionnaire modality preference. Families who spoke a primary language other than English did not show differential preferences. Moreover, for 2 minority subgroups, namely, families who requested an interpreter for the assessment and families of ATSI origin, we did not observe an increased preference for paper-based methods compared to EDC. While these findings require replication in larger samples, they indicate that EDC may be suitable for the diverse populations typically serviced by developmental clinics such as the CDU. A more detailed investigation of language and ethnicity, and how these characteristics relate to socioeconomic status, may reveal differences and warrants further investigation [25].

In the small group of respondents who indicated a preference for the paper form, a majority (15/22, 68%) reported nontertiary education as their highest level. It has been shown that mothers with a high-school certificate level education or lower were less likely to use the internet for health-related purposes than those with a tertiary education. This may also be associated with lower socioeconomic status, lower household income, and lack of access to a computer or internet at home [34]. Education level is a known social determinant of health behavior and one that is difficult to address [35,36]. Publicly funded educational programs for vulnerable families may be a useful strategy to assist these families in better understanding their clinical care and options. Moreover, from a practical perspective, an understanding of the families likely to prefer paper forms will allow services such as the CDU to refocus their resources, by providing greater support at service entry to those families who cannot access EDC methods or require assistance from a team member. However, while we observed an association between education level and preference for a paper form, it should be noted that only a small minority of respondents (22/173, 12.7%) indicated preference for a paper version, highlighting the overall acceptability of EDC in this group.

There are some limitations in this study, namely, the relatively small sample size and uneven numbers in the postal and EDC groups. As the study aimed to explore tolerability of EDC, modality preference was only asked of online users. Additionally, the digitized questionnaire was a relatively short measure, taking approximately 15 minutes to complete. Results may differ for larger batteries of questionnaires and this would warrant further investigation in larger sample sizes. While we did not include an economic analysis in this study for EDC methods over paper-based approaches, this is clearly an avenue for future research. Such research would highlight the potential economic value of investing in high-quality internet-based health services for public settings. Moreover, future research would benefit from examining the feasibility and efficacy of EDC for clinician-collected data and evaluating staff satisfaction with these EDC methods. Such work would demonstrate the feasibility of extending EDC methods beyond patient-collected data in clinical health services such as the CDU.

Overall, this pilot study suggests that EDC is feasible and well accepted in a busy hospital-based clinic and has potential benefits for patient care, clinical practice, and clinical research. The increased response rates for online completion and the increased preference for EDC as opposed to paper forms suggest that EDC platforms may better suit the needs of families accessing these services.