This study was a descriptive, single group survey design with participants who were overseeing the implementation of the ezParent program (administrators) or trained facilitators for hybrid ezParent delivery (facilitators).

The ezParent Program is the web-based delivery of the Chicago Parent Program (CPP). The CPP has been shown to be effective in improving positive parenting skills, parenting self-efficacy, and child behavior problems in a population of low-income, urban parents of children aged 2-5 years old [17,18]. The ezParent Program teaches parents and caregivers the evidence-based strategies of CPP using 6 modules that include a video narrator and vignettes of families using the skills, reflection questions, program activities, and in-home practice assignments [19]. In a pilot randomized controlled trial (RCT) of ezParent (n=83), parents completed 82% of the 6 modules, reported high satisfaction with the program, and we found comparable effect sizes in improvements in parenting practices and reductions in parenting stress and child behavior problems to the group-based CPP [20,21]. In an RCT of self-administered ezParent in primary care (n=287), we failed to find significant main effects for parent and child behaviors [22]. Based on our findings and the extant literature suggesting web-based programs are more effective when provided along with human support [22,23] we are currently testing 2 models of hybrid delivery. The first includes 1:1 brief coaching as part of an RCT funded study (see Greene et al [24]) and community-based delivery of ezParent paired with web-based group sessions [25].

The ezParent program tracks user progress as parents use the program. Custom data tables collect and store user data with timestamps for logins, completed modules, end-of-module surveys, badges, practice assignments, and practice reviews. The administrative dashboard (ezDashboard) was developed to include these data points for hybrid delivery facilitators to monitor parent progress in the program and to inform hybrid sessions to encourage and support parent program uptake.

ezDashboard logins are created internally by programming staff and are provided to administrators and facilitators. Users access a “Parent Lookup” form by entering the unique ID given to parents during ezParent enrollment and the parent’s last name. These form fields are used together to ensure participant data security. The parent is added to the home page or “Parent List” if the user ID and last name match an individual user in the database. The parent list displays the user ID, last name, last login date, and the last viewed module. When a parent has completed their time in the ezParent program, users can easily remove that parent from their home page.

From the parent list, ezDashboard users can access more details for an individual parent user by clicking on a “Details” button. The top of the “Details” page displays the user ID, name, phone number, and participant email for easy contact. Contact information is followed by detailed use metrics, including a scrollable list of all login dates, completed modules with a green check mark to indicate completion and the date they were completed, module survey responses, completed badges and dates earned, and responses for practice assignments and practice reviews. Finally, a downloadable, individualized completion certificate is accessible once the last (sixth) module is completed. See Figures 1 and 2 for ezDashboard screenshots.

The ezDashboard’s user interface was made in React (version 17.0.2; MetaOpenSource), using responsive web design principles to allow access across different device sizes. Microsoft SQL Server stores real time user data, which the user interface fetches and displays.

The research protocol for this study was determined exempt by the institutional review board at The Ohio State University (study number 2023E0289). Participants completed a web-based consent form. All survey data were deidentified and participants were informed that deidentified data may be used or shared without additional informed consent. Participants who consented and completed the survey received a US $25 gift card for participation in this study.

We invited ezParent administrators (n=4; individuals overseeing implementation of ezParent) and facilitators (n=19; trained facilitators for hybrid ezParent delivery) to participate in the usability study with a goal for a sample size between 10 and 15. Administrators and facilitators were invited as they may have unique perspectives related to usability depending on their ezParent role. Macefield [26] suggests that a sample size of 3-20 is valid in a usability study to discover usability and potential problems and a sample of 10 will probably reveal a minimum of 82% of the problems and be useful in future design changes. Of the 23 invited to participate in the usability study, 16 consented to participate, and 15 completed the review of the ezDashboard and usability survey.

Potential participants received an email inviting them to participate in the usability study. The email invitation included a description of the project and a link to the consent form in REDCap (Research Electronic Data Capture; Vanderbilt University) [27]. Once consent was obtained, participants were instructed to spend at least 30 minutes reviewing the ezDashboard and evaluating the features. Participants who were not current users of the ezDashboard were provided with login access and instructed to add 2 sample users to their Parent List. Current users of the ezDashboard were instructed to sign into their accounts and review user accounts on their existing Parent List. Both groups were instructed to evaluate their users’ activity on the main page and on the detailed page of all modules in order to evaluate the usefulness of the ezDashboard. These procedures include typical tasks that a facilitator would take in evaluating and monitoring parent use of ezParent. Then, participants were prompted to complete the survey and asked to identify their current role using the ezParent program (eg, facilitator or administrator). At the completion of the survey, participants received a US $25 gift card for their participation.

The 10-item System Usability Scale (SUS) is a tool for assessing the usability of a product (eg, websites, cell phones, and apps). The 10 items are scored on a 5-point Likert scale (range 0 strongly disagree to 4 strongly agree). A total usability score, representing a composite measure of usability is created by reversing the score of even-numbered items, summing the items, and multiplying by 2.5 to convert the original total scores of 0-40 to a 0-100 scale [28]. A score of 70 is considered average, 80 good, and 90 or above excellent usability [29,30]. The SUS is a simple and efficient tool for assessing the usability and user-friendliness of a technological platform with demonstrated reliability across multiple studies (α=.91) and strong evidence of a single factor structure [29]. Our administration of SUS reflected the revised language and overall rating of the user-friendliness introduced in Bangor and colleagues [30]. Survey responses were collected in REDCap [31,32].

Following completion of the SUS, participants were prompted to respond to open-ended questions to elicit further information on their opinions of the ezDashboard. Questions included the frequency of use and the helpfulness of the ezDashboard and items that may be missing or could be changed in future iterations. Facilitators were asked specifically how access to this real time information helps them work with parents using the ezParent program and administrators were asked how the information helps them with overall management of the program. Finally, participants were asked to self-report their race or ethnicity, age, and gender.

The composite SUS score was calculated by reverse scoring even-numbered items so that all items were scored in the same direction. Composite scores were then calculated by summing the item responses and multiplying them by 2.5 so that they fell on a scale of 0-100. Summary statistics were calculated for the composite scores and frequencies are reported for the overall rating of user-friendliness.

For analysis of the open-ended questions, 2 authors organized the responses by group (eg, administrators and facilitators) and conducted a thematic analysis of all responses based on the steps outlined by Braun and Clarke [33]. First, we reviewed all the responses and generated initial themes and categories. These categories were reviewed by the 2 authors and confirmed by a third author. Finally, we categorized the themes in order to provide a description and examples in this report. We quantified the comments in each category in order to provide a frequency related to participants’ ideas, suggestions, and ideas provided related to the usability of the ezDashboard.

A majority of participants (N=15) were women (n=12, 80%), White (n=11, 73%), with a mean age of 40.9 years (SD 13.9; range 20-68; Table 1). Participants were ezParent facilitators (n=10) and administrators (n=4); 1 participant did not report their role with ezParent. Participants (n=9) who were actively using the ezDashboard at the time of the survey reported weekly use (n=3, 33%), once every few weeks (n=1, 11%), monthly (n=1, 11%), and less than monthly (n=4, 44%).

Scoring of the SUS indicated high usability of the ezDashboard with a total mean score of 83.5 (range 47.5-100; SD 16.3; median 90, IQR 72.5-97.5). Overall, most participants (n=10, 67%) rated the overall user-friendliness of the ezDashboard as excellent (n=9, 60%), or best imaginable (n=1, 7%; Table 2). On average new users (n=6) ranked the ezDashboard higher (mean 90.8, SD 8.01) than existing users (n=9; mean 78.6, SD 18.84); however, this difference was not statistically significant (P=.16) and may be the result of high variability associated with small samples.

The purpose of this usability study was to identify the overall performance of the ezParent administrative dashboard and understand users’ perceptions of the ease of navigation and usefulness of the ezDashboard in implementing hybrid delivery of the ezParent program. Further, we were interested in collecting users’ suggestions for changes or additions to the ezDashboard to improve the overall user experience for real-world use.

Overall, ratings on the SUS indicate good usability (mean 83.5, SD 16.3; median 90, IQR 72.5-97.5). According to Bangor and colleagues [29], a mean score of 83.5 is in the fourth quartile of scores, rated as acceptable, and falls between the good to excellent range using an adjective rating scale. In addition, overall ratings of the user-friendliness of the ezDashboard among all participants were positive (ok, good, excellent, and best imaginable). These initial ratings are promising, show an acceptable level of usability, and the written feedback provides us with concrete methods for improving the ezDashboard.

This study’s participants reported variable use of the ezDashboard (eg, ranging from weekly to less than monthly). The differences in use may be a function of the individual’s role using ezParent (eg, administrators may only need to use the dashboard monthly for overall program management while a coach conducting weekly calls would use it weekly). We do, however, need to consider the variability of use as a potential function of the overall usefulness of the information provided in the ezDashboard. Therefore, our next steps will be to provide clear instructions and descriptions of ezDashboard use as well as integrate suggested changes. Changes to the ezDashboard to provide desired information of parent use may increase regular use and uptake.

Harrington and colleagues [10] highlight the importance of including program usage metrics in dashboards to allow interventionists to make data-driven decisions, identify areas for program improvement, and support the ability to provide personalized support to program users. Further, real-time data can support tailored approaches to increase program uptake. Overall, our participants reported that they were able to use the ezDashboard information to take an individualized approach to the hybrid delivery of the ezParent program, and administrators used the ezDashboard data to provide oversight and incentives to parent participants. In addition, there were several excellent suggestions for ezDashboard improvements. The suggested changes varied by participant role with ezParent.

Facilitator suggestions were primarily fine-tuning the data presented in the ezDashboard to provide more nuanced use metrics beyond module completion. Since most digital analytics provide summary statistics, the ability to gather use metrics at the individual level provides important information for the facilitation of program engagement [14]. The data points suggested were partial module completion, identifying the actual location within the module of last use, and average time spent on the page. While the ezDashboard currently provides time stamps for date and time, the modules were completed and allows for a rough estimate of the speed at which a parent is moving through the program; more specific analytics for time in the program could provide more meaningful estimates for engagement. In our previous work, we found that on average parents spent 37.2 (SD 22.2) minutes per module with a range from 26.4 (module 5) to 47.9 (module 2) minutes [21]. Variations across modules occurred because of variations in pages per module; however, there was a significant decrease in minutes/module over time (eg, participants spent less time on later modules) [21]. This information is important for facilitators to monitor parent engagement and to encourage active involvement in the modules, particularly in later modules when data support a decrease in overall time of engagement.

The administrative participants identified other types of metrics and functions that would be helpful for program management (eg, the ability to download individual and cohort reports of usage metrics to allow for more streamlined monitoring and administrative access to modify participant demographics). The goal of the ezDashboard is to present individual use data and adaptations to include monitoring cohorts, which could increase administrative efficiencies. A benefit of web-based interventions is the reach and accessibility of programs and the integration of administrator dashboards for monitoring and management and has the potential to increase program uptake and overall efficiencies in program delivery. Future evaluation of the ezDashboard will focus on the effects of ezDashboard use on implementation factors related to organizational and individual program uptake and delivery.

Although deemed sufficient for usability testing to acquire feedback on user experience, our sample size was small. This study was an initial evaluation of the ezDashboard and provides valuable information for modifications. Our next steps are to integrate these findings into the ezDashboard to evaluate overall use of ezParent and dashboard use in a pragmatic trial with a larger sample of facilitators and administrators.

Additional limitations of this study include the usability testing being done virtually and at only 1 timepoint, which precluded our ability to examine participants’ actions in real time while using the ezDashboard in a pragmatic ongoing setting. Users’ feedback may change after continued use of the ezDashboard. The administrative dashboard was only tested for 1 specific intervention, ezParent; however, we believe these results could be applicable and inform the development of dashboards for different types of web-based programs.

The ezDashboard was initially developed to provide individual parent usage data to facilitators in real time to monitor parent progress in the program and support parent program uptake. The results of usability testing indicate that the ezDashboard is easy to use and provides functional information to facilitators in delivering the ezParent intervention. Providing resources for facilitators and administrators to aid in facilitation of the hybrid intervention may lead to improved parent uptake and outcomes [34]. Qualitative results indicate that integrating suggested features into the ezDashboard may help provide a smoother experience for facilitators, administrators, and ultimately the parents using the program. Administrative dashboards that provide real-time program usage data require an investment in the upfront cost of program development. The user facing program, in this case the ezParent, must be built to collect the user data that are to be displayed in the dashboard. For those considering integrating a dashboard for a web-based intervention, we suggest early planning during the initial development.